StarPU Handbook
|
In order to enable online performance monitoring, the application can call starpu_profiling_status_set() with the parameter STARPU_PROFILING_ENABLE. It is possible to detect whether monitoring is already enabled or not by calling starpu_profiling_status_get(). Enabling monitoring also reinitialize all previously collected feedback. The environment variable STARPU_PROFILING can also be set to 1
to achieve the same effect. The function starpu_profiling_init() can also be called during the execution to reinitialize performance counters and to start the profiling if the environment variable STARPU_PROFILING is set to 1
.
Likewise, performance monitoring is stopped by calling starpu_profiling_status_set() with the parameter STARPU_PROFILING_DISABLE. Note that this does not reset the performance counters so that the application may consult them later on.
More details about the performance monitoring API are available in Profiling.
If profiling is enabled, a pointer to a structure starpu_profiling_task_info is put in the field starpu_task::profiling_info when a task terminates. This structure is automatically destroyed when the task structure is destroyed, either automatically or by calling starpu_task_destroy().
The structure starpu_profiling_task_info indicates the date when the task was submitted (starpu_profiling_task_info::submit_time), started (starpu_profiling_task_info::start_time), and terminated (starpu_profiling_task_info::end_time), relative to the initialization of StarPU with starpu_init(). It also specifies the identifier of the worker that has executed the task (starpu_profiling_task_info::workerid). These date are stored as timespec
structures which the user may convert into micro-seconds using the helper function starpu_timing_timespec_to_us().
It it worth noting that the application may directly access this structure from the callback executed at the end of the task. The structure starpu_task associated to the callback currently being executed is indeed accessible with the function starpu_task_get_current().
The field starpu_codelet::per_worker_stats is an array of counters. The i
-th entry of the array is incremented every time a task implementing the codelet is executed on the i
-th worker. This array is not reinitialized when profiling is enabled or disabled.
The second argument returned by the function starpu_profiling_worker_get_info() is a structure starpu_profiling_worker_info that gives statistics about the specified worker. This structure specifies when StarPU started collecting profiling information for that worker (starpu_profiling_worker_info::start_time), the duration of the profiling measurement interval (starpu_profiling_worker_info::total_time), the time spent executing kernels (starpu_profiling_worker_info::executing_time), the time spent sleeping because there is no task to execute at all (starpu_profiling_worker_info::sleeping_time), and the number of tasks that were executed while profiling was enabled. These values give an estimation of the proportion of time spent do real work, and the time spent either sleeping because there are not enough executable tasks or simply wasted in pure StarPU overhead.
Calling starpu_profiling_worker_get_info() resets the profiling information associated to a worker.
To easily display all this information, the environment variable STARPU_WORKER_STATS can be set to 1
(in addition to setting STARPU_PROFILING to 1). A summary will then be displayed at program termination. To display the summary in a file instead of the standard error stream, use the environment variable STARPU_WORKER_STATS_FILE.
Worker stats: CUDA 0.0 (4.7 GiB) 480 task(s) total: 1574.82 ms executing: 1510.72 ms sleeping: 0.00 ms overhead 64.10 ms 325.217970 GFlop/s CPU 0 22 task(s) total: 1574.82 ms executing: 1364.81 ms sleeping: 0.00 ms overhead 210.01 ms 7.512057 GFlop/s CPU 1 14 task(s) total: 1574.82 ms executing: 1500.13 ms sleeping: 0.00 ms overhead 74.69 ms 6.675853 GFlop/s CPU 2 14 task(s) total: 1574.82 ms executing: 1553.12 ms sleeping: 0.00 ms overhead 21.70 ms 7.152886 GFlop/s
The number of GFlops/s is available because the starpu_task::flops field of the tasks were filled (or STARPU_FLOPS used in starpu_task_insert()).
When an FxT trace is generated (see Generating Traces With FxT), it is also possible to use the tool starpu_workers_activity
(see Monitoring Activity) to generate a graphic showing the evolution of these values during the time, for the different workers.
The bus speed measured by StarPU can be displayed by using the tool starpu_machine_display
, for instance:
StarPU has found: 3 CUDA devices CUDA 0 (Tesla C2050 02:00.0) CUDA 1 (Tesla C2050 03:00.0) CUDA 2 (Tesla C2050 84:00.0) from to RAM to CUDA 0 to CUDA 1 to CUDA 2 RAM 0.000000 5176.530428 5176.492994 5191.710722 CUDA 0 4523.732446 0.000000 2414.074751 2417.379201 CUDA 1 4523.718152 2414.078822 0.000000 2417.375119 CUDA 2 4534.229519 2417.069025 2417.060863 0.000000
Statistics about the data transfers which were performed and temporal average of bandwidth usage can be obtained by setting the environment variable STARPU_BUS_STATS to 1
; a summary will then be displayed at program termination. To display the summary in a file instead of the standard error stream, use the environment variable STARPU_BUS_STATS_FILE.
Data transfer stats: RAM 0 -> CUDA 0 319.92 MB 213.10 MB/s (transfers : 91 - avg 3.52 MB) CUDA 0 -> RAM 0 214.45 MB 142.85 MB/s (transfers : 61 - avg 3.52 MB) RAM 0 -> CUDA 1 302.34 MB 201.39 MB/s (transfers : 86 - avg 3.52 MB) CUDA 1 -> RAM 0 133.59 MB 88.99 MB/s (transfers : 38 - avg 3.52 MB) CUDA 0 -> CUDA 1 144.14 MB 96.01 MB/s (transfers : 41 - avg 3.52 MB) CUDA 1 -> CUDA 0 130.08 MB 86.64 MB/s (transfers : 37 - avg 3.52 MB) RAM 0 -> CUDA 2 312.89 MB 208.42 MB/s (transfers : 89 - avg 3.52 MB) CUDA 2 -> RAM 0 133.59 MB 88.99 MB/s (transfers : 38 - avg 3.52 MB) CUDA 0 -> CUDA 2 151.17 MB 100.69 MB/s (transfers : 43 - avg 3.52 MB) CUDA 2 -> CUDA 0 105.47 MB 70.25 MB/s (transfers : 30 - avg 3.52 MB) CUDA 1 -> CUDA 2 175.78 MB 117.09 MB/s (transfers : 50 - avg 3.52 MB) CUDA 2 -> CUDA 1 203.91 MB 135.82 MB/s (transfers : 58 - avg 3.52 MB) Total transfers: 2.27 GB
Statistics about the data transfers which were performed over MPI can be obtained by setting the environment variable STARPU_COMM_STATS to 1
; a summary will then be displayed at program termination:
[starpu_comm_stats][1] TOTAL: 456.000000 B 0.000435 MB 0.000188 B/s 0.000000 MB/s [starpu_comm_stats][1:0] 456.000000 B 0.000435 MB 0.000188 B/s 0.000000 MB/s [starpu_comm_stats][0] TOTAL: 456.000000 B 0.000435 MB 0.000188 B/s 0.000000 MB/s [starpu_comm_stats][0:1] 456.000000 B 0.000435 MB 0.000188 B/s 0.000000 MB/s
These statistics can be plotted as heatmaps using StarPU tool starpu_mpi_comm_matrix.py
(see Debugging MPI).
A full example showing how to use the profiling API is available in the StarPU sources in the directory examples/profiling/
.
To achieve good scheduling, StarPU scheduling policies need to be able to estimate in advance the duration of a task. This is done by giving to codelets a performance model, by defining a structure starpu_perfmodel and providing its address in the field starpu_codelet::model. The fields starpu_perfmodel::symbol and starpu_perfmodel::type are mandatory, to give a name to the model, and the type of the model, since there are several kinds of performance models. For compatibility, make sure to initialize the whole structure to zero, either by using explicit memset(), or by letting the compiler implicitly do it as examplified below.
Measured at runtime (model type STARPU_HISTORY_BASED). This assumes that for a given set of data input/output sizes, the performance will always be about the same. This is very true for regular kernels on GPUs for instance (<0.1% error), and just a bit less true on CPUs (~=1% error). This also assumes that there are few different sets of data input/output sizes. StarPU will then keep record of the average time of previous executions on the various processing units, and use it as an estimation. History is done per task size, by using a hash of the input and ouput sizes as an index. It will also save it in $STARPU_HOME/.starpu/sampling/codelets
for further executions, and can be observed by using the tool starpu_perfmodel_display
, or drawn by using the tool starpu_perfmodel_plot
(Performance Model Calibration). The models are indexed by machine name. To share the models between machines (e.g. for a homogeneous cluster), use export STARPU_HOSTNAME=some_global_name
. Measurements are only done when using a task scheduler which makes use of it, such as dmda
. Measurements can also be provided explicitly by the application, by using the function starpu_perfmodel_update_history().
The following is a small code example.
If e.g. the code is recompiled with other compilation options, or several variants of the code are used, the symbol
string should be changed to reflect that, in order to recalibrate a new model from zero. The symbol
string can even be constructed dynamically at execution time, as long as this is done before submitting any task using it.
Measured at runtime and refined by regression (model types STARPU_REGRESSION_BASED and STARPU_NL_REGRESSION_BASED). This still assumes performance regularity, but works with various data input sizes, by applying regression over observed execution times. STARPU_REGRESSION_BASED uses an a*n^b
regression form, STARPU_NL_REGRESSION_BASED uses an a*n^b+c
(more precise than STARPU_REGRESSION_BASED, but costs a lot more to compute).
For instance, tests/perfmodels/regression_based.c
uses a regression-based performance model for the function memset().
Of course, the application has to issue tasks with varying size so that the regression can be computed. StarPU will not trust the regression unless there is at least 10% difference between the minimum and maximum observed input size. It can be useful to set the environment variable STARPU_CALIBRATE to 1
and run the application on varying input sizes with STARPU_SCHED set to dmda
scheduler, so as to feed the performance model for a variety of inputs. The application can also provide the measurements explictly by using the function starpu_perfmodel_update_history(). The tools starpu_perfmodel_display
and starpu_perfmodel_plot
can be used to observe how much the performance model is calibrated (Performance Model Calibration); when their output look good, STARPU_CALIBRATE can be reset to 0
to let StarPU use the resulting performance model without recording new measures, and STARPU_SCHED can be set to dmda
to benefit from the performance models. If the data input sizes vary a lot, it is really important to set STARPU_CALIBRATE to 0
, otherwise StarPU will continue adding the measures, and result with a very big performance model, which will take time a lot of time to load and save.
For non-linear regression, since computing it is quite expensive, it is only done at termination of the application. This means that the first execution of the application will use only history-based performance model to perform scheduling, without using regression.
Another type of model is STARPU_MULTIPLE_REGRESSION_BASED, which is based on multiple linear regression. In this model, the user defines both the relevant parameters and the equation for computing the task duration.
are the parameters of the task, added at the task creation. These need to be extracted by the
cl_perf_func
function, which should be defined by the user. are the exponents defined by the user in
model->combinations
table. Finally, coefficients are computed automatically by the StarPU at the end of the execution, using least squares method of the
dgels_
LAPACK function.
examples/mlr/mlr.c
example provides more details on the usage of STARPU_MULTIPLE_REGRESSION_BASED models. The --enable-mlr configure option needs to be set to calibrate the model.
Coefficients computation is done at the end of the execution, and the results are stored in standard codelet perfmodel files. Additional files containing the duration of task together with the value of each parameter are stored in .starpu/sampling/codelets/tmp/
directory. These files are reused when STARPU_CALIBRATE environment variable is set to 1
, to recompute coefficients based on the current, but also on the previous executions. By default StarPU uses a lightweight dgels implementation, but the --enable-mlr-system-blas configure option can be used to make StarPU use a system-provided dgels BLAS.
Additionally, when multiple linear regression models are not enabled through --enable-mlr or when the model->combinations
are not defined, StarPU will still write output files into .starpu/sampling/codelets/tmp/
to allow performing an analysis. This analysis typically aims at finding the most appropriate equation for the codelet and tools/starpu_mlr_analysis
script provides an example of how to perform such study.
Provided as an estimation from the application itself (model type STARPU_COMMON and field starpu_perfmodel::cost_function), see for instance examples/common/blas_model.h
and examples/common/blas_model.c
.
.per_arch[arch][nimpl].cost_function
have to be filled with pointers to functions which return the expected duration of the task in micro-seconds, one per architecture, see for instance tests/datawizard/locality.c
For STARPU_HISTORY_BASED, STARPU_REGRESSION_BASED, and STARPU_NL_REGRESSION_BASED, the dimensions of task data (both input and output) are used as an index by default. STARPU_HISTORY_BASED uses a CRC hash of the dimensions as an index to distinguish histories, and STARPU_REGRESSION_BASED and STARPU_NL_REGRESSION_BASED use the total size as an index for the regression.
The starpu_perfmodel::size_base and starpu_perfmodel::footprint fields however permit the application to override that, when for instance some of the data do not matter for task cost (e.g. mere reference table), or when using sparse structures (in which case it is the number of non-zeros which matter), or when there is some hidden parameter such as the number of iterations, or when the application actually has a very good idea of the complexity of the algorithm, and just not the speed of the processor, etc. The example in the directory examples/pi
uses this to include the number of iterations in the base size. starpu_perfmodel::size_base should be used when the variance of the actual performance is known (i.e. bigger return value is longer execution time), and thus particularly useful for STARPU_REGRESSION_BASED or STARPU_NL_REGRESSION_BASED. starpu_perfmodel::footprint can be used when the variance of the actual performance is unknown (irregular performance behavior, etc.), and thus only useful for STARPU_HISTORY_BASED. starpu_task_data_footprint() can be used as a base and combined with other parameters through starpu_hash_crc32c_be() for instance.
StarPU will automatically determine when the performance model is calibrated, or rather, it will assume the performance model is calibrated until the application submits a task for which the performance can not be predicted. For STARPU_HISTORY_BASED, StarPU will require 10 (STARPU_CALIBRATE_MINIMUM) measurements for a given size before estimating that an average can be taken as estimation for further executions with the same size. For STARPU_REGRESSION_BASED and STARPU_NL_REGRESSION_BASED, StarPU will require 10 (STARPU_CALIBRATE_MINIMUM) measurements, and that the minimum measured data size is smaller than 90% of the maximum measured data size (i.e. the measurement interval is large enough for a regression to have a meaning). Calibration can also be forced by setting the STARPU_CALIBRATE environment variable to 1
, or even reset by setting it to 2
.
How to use schedulers which can benefit from such performance model is explained in Task Scheduling Policies.
The same can be done for task energy consumption estimation, by setting the field starpu_codelet::energy_model the same way as the field starpu_codelet::model. Note: for now, the application has to give to the energy consumption performance model a name which is different from the execution time performance model.
The application can request time estimations from the StarPU performance models by filling a task structure as usual without actually submitting it. The data handles can be created by calling any of the functions starpu_*_data_register
with a NULL
pointer and -1
node and the desired data sizes, and need to be unregistered as usual. The functions starpu_task_expected_length() and starpu_task_expected_energy() can then be called to get an estimation of the task cost on a given arch. starpu_task_footprint() can also be used to get the footprint used for indexing history-based performance models. starpu_task_destroy() needs to be called to destroy the dummy task afterwards. See tests/perfmodels/regression_based.c
for an example.
The application can also request an on-the-fly XML report of the performance model, by calling starpu_perfmodel_dump_xml() to print the report to a FILE*
.