StarPU Handbook
Building and Installing StarPU

Installing a Binary Package

One of the StarPU developers being a Debian Developer, the packages are well integrated and very uptodate. To see which packages are available, simply type:

$ apt-cache search starpu

To install what you need, type for example:

$ sudo apt-get install libstarpu-1.3 libstarpu-dev

Installing from Source

StarPU can be built and installed by the standard means of the GNU autotools. The following chapter is intended to briefly remind how these tools can be used to install StarPU.

Optional Dependencies

The hwloc (http://www.open-mpi.org/software/hwloc) topology discovery library is not mandatory to use StarPU but strongly recommended. It allows for topology aware scheduling, which improves performance. hwloc is available in major free operating system distributions, and for most operating systems. Make sure to not only install a hwloc or libhwloc package, but also hwloc-devel or libhwloc-dev so as to have hwloc headers etc.

If libhwloc is installed in a standard location, no option is required, it will be detected automatically, otherwise --with-hwloc=<directory> should be used to specify its location.

If libhwloc is not available on your system, the option --without-hwloc should be explicitely given when calling the script configure.

Getting Sources

StarPU's sources can be obtained from the download page of the StarPU website (https://starpu.gitlabpages.inria.fr/files/).

All releases and the development tree of StarPU are freely available on StarPU SCM server under the LGPL license. Some releases are available under the BSD license.

The latest release can be downloaded from the StarPU download page (https://starpu.gitlabpages.inria.fr/files/).

The latest nightly snapshot can be downloaded from the StarPU website (https://starpu.gitlabpages.inria.fr/files/testing/).

And finally, current development version is also accessible via git. It should only be used if you need the very latest changes (i.e. less than a day old!).

$ git clone git@gitlab.inria.fr:starpu/starpu.git

Configuring StarPU

Running autogen.sh is not necessary when using the tarball releases of StarPU. However when using the source code from the git repository, you first need to generate the script configure and the different Makefiles. This requires the availability of autoconf and automake >= 2.60.

$ ./autogen.sh

You then need to configure StarPU. Details about options that are useful to give to configure are given in Compilation Configuration.

$ ./configure

If configure does not detect some software or produces errors, please make sure to post the contents of the file config.log when reporting the issue.

By default, the files produced during the compilation are placed in the source directory. As the compilation generates a lot of files, it is advised to put them all in a separate directory. It is then easier to cleanup, and this allows to compile several configurations out of the same source tree. To do so, simply enter the directory where you want the compilation to produce its files, and invoke the script configure located in the StarPU source directory.

$ mkdir build
$ cd build
$ ../configure

By default, StarPU will be installed in /usr/local/bin, /usr/local/lib, etc. You can specify an installation prefix other than /usr/local using the option –prefix, for instance:

$ ../configure --prefix=$HOME/starpu

Building StarPU

$ make

Once everything is built, you may want to test the result. An extensive set of regression tests is provided with StarPU. Running the tests is done by calling make check. These tests are run every night and the result from the main profile is publicly available (https://starpu.gitlabpages/files/testing/master/).

$ make check

Installing StarPU

In order to install StarPU at the location which was specified during configuration:

$ make install

If you have let StarPU install in /usr/local/, you additionally need to run

$ sudo ldconfig

so the libraries can be found by the system.

Libtool interface versioning information are included in libraries names (libstarpu-1.3.so, libstarpumpi-1.3.so and libstarpufft-1.3.so).

Setting up Your Own Code

Setting Flags for Compiling, Linking and Running Applications

StarPU provides a pkg-config executable to obtain relevant compiler and linker flags. As compiling and linking an application against StarPU may require to use specific flags or libraries (for instance CUDA or libspe2).

If StarPU was not installed at some standard location, the path of StarPU's library must be specified in the environment variable PKG_CONFIG_PATH to allow pkg-config to find it. For example if StarPU was installed in $STARPU_PATH:

$ export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:$STARPU_PATH/lib/pkgconfig

The flags required to compile or link against StarPU are then accessible with the following commands:

$ pkg-config --cflags starpu-1.3  # options for the compiler
$ pkg-config --libs starpu-1.3    # options for the linker

Note that it is still possible to use the API provided in the version 1.0 of StarPU by calling pkg-config with the starpu-1.0 package. Similar packages are provided for starpumpi-1.0 and starpufft-1.0. It is also possible to use the API provided in the version 0.9 of StarPU by calling pkg-config with the libstarpu package. Similar packages are provided for libstarpumpi and libstarpufft.

Make sure that pkg-config –libs starpu-1.3 actually produces some output before going further: PKG_CONFIG_PATH has to point to the place where starpu-1.3.pc was installed during make install.

Also pass the option –static if the application is to be linked statically.

It is also necessary to set the environment variable LD_LIBRARY_PATH to locate dynamic libraries at runtime.

$ export LD_LIBRARY_PATH=$STARPU_PATH/lib:$LD_LIBRARY_PATH

And it is useful to get access to the StarPU tools:

$ export PATH=$PATH:$STARPU_PATH/bin

It is then useful to check that StarPU executes correctly and finds your hardware:

$ starpu_machine_display

If it does not, please check the output of lstopo from hwloc and report the issue to the hwloc project, since this is what StarPU uses to detect the hardware.


A tool is provided to help setting all the environment variables needed by StarPU. Once StarPU is installed in a specific directory, calling the script bin/starpu_env will set in your current environment the variables STARPU_PATH, LD_LIBRARY_PATH, PKG_CONFIG_PATH, PATH and MANPATH.

$ source $STARPU_PATH/bin/starpu_env

Integrating StarPU in a Build System

Integrating StarPU in a Make Build System

When using a Makefile, the following lines can be added to set the options for the compiler and the linker:

CFLAGS          +=      $$(pkg-config --cflags starpu-1.3)
LDLIBS          +=      $$(pkg-config --libs starpu-1.3)

If you have a test-starpu.c file containing for instance:

#include <starpu.h>
#include <stdio.h>
int main(void)
{
int ret;
ret = starpu_init(NULL);
if (ret != 0)
{
return 1;
}
return 0;
}
void starpu_shutdown(void)
int starpu_init(struct starpu_conf *conf) STARPU_WARN_UNUSED_RESULT
int starpu_worker_get_count_by_type(enum starpu_worker_archtype type)
@ STARPU_CUDA_WORKER
Definition: starpu_worker.h:65
@ STARPU_CPU_WORKER
Definition: starpu_worker.h:64
@ STARPU_OPENCL_WORKER
Definition: starpu_worker.h:66

You can build it with make test-starpu and run it with ./test-starpu

Integrating StarPU in a CMake Build System

This section shows a minimal example integrating StarPU in an existing application's CMake build system.

Let's assume we want to build an executable from the following source code using CMake:

#include <starpu.h>
#include <stdio.h>
int main(void)
{
int ret;
ret = starpu_init(NULL);
if (ret != 0)
{
return 1;
}
return 0;
}

The CMakeLists.txt file below uses the Pkg-Config support from CMake to autodetect the StarPU installation and library dependences (such as libhwloc) provided that the PKG_CONFIG_PATH variable is set, and is sufficient to build a statically-linked executable. This example has been successfully tested with CMake 3.2, though it may work with earlier CMake 3.x versions.

{File CMakeLists.txt}
cmake_minimum_required (VERSION 3.2)
project (hello_starpu)
find_package(PkgConfig)
pkg_check_modules(STARPU REQUIRED starpu-1.3)
if (STARPU_FOUND)
include_directories (${STARPU_INCLUDE_DIRS})
link_directories (${STARPU_STATIC_LIBRARY_DIRS})
link_libraries (${STARPU_STATIC_LIBRARIES})
else (STARPU_FOUND)
message(FATAL_ERROR "StarPU not found")
endif()
add_executable(hello_starpu hello_starpu.c)

The following CMakeLists.txt implements an alternative, more complex strategy, still relying on Pkg-Config, but also taking into account additional flags. While more complete, this approach makes CMake's build types (Debug, Release, ...) unavailable because of the direct affectation to variable CMAKE_C_FLAGS. If both the full flags support and the build types support are needed, the CMakeLists.txt below may be altered to work with CMAKE_C_FLAGS_RELEASE, CMAKE_C_FLAGS_DEBUG, and others as needed. This example has been successfully tested with CMake 3.2, though it may work with earlier CMake 3.x versions.

{File CMakeLists.txt}
cmake_minimum_required (VERSION 3.2)
project (hello_starpu)
find_package(PkgConfig)
pkg_check_modules(STARPU REQUIRED starpu-1.3)
# This section must appear before 'add_executable'
if (STARPU_FOUND)
# CFLAGS other than -I
foreach(CFLAG ${STARPU_CFLAGS_OTHER})
set (CMAKE_C_FLAGS "${CMAKE_C_FLAGS} ${CFLAG}")
endforeach()
# Static LDFLAGS other than -L
foreach(LDFLAG ${STARPU_STATIC_LDFLAGS_OTHER})
set (CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} ${LDFLAG}")
endforeach()
# -L directories
link_directories(${STARPU_STATIC_LIBRARY_DIRS})
else (STARPU_FOUND)
message(FATAL_ERROR "StarPU not found")
endif()
add_executable(hello_starpu hello_starpu.c)
# This section must appear after 'add_executable'
if (STARPU_FOUND)
# -I directories
target_include_directories(hello_starpu PRIVATE ${STARPU_INCLUDE_DIRS})
# Static -l libs
target_link_libraries(hello_starpu PRIVATE ${STARPU_STATIC_LIBRARIES})
endif()

Running a Basic StarPU Application

Basic examples using StarPU are built in the directory examples/basic_examples/ (and installed in $STARPU_PATH/lib/starpu/examples/). You can for example run the example vector_scal.

$ ./examples/basic_examples/vector_scal
BEFORE: First element was 1.000000
AFTER: First element is 3.140000

When StarPU is used for the first time, the directory $STARPU_HOME/.starpu/ is created, performance models will be stored in this directory (STARPU_HOME).

Please note that buses are benchmarked when StarPU is launched for the first time. This may take a few minutes, or less if libhwloc is installed. This step is done only once per user and per machine.

Running a Basic StarPU Application on Microsoft Visual C

Batch files are provided to run StarPU applications under Microsoft Visual C. They are installed in $STARPU_PATH/bin/msvc.

To execute a StarPU application, you first need to set the environment variable STARPU_PATH.

c:\....> cd c:\cygwin\home\ci\starpu\
c:\....> set STARPU_PATH=c:\cygwin\home\ci\starpu\
c:\....> cd bin\msvc
c:\....> starpu_open.bat starpu_simple.c

The batch script will run Microsoft Visual C with a basic project file to run the given application.

The batch script starpu_clean.bat can be used to delete all compilation generated files.

The batch script starpu_exec.bat can be used to compile and execute a StarPU application from the command prompt.

c:\....> cd c:\cygwin\home\ci\starpu\
c:\....> set STARPU_PATH=c:\cygwin\home\ci\starpu\
c:\....> cd bin\msvc
c:\....> starpu_exec.bat ..\..\..\..\examples\basic_examples\hello_world.c
MSVC StarPU Execution
...
/out:hello_world.exe
...
Hello world (params = {1, 2.00000})
Callback function got argument 0000042
c:\....>

Kernel Threads Started by StarPU

StarPU automatically binds one thread per CPU core. It does not use SMT/hyperthreading because kernels are usually already optimized for using a full core, and using hyperthreading would make kernel calibration rather random.

Since driving GPUs is a CPU-consuming task, StarPU dedicates one core per GPU.

While StarPU tasks are executing, the application is not supposed to do computations in the threads it starts itself, tasks should be used instead.

If the application needs to reserve some cores for its own computations, it can do so with the field starpu_conf::reserve_ncpus, get the core IDs with starpu_get_next_bindid(), and bind to them with starpu_bind_thread_on().

Another option is for the application to pause StarPU by calling starpu_pause(), then to perform its own computations, and then to resume StarPU by calling starpu_resume() so that StarPU can execute tasks.

Enabling OpenCL

When both CUDA and OpenCL drivers are enabled, StarPU will launch an OpenCL worker for NVIDIA GPUs only if CUDA is not already running on them. This design choice was necessary as OpenCL and CUDA can not run at the same time on the same NVIDIA GPU, as there is currently no interoperability between them.

To enable OpenCL, you need either to disable CUDA when configuring StarPU:

$ ./configure --disable-cuda

or when running applications:

$ STARPU_NCUDA=0 ./application

OpenCL will automatically be started on any device not yet used by CUDA. So on a machine running 4 GPUS, it is therefore possible to enable CUDA on 2 devices, and OpenCL on the 2 other devices by doing so:

$ STARPU_NCUDA=2 ./application

Benchmarking StarPU

Some interesting benchmarks are installed among examples in $STARPU_PATH/lib/starpu/examples/. Make sure to try various schedulers, for instance STARPU_SCHED=dmda.

Task Size Overhead

This benchmark gives a glimpse into how long a task should be (in µs) for StarPU overhead to be low enough to keep efficiency. Running tasks_size_overhead.sh generates a plot of the speedup of tasks of various sizes, depending on the number of CPUs being used.

Data Transfer Latency

local_pingpong performs a ping-pong between the first two CUDA nodes, and prints the measured latency.

Matrix-Matrix Multiplication

sgemm and dgemm perform a blocked matrix-matrix multiplication using BLAS and cuBLAS. They output the obtained GFlops.

Cholesky Factorization

cholesky_* perform a Cholesky factorization (single precision). They use different dependency primitives.

LU Factorization

lu_* perform an LU factorization. They use different dependency primitives.

Simulated Benchmarks

It can also be convenient to try simulated benchmarks, if you want to give a try at CPU-GPU scheduling without actually having a GPU at hand. This can be done by using the SimGrid version of StarPU: first install the SimGrid simulator from http://simgrid.gforge.inria.fr/ (we tested with SimGrid from 3.11 to 3.16, and 3.18 to 3.30. SimGrid versions 3.25 and above need to be configured with -Denable_msg=ON. Other versions may have compatibility issues, 3.17 notably does not build at all. MPI simulation does not work with version 3.22). Then configure StarPU with --enable-simgrid and rebuild and install it, and then you can simulate the performance for a few virtualized systems shipped along StarPU: attila, mirage, idgraf, and sirocco.

For instance:

$ export STARPU_PERF_MODEL_DIR=$STARPU_PATH/share/starpu/perfmodels/sampling
$ export STARPU_HOSTNAME=attila
$ $STARPU_PATH/lib/starpu/examples/cholesky_implicit -size $((960*20)) -nblocks 20

Will show the performance of the cholesky factorization with the attila system. It will be interesting to try with different matrix sizes and schedulers.

Performance models are available for cholesky_*, lu_*, *gemm, with block sizes 320, 640, or 960 (plus 1440 for sirocco), and for stencil with block size 128x128x128, 192x192x192, and 256x256x256.

Read the chapter SimGrid Support for more information on the SimGrid support.