org.apache.spark.launcher

Class SparkLauncher

java.lang.Object

org.apache.spark.launcher.SparkLauncher

public class SparkLauncher
extends java.lang.Object

Launcher for Spark applications.

Use this class to start Spark applications programmatically. The class uses a builder pattern to allow clients to configure the Spark application and launch it as a child process.

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	CHILD_CONNECTION_TIMEOUT Maximum time (in ms) to wait for a child process to connect back to the launcher server when using @link{#start()}.
static java.lang.String	CHILD_PROCESS_LOGGER_NAME Logger name to use when launching a child process.
static java.lang.String	DEPLOY_MODE The Spark deploy mode.
static java.lang.String	DRIVER_EXTRA_CLASSPATH Configuration key for the driver class path.
static java.lang.String	DRIVER_EXTRA_JAVA_OPTIONS Configuration key for the driver VM options.
static java.lang.String	DRIVER_EXTRA_LIBRARY_PATH Configuration key for the driver native library path.
static java.lang.String	DRIVER_MEMORY Configuration key for the driver memory.
static java.lang.String	EXECUTOR_CORES Configuration key for the number of executor CPU cores.
static java.lang.String	EXECUTOR_EXTRA_CLASSPATH Configuration key for the executor class path.
static java.lang.String	EXECUTOR_EXTRA_JAVA_OPTIONS Configuration key for the executor VM options.
static java.lang.String	EXECUTOR_EXTRA_LIBRARY_PATH Configuration key for the executor native library path.
static java.lang.String	EXECUTOR_MEMORY Configuration key for the executor memory.
static java.lang.String	NO_RESOURCE A special value for the resource that tells Spark to not try to process the app resource as a file.
static java.lang.String	SPARK_MASTER The Spark master.

Constructor Summary

Constructors

Constructor and Description
SparkLauncher()
SparkLauncher(java.util.Map<java.lang.String,java.lang.String> env) Creates a launcher that will set the given environment variables in the child.

Method Summary

Methods

Modifier and Type	Method and Description
SparkLauncher	addAppArgs(java.lang.String... args) Adds command line arguments for the application.
SparkLauncher	addFile(java.lang.String file) Adds a file to be submitted with the application.
SparkLauncher	addJar(java.lang.String jar) Adds a jar file to be submitted with the application.
SparkLauncher	addPyFile(java.lang.String file) Adds a python file / zip / egg to be submitted with the application.
SparkLauncher	addSparkArg(java.lang.String arg) Adds a no-value argument to the Spark invocation.
SparkLauncher	addSparkArg(java.lang.String name, java.lang.String value) Adds an argument with a value to the Spark invocation.
java.lang.Process	launch() Launches a sub-process that will start the configured Spark application.
SparkLauncher	setAppName(java.lang.String appName) Set the application name.
SparkLauncher	setAppResource(java.lang.String resource) Set the main application resource.
SparkLauncher	setConf(java.lang.String key, java.lang.String value) Set a single configuration value for the application.
static void	setConfig(java.lang.String name, java.lang.String value) Set a configuration value for the launcher library.
SparkLauncher	setDeployMode(java.lang.String mode) Set the deploy mode for the application.
SparkLauncher	setJavaHome(java.lang.String javaHome) Set a custom JAVA_HOME for launching the Spark application.
SparkLauncher	setMainClass(java.lang.String mainClass) Sets the application class name for Java/Scala applications.
SparkLauncher	setMaster(java.lang.String master) Set the Spark master for the application.
SparkLauncher	setPropertiesFile(java.lang.String path) Set a custom properties file with Spark configuration for the application.
SparkLauncher	setSparkHome(java.lang.String sparkHome) Set a custom Spark installation location for the application.
SparkLauncher	setVerbose(boolean verbose) Enables verbose reporting for SparkSubmit.
SparkAppHandle	startApplication(SparkAppHandle.Listener... listeners) Starts a Spark application.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

SPARK_MASTER

public static final java.lang.String SPARK_MASTER

The Spark master.

See Also:

Constant Field Values

DEPLOY_MODE

public static final java.lang.String DEPLOY_MODE

The Spark deploy mode.

See Also:

Constant Field Values

DRIVER_MEMORY

public static final java.lang.String DRIVER_MEMORY

Configuration key for the driver memory.

See Also:

Constant Field Values

DRIVER_EXTRA_CLASSPATH

public static final java.lang.String DRIVER_EXTRA_CLASSPATH

Configuration key for the driver class path.

See Also:

Constant Field Values

DRIVER_EXTRA_JAVA_OPTIONS

public static final java.lang.String DRIVER_EXTRA_JAVA_OPTIONS

Configuration key for the driver VM options.

See Also:

Constant Field Values

DRIVER_EXTRA_LIBRARY_PATH

public static final java.lang.String DRIVER_EXTRA_LIBRARY_PATH

Configuration key for the driver native library path.

See Also:

Constant Field Values

EXECUTOR_MEMORY

public static final java.lang.String EXECUTOR_MEMORY

Configuration key for the executor memory.

See Also:

Constant Field Values

EXECUTOR_EXTRA_CLASSPATH

public static final java.lang.String EXECUTOR_EXTRA_CLASSPATH

Configuration key for the executor class path.

See Also:

Constant Field Values

EXECUTOR_EXTRA_JAVA_OPTIONS

public static final java.lang.String EXECUTOR_EXTRA_JAVA_OPTIONS

Configuration key for the executor VM options.

See Also:

Constant Field Values

EXECUTOR_EXTRA_LIBRARY_PATH

public static final java.lang.String EXECUTOR_EXTRA_LIBRARY_PATH

Configuration key for the executor native library path.

See Also:

Constant Field Values

EXECUTOR_CORES

public static final java.lang.String EXECUTOR_CORES

Configuration key for the number of executor CPU cores.

See Also:

Constant Field Values

CHILD_PROCESS_LOGGER_NAME

public static final java.lang.String CHILD_PROCESS_LOGGER_NAME

Logger name to use when launching a child process.

See Also:

Constant Field Values

NO_RESOURCE

public static final java.lang.String NO_RESOURCE

A special value for the resource that tells Spark to not try to process the app resource as a file. This is useful when the class being executed is added to the application using other means - for example, by adding jars using the package download feature.

See Also:

Constant Field Values

CHILD_CONNECTION_TIMEOUT

public static final java.lang.String CHILD_CONNECTION_TIMEOUT

Maximum time (in ms) to wait for a child process to connect back to the launcher server when using @link{#start()}.

See Also:

Constant Field Values

Constructor Detail

SparkLauncher

public SparkLauncher()

SparkLauncher

public SparkLauncher(java.util.Map<java.lang.String,java.lang.String> env)

Creates a launcher that will set the given environment variables in the child.

Parameters:

env - Environment variables to set.

Method Detail

setConfig

public static void setConfig(java.lang.String name,
             java.lang.String value)

Set a configuration value for the launcher library. These config values do not affect the launched application, but rather the behavior of the launcher library itself when managing applications.

Parameters:

name - Config name.

value - Config value.

Since:

1.6.0

setJavaHome

public SparkLauncher setJavaHome(java.lang.String javaHome)

Set a custom JAVA_HOME for launching the Spark application.

Parameters:

javaHome - Path to the JAVA_HOME to use.

Returns:

This launcher.

setSparkHome

public SparkLauncher setSparkHome(java.lang.String sparkHome)

Set a custom Spark installation location for the application.

Parameters:

sparkHome - Path to the Spark installation to use.

Returns:

This launcher.

setPropertiesFile

public SparkLauncher setPropertiesFile(java.lang.String path)

Set a custom properties file with Spark configuration for the application.

Parameters:

path - Path to custom properties file to use.

Returns:

This launcher.

setConf

public SparkLauncher setConf(java.lang.String key,
                    java.lang.String value)

Set a single configuration value for the application.

Parameters:

key - Configuration key.

value - The value to use.

Returns:

This launcher.

setAppName

public SparkLauncher setAppName(java.lang.String appName)

Set the application name.

Parameters:

appName - Application name.

Returns:

This launcher.

setMaster

public SparkLauncher setMaster(java.lang.String master)

Set the Spark master for the application.

Parameters:

master - Spark master.

Returns:

This launcher.

setDeployMode

public SparkLauncher setDeployMode(java.lang.String mode)

Set the deploy mode for the application.

Parameters:

mode - Deploy mode.

Returns:

This launcher.

setAppResource

public SparkLauncher setAppResource(java.lang.String resource)

Set the main application resource. This should be the location of a jar file for Scala/Java applications, or a python script for PySpark applications.

Parameters:

resource - Path to the main application resource.

Returns:

This launcher.

setMainClass

public SparkLauncher setMainClass(java.lang.String mainClass)

Sets the application class name for Java/Scala applications.

Parameters:

mainClass - Application's main class.

Returns:

This launcher.

addSparkArg

public SparkLauncher addSparkArg(java.lang.String arg)

Adds a no-value argument to the Spark invocation. If the argument is known, this method validates whether the argument is indeed a no-value argument, and throws an exception otherwise.

Use this method with caution. It is possible to create an invalid Spark command by passing unknown arguments to this method, since those are allowed for forward compatibility.

Parameters:

arg - Argument to add.

Returns:

This launcher.

Since:

1.5.0

addSparkArg

public SparkLauncher addSparkArg(java.lang.String name,
                        java.lang.String value)

Adds an argument with a value to the Spark invocation. If the argument name corresponds to a known argument, the code validates that the argument actually expects a value, and throws an exception otherwise.

It is safe to add arguments modified by other methods in this class (such as setMaster(String) - the last invocation will be the one to take effect.

Use this method with caution. It is possible to create an invalid Spark command by passing unknown arguments to this method, since those are allowed for forward compatibility.

Parameters:

name - Name of argument to add.

value - Value of the argument.

Returns:

This launcher.

Since:

1.5.0

addAppArgs

public SparkLauncher addAppArgs(java.lang.String... args)

Adds command line arguments for the application.

Parameters:

args - Arguments to pass to the application's main class.

Returns:

This launcher.

addJar

public SparkLauncher addJar(java.lang.String jar)

Adds a jar file to be submitted with the application.

Parameters:

jar - Path to the jar file.

Returns:

This launcher.

addFile

public SparkLauncher addFile(java.lang.String file)

Adds a file to be submitted with the application.

Parameters:

file - Path to the file.

Returns:

This launcher.

addPyFile

public SparkLauncher addPyFile(java.lang.String file)

Adds a python file / zip / egg to be submitted with the application.

Parameters:

file - Path to the file.

Returns:

This launcher.

setVerbose

public SparkLauncher setVerbose(boolean verbose)

Enables verbose reporting for SparkSubmit.

Parameters:

verbose - Whether to enable verbose output.

Returns:

This launcher.

launch

public java.lang.Process launch()
                         throws java.io.IOException

Launches a sub-process that will start the configured Spark application.

The startApplication(SparkAppHandle.Listener...) method is preferred when launching Spark, since it provides better control of the child application.

Returns:

A process handle for the Spark app.

Throws:

java.io.IOException

startApplication

public SparkAppHandle startApplication(SparkAppHandle.Listener... listeners)
                                throws java.io.IOException

Starts a Spark application.

This method returns a handle that provides information about the running application and can be used to do basic interaction with it.

The returned handle assumes that the application will instantiate a single SparkContext during its lifetime. Once that context reports a final state (one that indicates the SparkContext has stopped), the handle will not perform new state transitions, so anything that happens after that cannot be monitored. If the underlying application is launched as a child process, SparkAppHandle.kill() can still be used to kill the child process.

Currently, all applications are launched as child processes. The child's stdout and stderr are merged and written to a logger (see java.util.logging). The logger's name can be defined by setting CHILD_PROCESS_LOGGER_NAME in the app's configuration. If that option is not set, the code will try to derive a name from the application's name or main class / script file. If those cannot be determined, an internal, unique name will be used. In all cases, the logger name will start with "org.apache.spark.launcher.app", to fit more easily into the configuration of commonly-used logging systems.

Parameters:

listeners - Listeners to add to the handle before the app is launched.

Returns:

A handle for the launched application.

Throws:

java.io.IOException

Since:

1.6.0