org.apache.spark.sql

Class SparkSession.Builder

Object

org.apache.spark.sql.SparkSession.Builder

All Implemented Interfaces:

Logging

Enclosing class:

SparkSession

public static class SparkSession.Builder
extends Object
implements Logging

Builder for SparkSession.

Constructor Summary

Constructors

Constructor and Description
Builder()

Method Summary

Modifier and Type	Method and Description
SparkSession.Builder	appName(String name) Sets a name for the application, which will be shown in the Spark web UI.
SparkSession.Builder	config(SparkConf conf) Sets a list of config options based on the given SparkConf.
SparkSession.Builder	config(String key, boolean value) Sets a config option.
SparkSession.Builder	config(String key, double value) Sets a config option.
SparkSession.Builder	config(String key, long value) Sets a config option.
SparkSession.Builder	config(String key, String value) Sets a config option.
SparkSession.Builder	enableHiveSupport() Enables Hive support, including connectivity to a persistent Hive metastore, support for Hive serdes, and Hive user-defined functions.
SparkSession	getOrCreate() Gets an existing SparkSession or, if there is no existing one, creates a new one based on the options set in this builder.
SparkSession.Builder	master(String master) Sets the Spark master URL to connect to, such as "local" to run locally, "local[4]" to run locally with 4 cores, or "spark://master:7077" to run on a Spark standalone cluster.
SparkSession.Builder	withExtensions(scala.Function1<SparkSessionExtensions,scala.runtime.BoxedUnit> f) Inject extensions into the SparkSession.

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.internal.Logging

initializeLogging, initializeLogIfNecessary, initializeLogIfNecessary, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning

Constructor Detail

Builder

public Builder()

Method Detail

appName

public SparkSession.Builder appName(String name)

Sets a name for the application, which will be shown in the Spark web UI. If no application name is set, a randomly generated name will be used.

Parameters:

name - (undocumented)

Returns:

(undocumented)

Since:

2.0.0

config

public SparkSession.Builder config(String key,
                                   String value)

Sets a config option. Options set using this method are automatically propagated to both SparkConf and SparkSession's own configuration.

Parameters:

key - (undocumented)

value - (undocumented)

Returns:

(undocumented)

Since:

2.0.0

config

public SparkSession.Builder config(String key,
                                   long value)

Sets a config option. Options set using this method are automatically propagated to both SparkConf and SparkSession's own configuration.

Parameters:

key - (undocumented)

value - (undocumented)

Returns:

(undocumented)

Since:

2.0.0

config

public SparkSession.Builder config(String key,
                                   double value)

Sets a config option. Options set using this method are automatically propagated to both SparkConf and SparkSession's own configuration.

Parameters:

key - (undocumented)

value - (undocumented)

Returns:

(undocumented)

Since:

2.0.0

config

public SparkSession.Builder config(String key,
                                   boolean value)

Sets a config option. Options set using this method are automatically propagated to both SparkConf and SparkSession's own configuration.

Parameters:

key - (undocumented)

value - (undocumented)

Returns:

(undocumented)

Since:

2.0.0

config

public SparkSession.Builder config(SparkConf conf)

Sets a list of config options based on the given SparkConf.

Parameters:

conf - (undocumented)

Returns:

(undocumented)

Since:

2.0.0

enableHiveSupport

public SparkSession.Builder enableHiveSupport()

Enables Hive support, including connectivity to a persistent Hive metastore, support for Hive serdes, and Hive user-defined functions.

Returns:

(undocumented)

Since:

2.0.0

getOrCreate

public SparkSession getOrCreate()

Gets an existing SparkSession or, if there is no existing one, creates a new one based on the options set in this builder.

This method first checks whether there is a valid thread-local SparkSession, and if yes, return that one. It then checks whether there is a valid global default SparkSession, and if yes, return that one. If no valid global default SparkSession exists, the method creates a new SparkSession and assigns the newly created SparkSession as the global default.

In case an existing SparkSession is returned, the non-static config options specified in this builder will be applied to the existing SparkSession.

Returns:

(undocumented)

Since:

2.0.0

master

public SparkSession.Builder master(String master)

Sets the Spark master URL to connect to, such as "local" to run locally, "local[4]" to run locally with 4 cores, or "spark://master:7077" to run on a Spark standalone cluster.

Parameters:

master - (undocumented)

Returns:

(undocumented)

Since:

2.0.0

withExtensions

public SparkSession.Builder withExtensions(scala.Function1<SparkSessionExtensions,scala.runtime.BoxedUnit> f)

Inject extensions into the SparkSession. This allows a user to add Analyzer rules, Optimizer rules, Planning Strategies or a customized parser.

Parameters:

f - (undocumented)

Returns:

(undocumented)

Since:

2.2.0