Value Members

1.  final def !=(arg0: Any): Boolean

   Definition Classes

   AnyRef → Any

2.  final def ##(): Int

   Definition Classes

   AnyRef → Any

3.  final def ==(arg0: Any): Boolean

   Definition Classes

   AnyRef → Any

4.  def addStreamingListener(streamingListener: StreamingListener): Unit

   Add a org.apache.spark.streaming.scheduler.StreamingListener object for receiving system events related to streaming.

5.  final def asInstanceOf[T0]: T0

   Definition Classes

   Any

6.  def awaitTermination(): Unit

   Wait for the execution to stop.

   Wait for the execution to stop. Any exceptions that occurs during the execution will be thrown in this thread.

   Annotations

   @throws( ... )

7.  def awaitTerminationOrTimeout(timeout: Long): Boolean

   Wait for the execution to stop.

   Wait for the execution to stop. Any exceptions that occurs during the execution will be thrown in this thread.

   timeout

   time to wait in milliseconds

   returns

   true if it's stopped; or throw the reported error during the execution; or false if the waiting time elapsed before returning from the method.

   Annotations

   @throws( ... )

8.  def binaryRecordsStream(directory: String, recordLength: Int): JavaDStream[Array[Byte]]

   Create an input stream that monitors a Hadoop-compatible filesystem for new files and reads them as flat binary files with fixed record lengths, yielding byte arrays

   Create an input stream that monitors a Hadoop-compatible filesystem for new files and reads them as flat binary files with fixed record lengths, yielding byte arrays

   directory

   HDFS directory to monitor for new files

   recordLength

   The length at which to split the records

   Note

   We ensure that the byte array for each record in the resulting RDDs of the DStream has the provided record length.

9.  def checkpoint(directory: String): Unit

   Sets the context to periodically checkpoint the DStream operations for master fault-tolerance.

   Sets the context to periodically checkpoint the DStream operations for master fault-tolerance. The graph will be checkpointed every batch interval.

   directory

   HDFS-compatible directory where the checkpoint data will be reliably stored

10.  def clone(): AnyRef

    Attributes

    protected[lang]

    Definition Classes

    AnyRef

    Annotations

    @throws( ... ) @native()

11.  def close(): Unit

    Definition Classes

    JavaStreamingContext → Closeable → AutoCloseable

12.  final def eq(arg0: AnyRef): Boolean

    Definition Classes

    AnyRef

13.  def equals(arg0: Any): Boolean

    Definition Classes

    AnyRef → Any

14.  def fileStream[K, V, F <: InputFormat[K, V]](directory: String, kClass: Class[K], vClass: Class[V], fClass: Class[F], filter: Function[Path, Boolean], newFilesOnly: Boolean, conf: Configuration): JavaPairInputDStream[K, V]

    Create an input stream that monitors a Hadoop-compatible filesystem for new files and reads them using the given key-value types and input format.

    Create an input stream that monitors a Hadoop-compatible filesystem for new files and reads them using the given key-value types and input format. Files must be written to the monitored directory by "moving" them from another location within the same file system. File names starting with . are ignored.

    K

    Key type for reading HDFS file

    V

    Value type for reading HDFS file

    F

    Input format for reading HDFS file

    directory

    HDFS directory to monitor for new file

    kClass

    class of key for reading HDFS file

    vClass

    class of value for reading HDFS file

    fClass

    class of input format for reading HDFS file

    filter

    Function to filter paths to process

    newFilesOnly

    Should process only new files and ignore existing files in the directory

    conf

    Hadoop configuration

15.  def fileStream[K, V, F <: InputFormat[K, V]](directory: String, kClass: Class[K], vClass: Class[V], fClass: Class[F], filter: Function[Path, Boolean], newFilesOnly: Boolean): JavaPairInputDStream[K, V]

    Create an input stream that monitors a Hadoop-compatible filesystem for new files and reads them using the given key-value types and input format.

    Create an input stream that monitors a Hadoop-compatible filesystem for new files and reads them using the given key-value types and input format. Files must be written to the monitored directory by "moving" them from another location within the same file system. File names starting with . are ignored.

    K

    Key type for reading HDFS file

    V

    Value type for reading HDFS file

    F

    Input format for reading HDFS file

    directory

    HDFS directory to monitor for new file

    kClass

    class of key for reading HDFS file

    vClass

    class of value for reading HDFS file

    fClass

    class of input format for reading HDFS file

    filter

    Function to filter paths to process

    newFilesOnly

    Should process only new files and ignore existing files in the directory

16.  def fileStream[K, V, F <: InputFormat[K, V]](directory: String, kClass: Class[K], vClass: Class[V], fClass: Class[F]): JavaPairInputDStream[K, V]

    Create an input stream that monitors a Hadoop-compatible filesystem for new files and reads them using the given key-value types and input format.

    Create an input stream that monitors a Hadoop-compatible filesystem for new files and reads them using the given key-value types and input format. Files must be written to the monitored directory by "moving" them from another location within the same file system. File names starting with . are ignored.

    K

    Key type for reading HDFS file

    V

    Value type for reading HDFS file

    F

    Input format for reading HDFS file

    directory

    HDFS directory to monitor for new file

    kClass

    class of key for reading HDFS file

    vClass

    class of value for reading HDFS file

    fClass

    class of input format for reading HDFS file

17.  def finalize(): Unit

    Attributes

    protected[lang]

    Definition Classes

    AnyRef

    Annotations

    @throws( classOf[java.lang.Throwable] )

18.  final def getClass(): Class[_]

    Definition Classes

    AnyRef → Any

    Annotations

    @native()

19.  def getState(): StreamingContextState

    :: DeveloperApi ::

    :: DeveloperApi ::

    Return the current state of the context. The context can be in three possible states -

    - StreamingContextState.INITIALIZED - The context has been created, but not been started yet. Input DStreams, transformations and output operations can be created on the context.

    - StreamingContextState.ACTIVE - The context has been started, and been not stopped. Input DStreams, transformations and output operations cannot be created on the context.

    - StreamingContextState.STOPPED - The context has been stopped and cannot be used any more.

20.  def hashCode(): Int

    Definition Classes

    AnyRef → Any

    Annotations

    @native()

21.  final def isInstanceOf[T0]: Boolean

    Definition Classes

    Any

22.  final def ne(arg0: AnyRef): Boolean

    Definition Classes

    AnyRef

23.  final def notify(): Unit

    Definition Classes

    AnyRef

    Annotations

    @native()

24.  final def notifyAll(): Unit

    Definition Classes

    AnyRef

    Annotations

    @native()

25.  def queueStream[T](queue: Queue[JavaRDD[T]], oneAtATime: Boolean, defaultRDD: JavaRDD[T]): JavaInputDStream[T]

    Create an input stream from a queue of RDDs.

    Create an input stream from a queue of RDDs. In each batch, it will process either one or all of the RDDs returned by the queue.

    T

    Type of objects in the RDD

    queue

    Queue of RDDs

    oneAtATime

    Whether only one RDD should be consumed from the queue in every interval

    defaultRDD

    Default RDD is returned by the DStream when the queue is empty

    Note

    1. Changes to the queue after the stream is created will not be recognized. 2. Arbitrary RDDs can be added to queueStream, there is no way to recover data of those RDDs, so queueStream doesn't support checkpointing.

26.  def queueStream[T](queue: Queue[JavaRDD[T]], oneAtATime: Boolean): JavaInputDStream[T]

    Create an input stream from a queue of RDDs.

    Create an input stream from a queue of RDDs. In each batch, it will process either one or all of the RDDs returned by the queue.

    T

    Type of objects in the RDD

    queue

    Queue of RDDs

    oneAtATime

    Whether only one RDD should be consumed from the queue in every interval

    Note

    1. Changes to the queue after the stream is created will not be recognized. 2. Arbitrary RDDs can be added to queueStream, there is no way to recover data of those RDDs, so queueStream doesn't support checkpointing.

27.  def queueStream[T](queue: Queue[JavaRDD[T]]): JavaDStream[T]

    Create an input stream from a queue of RDDs.

    Create an input stream from a queue of RDDs. In each batch, it will process either one or all of the RDDs returned by the queue.

    T

    Type of objects in the RDD

    queue

    Queue of RDDs

    Note

    1. Changes to the queue after the stream is created will not be recognized. 2. Arbitrary RDDs can be added to queueStream, there is no way to recover data of those RDDs, so queueStream doesn't support checkpointing.

28.  def rawSocketStream[T](hostname: String, port: Int): JavaReceiverInputDStream[T]

    Create an input stream from network source hostname:port, where data is received as serialized blocks (serialized using the Spark's serializer) that can be directly pushed into the block manager without deserializing them.

    Create an input stream from network source hostname:port, where data is received as serialized blocks (serialized using the Spark's serializer) that can be directly pushed into the block manager without deserializing them. This is the most efficient way to receive data.

    T

    Type of the objects in the received blocks

    hostname

    Hostname to connect to for receiving data

    port

    Port to connect to for receiving data

29.  def rawSocketStream[T](hostname: String, port: Int, storageLevel: StorageLevel): JavaReceiverInputDStream[T]

    Create an input stream from network source hostname:port, where data is received as serialized blocks (serialized using the Spark's serializer) that can be directly pushed into the block manager without deserializing them.

    Create an input stream from network source hostname:port, where data is received as serialized blocks (serialized using the Spark's serializer) that can be directly pushed into the block manager without deserializing them. This is the most efficient way to receive data.

    T

    Type of the objects in the received blocks

    hostname

    Hostname to connect to for receiving data

    port

    Port to connect to for receiving data

    storageLevel

    Storage level to use for storing the received objects

30.  def receiverStream[T](receiver: Receiver[T]): JavaReceiverInputDStream[T]

    Create an input stream with any arbitrary user implemented receiver.

    Create an input stream with any arbitrary user implemented receiver. Find more details at: https://spark.apache.org/docs/latest/streaming-custom-receivers.html

    receiver

    Custom implementation of Receiver

31.  def remember(duration: Duration): Unit

    Sets each DStreams in this context to remember RDDs it generated in the last given duration.

    Sets each DStreams in this context to remember RDDs it generated in the last given duration. DStreams remember RDDs only for a limited duration of duration and releases them for garbage collection. This method allows the developer to specify how long to remember the RDDs ( if the developer wishes to query old data outside the DStream computation).

    duration

    Minimum duration that each DStream should remember its RDDs

32.  def socketStream[T](hostname: String, port: Int, converter: Function[InputStream, Iterable[T]], storageLevel: StorageLevel): JavaReceiverInputDStream[T]

    Create an input stream from network source hostname:port.

    Create an input stream from network source hostname:port. Data is received using a TCP socket and the receive bytes it interpreted as object using the given converter.

    T

    Type of the objects received (after converting bytes to objects)

    hostname

    Hostname to connect to for receiving data

    port

    Port to connect to for receiving data

    converter

    Function to convert the byte stream to objects

    storageLevel

    Storage level to use for storing the received objects

33.  def socketTextStream(hostname: String, port: Int): JavaReceiverInputDStream[String]

    Create an input stream from network source hostname:port.

    Create an input stream from network source hostname:port. Data is received using a TCP socket and the receive bytes is interpreted as UTF8 encoded \n delimited lines. Storage level of the data will be the default StorageLevel.MEMORY_AND_DISK_SER_2.

    hostname

    Hostname to connect to for receiving data

    port

    Port to connect to for receiving data

34.  def socketTextStream(hostname: String, port: Int, storageLevel: StorageLevel): JavaReceiverInputDStream[String]

    Create an input stream from network source hostname:port.

    Create an input stream from network source hostname:port. Data is received using a TCP socket and the receive bytes is interpreted as UTF8 encoded \n delimited lines.

    hostname

    Hostname to connect to for receiving data

    port

    Port to connect to for receiving data

    storageLevel

    Storage level to use for storing the received objects

35.  val sparkContext: JavaSparkContext

    The underlying SparkContext

36.  val ssc: StreamingContext
37.  def start(): Unit

    Start the execution of the streams.

38.  def stop(stopSparkContext: Boolean, stopGracefully: Boolean): Unit

    Stop the execution of the streams.

    Stop the execution of the streams.

    stopSparkContext

    Stop the associated SparkContext or not

    stopGracefully

    Stop gracefully by waiting for the processing of all received data to be completed

39.  def stop(stopSparkContext: Boolean): Unit

    Stop the execution of the streams.

    Stop the execution of the streams.

    stopSparkContext

    Stop the associated SparkContext or not

40.  def stop(): Unit

    Stop the execution of the streams.

    Stop the execution of the streams. Will stop the associated JavaSparkContext as well.

41.  final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes

    AnyRef

42.  def textFileStream(directory: String): JavaDStream[String]

    Create an input stream that monitors a Hadoop-compatible filesystem for new files and reads them as text files (using key as LongWritable, value as Text and input format as TextInputFormat).

    Create an input stream that monitors a Hadoop-compatible filesystem for new files and reads them as text files (using key as LongWritable, value as Text and input format as TextInputFormat). Files must be written to the monitored directory by "moving" them from another location within the same file system. File names starting with . are ignored. The text files must be encoded as UTF-8.

    directory

    HDFS directory to monitor for new file

43.  def toString(): String

    Definition Classes

    AnyRef → Any

44.  def transform[T](dstreams: List[JavaDStream[_]], transformFunc: Function2[List[JavaRDD[_]], Time, JavaRDD[T]]): JavaDStream[T]

    Create a new DStream in which each RDD is generated by applying a function on RDDs of the DStreams.

    Create a new DStream in which each RDD is generated by applying a function on RDDs of the DStreams. The order of the JavaRDDs in the transform function parameter will be the same as the order of corresponding DStreams in the list.

    Note

    For adding a JavaPairDStream in the list of JavaDStreams, convert it to a JavaDStream using org.apache.spark.streaming.api.java.JavaPairDStream.toJavaDStream(). In the transform function, convert the JavaRDD corresponding to that JavaDStream to a JavaPairRDD using org.apache.spark.api.java.JavaPairRDD.fromJavaRDD().

45.  def transformToPair[K, V](dstreams: List[JavaDStream[_]], transformFunc: Function2[List[JavaRDD[_]], Time, JavaPairRDD[K, V]]): JavaPairDStream[K, V]

    Create a new DStream in which each RDD is generated by applying a function on RDDs of the DStreams.

    Create a new DStream in which each RDD is generated by applying a function on RDDs of the DStreams. The order of the JavaRDDs in the transform function parameter will be the same as the order of corresponding DStreams in the list.

    Note

    For adding a JavaPairDStream in the list of JavaDStreams, convert it to a JavaDStream using org.apache.spark.streaming.api.java.JavaPairDStream.toJavaDStream(). In the transform function, convert the JavaRDD corresponding to that JavaDStream to a JavaPairRDD using org.apache.spark.api.java.JavaPairRDD.fromJavaRDD().

46.  def union[K, V](jdstreams: JavaPairDStream[K, V]*): JavaPairDStream[K, V]

    Create a unified DStream from multiple DStreams of the same type and same slide duration.

    Create a unified DStream from multiple DStreams of the same type and same slide duration.

    Annotations

    @varargs()

47.  def union[T](jdstreams: JavaDStream[T]*): JavaDStream[T]

    Create a unified DStream from multiple DStreams of the same type and same slide duration.

    Create a unified DStream from multiple DStreams of the same type and same slide duration.

    Annotations

    @varargs()

48.  final def wait(): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

49.  final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

50.  final def wait(arg0: Long): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws( ... ) @native()