org.apache.spark.streaming.dstream

Class InputDStream<T>

java.lang.Object

org.apache.spark.streaming.dstream.DStream<T>

org.apache.spark.streaming.dstream.InputDStream<T>

All Implemented Interfaces:

java.io.Serializable

Direct Known Subclasses:

ConstantInputDStream, ReceiverInputDStream

public abstract class InputDStream<T>
extends DStream<T>

This is the abstract base class for all input streams. This class provides methods start() and stop() which are called by Spark Streaming system to start and stop receiving data, respectively. Input streams that can generate RDDs from new data by running a service/thread only on the driver node (that is, without running a receiver on worker nodes), can be implemented by directly inheriting this InputDStream. For example, FileInputDStream, a subclass of InputDStream, monitors a HDFS directory from the driver for new files and generates RDDs with the new files. For implementing input streams that requires running a receiver on the worker nodes, use ReceiverInputDStream as the parent class.

param: _ssc Streaming context that will execute this input stream

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
InputDStream(StreamingContext _ssc, scala.reflect.ClassTag<T> evidence$1)

Method Summary

Methods

Modifier and Type	Method and Description
protected scala.Option<java.lang.String>	baseScope() The base scope associated with the operation that created this DStream.
scala.collection.immutable.List<DStream<?>>	dependencies() List of parent DStreams on which this DStream depends on
int	id() This is an unique identifier for the input stream.
protected scala.Option<org.apache.spark.streaming.scheduler.RateController>	rateController()
Duration	slideDuration() Time interval after which the DStream generates a RDD
abstract void	start() Method called to start receiving data.
abstract void	stop() Method called to stop receiving data.

Methods inherited from class org.apache.spark.streaming.dstream.DStream

cache, checkpoint, compute, context, count, countByValue, countByValueAndWindow, countByWindow, createRDDWithLocalProperties, filter, flatMap, foreachRDD, foreachRDD, glom, initializeLogIfNecessary, isTraceEnabled, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning, map, mapPartitions, persist, persist, print, print, reduce, reduceByWindow, reduceByWindow, repartition, saveAsObjectFiles, saveAsTextFiles, slice, slice, toPairDStreamFunctions, transform, transform, transformWith, transformWith, union, window, window

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

InputDStream

public InputDStream(StreamingContext _ssc,
            scala.reflect.ClassTag<T> evidence$1)

Method Detail

id

public int id()

This is an unique identifier for the input stream.

rateController

protected scala.Option<org.apache.spark.streaming.scheduler.RateController> rateController()

baseScope

protected scala.Option<java.lang.String> baseScope()

The base scope associated with the operation that created this DStream.

For InputDStreams, we use the name of this DStream as the scope name. If an outer scope is given, we assume that it includes an alternative name for this stream.

Overrides:

baseScope in class DStream<T>

Returns:

(undocumented)

dependencies

public scala.collection.immutable.List<DStream<?>> dependencies()

Description copied from class: DStream

List of parent DStreams on which this DStream depends on

Specified by:

dependencies in class DStream<T>

slideDuration

public Duration slideDuration()

Description copied from class: DStream

Time interval after which the DStream generates a RDD

Specified by:

slideDuration in class DStream<T>

start

public abstract void start()

Method called to start receiving data. Subclasses must implement this method.

stop

public abstract void stop()

Method called to stop receiving data. Subclasses must implement this method.