streaming

Type Members

1. final class DataStreamReader extends Logging

   

   Interface used to load a streaming Dataset from external storage systems (e.g.

   Interface used to load a streaming Dataset from external storage systems (e.g. file systems, key-value stores, etc). Use SparkSession.readStream to access this.

   Annotations

   @Experimental() @Evolving()

   Since

   2.0.0

2. final class DataStreamWriter[T] extends AnyRef

   

   :: Experimental :: Interface used to write a streaming Dataset to external storage systems (e.g.

   :: Experimental :: Interface used to write a streaming Dataset to external storage systems (e.g. file systems, key-value stores, etc). Use Dataset.writeStream to access this.

   Annotations

   @Experimental() @Evolving()

   Since

   2.0.0

3. class OutputMode extends AnyRef

   

4. case class ProcessingTime(intervalMs: Long) extends Trigger with Product with Serializable

   

   :: Experimental :: A trigger that runs a query periodically based on the processing time.

   :: Experimental :: A trigger that runs a query periodically based on the processing time. If interval is 0, the query will run as fast as possible.

   Scala Example:

   df.write.trigger(ProcessingTime("10 seconds"))
   
   import scala.concurrent.duration._
   df.write.trigger(ProcessingTime(10.seconds))

   Java Example:

   df.write.trigger(ProcessingTime.create("10 seconds"))
   
   import java.util.concurrent.TimeUnit
   df.write.trigger(ProcessingTime.create(10, TimeUnit.SECONDS))

   Annotations

   @Experimental() @Evolving()

   Since

   2.0.0

5. class SinkProgress extends Serializable

   

   :: Experimental :: Information about progress made for a sink in the execution of a StreamingQuery during a trigger.

   :: Experimental :: Information about progress made for a sink in the execution of a StreamingQuery during a trigger. See StreamingQueryProgress for more information.

   Annotations

   @Experimental() @Evolving()

   Since

   2.1.0

6. class SourceProgress extends Serializable

   

   :: Experimental :: Information about progress made for a source in the execution of a StreamingQuery during a trigger.

   :: Experimental :: Information about progress made for a source in the execution of a StreamingQuery during a trigger. See StreamingQueryProgress for more information.

   Annotations

   @Experimental() @Evolving()

   Since

   2.1.0

7. class StateOperatorProgress extends Serializable

   

   :: Experimental :: Information about updates made to stateful operators in a StreamingQuery during a trigger.

   :: Experimental :: Information about updates made to stateful operators in a StreamingQuery during a trigger.

   Annotations

   @Experimental() @Evolving()

8. trait StreamingQuery extends AnyRef

   

   :: Experimental :: A handle to a query that is executing continuously in the background as new data arrives.

   :: Experimental :: A handle to a query that is executing continuously in the background as new data arrives. All these methods are thread-safe.

   Annotations

   @Experimental() @Evolving()

   Since

   2.0.0

9. class StreamingQueryException extends Exception

   

   :: Experimental :: Exception that stopped a StreamingQuery.

   :: Experimental :: Exception that stopped a StreamingQuery. Use cause get the actual exception that caused the failure.

   Annotations

   @Experimental() @Evolving()

   Since

   2.0.0

10. abstract class StreamingQueryListener extends AnyRef

    

    :: Experimental :: Interface for listening to events related to StreamingQueries.

    :: Experimental :: Interface for listening to events related to StreamingQueries.

    Annotations

    @Experimental() @Evolving()

    Since

    2.0.0

    Note

    The methods are not thread-safe as they may be called from different threads.

11. class StreamingQueryManager extends AnyRef

    

    :: Experimental :: A class to manage all the StreamingQuery active on a SparkSession.

    :: Experimental :: A class to manage all the StreamingQuery active on a SparkSession.

    Annotations

    @Experimental() @Evolving()

    Since

    2.0.0

12. class StreamingQueryProgress extends Serializable

    

    :: Experimental :: Information about progress made in the execution of a StreamingQuery during a trigger.

    :: Experimental :: Information about progress made in the execution of a StreamingQuery during a trigger. Each event relates to processing done for a single trigger of the streaming query. Events are emitted even when no new data is available to be processed.

    Annotations

    @Experimental() @Evolving()

    Since

    2.1.0

13. class StreamingQueryStatus extends Serializable

    

    :: Experimental :: Reports information about the instantaneous status of a streaming query.

    :: Experimental :: Reports information about the instantaneous status of a streaming query.

    Annotations

    @Experimental() @Evolving()

    Since

    2.1.0

14. sealed trait Trigger extends AnyRef

    

    :: Experimental :: Used to indicate how often results should be produced by a StreamingQuery.

    :: Experimental :: Used to indicate how often results should be produced by a StreamingQuery.

    Annotations

    @Experimental() @Evolving()

    Since

    2.0.0