org.apache.spark.sql

Class ForeachWriter<T>

Object

org.apache.spark.sql.ForeachWriter<T>

All Implemented Interfaces:

java.io.Serializable

public abstract class ForeachWriter<T>
extends Object
implements scala.Serializable

A class to consume data generated by a StreamingQuery. Typically this is used to send the generated data to external systems. Each partition will use a new deserialized instance, so you usually should do all the initialization (e.g. opening a connection or initiating a transaction) in the open method.

Scala example:

   datasetOfString.writeStream.foreach(new ForeachWriter[String] {

     def open(partitionId: Long, version: Long): Boolean = {
       // open connection
     }

     def process(record: String) = {
       // write string to connection
     }

     def close(errorOrNull: Throwable): Unit = {
       // close the connection
     }
   })
 

Java example:

  datasetOfString.writeStream().foreach(new ForeachWriter<String>() {

    @Override
    public boolean open(long partitionId, long version) {
      // open connection
    }

    @Override
    public void process(String value) {
      // write string to connection
    }

    @Override
    public void close(Throwable errorOrNull) {
      // close the connection
    }
  });
 

Since:

2.0.0

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
ForeachWriter()

Method Summary

Modifier and Type	Method and Description
abstract void	close(Throwable errorOrNull) Called when stopping to process one partition of new data in the executor side.
abstract boolean	open(long partitionId, long version) Called when starting to process one partition of new data in the executor.
abstract void	process(T value) Called to process the data in the executor side.

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ForeachWriter

public ForeachWriter()

Method Detail

open

public abstract boolean open(long partitionId,
                             long version)

Called when starting to process one partition of new data in the executor. The version is for data deduplication when there are failures. When recovering from a failure, some data may be generated multiple times but they will always have the same version.

If this method finds using the partitionId and version that this partition has already been processed, it can return false to skip the further data processing. However, close still will be called for cleaning up resources.

Parameters:

partitionId - the partition id.

version - a unique id for data deduplication.

Returns:

true if the corresponding partition and version id should be processed. false indicates the partition should be skipped.

process

public abstract void process(T value)

Called to process the data in the executor side. This method will be called only when open returns true.

Parameters:

value - (undocumented)

close

public abstract void close(Throwable errorOrNull)

Called when stopping to process one partition of new data in the executor side. This is guaranteed to be called either open returns true or false. However, close won't be called in the following cases: - JVM crashes without throwing a Throwable - open throws a Throwable.

Parameters:

errorOrNull - the error thrown during processing data or null if there was no error.