execution

Type Members

1. case class CreateHiveTableAsSelectCommand(tableDesc: CatalogTable, query: LogicalPlan, ignoreIfExists: Boolean) extends LeafNode with RunnableCommand with Product with Serializable

   

   Create table and insert the query result into it.

   Create table and insert the query result into it.

   tableDesc

   the Table Describe, which may contains serde, storage handler etc.

   query

   the query whose result will be insert into the new relation

   ignoreIfExists

   allow continue working if it's already exists, otherwise raise exception

2. case class HiveScriptIOSchema(inputRowFormat: Seq[(String, String)], outputRowFormat: Seq[(String, String)], inputSerdeClass: Option[String], outputSerdeClass: Option[String], inputSerdeProps: Seq[(String, String)], outputSerdeProps: Seq[(String, String)], recordReaderClass: Option[String], recordWriterClass: Option[String], schemaLess: Boolean) extends HiveInspectors with Product with Serializable

   

   The wrapper class of Hive input and output schema properties

3. case class InsertIntoHiveTable(table: MetastoreRelation, partition: Map[String, Option[String]], child: SparkPlan, overwrite: Boolean, ifNotExists: Boolean) extends SparkPlan with UnaryExecNode with Product with Serializable

   

   Command for writing data out to a Hive table.

   Command for writing data out to a Hive table.

   This class is mostly a mess, for legacy reasons (since it evolved in organic ways and had to follow Hive's internal implementations closely, which itself was a mess too). Please don't blame Reynold for this! He was just moving code around!

   In the future we should converge the write path for Hive with the normal data source write path, as defined in org.apache.spark.sql.execution.datasources.FileFormatWriter.

   table

   the logical plan representing the table. In the future this should be a org.apache.spark.sql.catalyst.catalog.CatalogTable once we converge Hive tables and data source tables.

   partition

   a map from the partition key to the partition value (optional). If the partition value is optional, dynamic partition insert will be performed. As an example, INSERT INTO tbl PARTITION (a=1, b=2) AS ... would have

   Map('a' -> Some('1'), 'b' -> Some('2'))

   and INSERT INTO tbl PARTITION (a=1, b) AS ... would have

   Map('a' -> Some('1'), 'b' -> None)

   .

   child

   the logical plan representing data to write to.

   overwrite

   overwrite existing table or partitions.

   ifNotExists

   If true, only write if the table or partition does not exist.

4. case class ScriptTransformation(input: Seq[Expression], script: String, output: Seq[Attribute], child: SparkPlan, ioschema: HiveScriptIOSchema) extends SparkPlan with UnaryExecNode with Product with Serializable

   

   Transforms the input by forking and running the specified script.

   Transforms the input by forking and running the specified script.

   input

   the set of expression that should be passed to the script.

   script

   the command that should be executed.

   output

   the attributes that are produced by the script.