HiveScriptIOSchema

Instance Constructors

new HiveScriptIOSchema(inputRowFormat: Seq[(String, String)], outputRowFormat: Seq[(String, String)], inputSerdeClass: Option[String], outputSerdeClass: Option[String], inputSerdeProps: Seq[(String, String)], outputSerdeProps: Seq[(String, String)], recordReaderClass: Option[String], recordWriterClass: Option[String], schemaLess: Boolean)

Type Members

implicit class typeInfoConversions extends AnyRef

Definition Classes
HiveInspectors

Value Members

final def !=(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def ##(): Int

Definition Classes
AnyRef → Any
final def ==(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def asInstanceOf[T0]: T0

Definition Classes
Any
def clone(): AnyRef

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( ... )
final def eq(arg0: AnyRef): Boolean

Definition Classes
AnyRef
def finalize(): Unit

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( classOf[java.lang.Throwable] )
final def getClass(): Class[_]

Definition Classes
AnyRef → Any
def initInputSerDe(input: Seq[Expression]): Option[(AbstractSerDe, ObjectInspector)]
def initOutputSerDe(output: Seq[Attribute]): Option[(AbstractSerDe, StructObjectInspector)]
val inputRowFormat: Seq[(String, String)]
val inputRowFormatMap: Map[String, String]
val inputSerdeClass: Option[String]
val inputSerdeProps: Seq[(String, String)]
def inspectorToDataType(inspector: ObjectInspector): DataType

Definition Classes
HiveInspectors
final def isInstanceOf[T0]: Boolean

Definition Classes
Any
def javaClassToDataType(clz: Class[_]): DataType

Definition Classes
HiveInspectors
final def ne(arg0: AnyRef): Boolean

Definition Classes
AnyRef
final def notify(): Unit

Definition Classes
AnyRef
final def notifyAll(): Unit

Definition Classes
AnyRef
val outputRowFormat: Seq[(String, String)]
val outputRowFormatMap: Map[String, String]
val outputSerdeClass: Option[String]
val outputSerdeProps: Seq[(String, String)]
def recordReader(inputStream: InputStream, conf: Configuration): Option[RecordReader]
val recordReaderClass: Option[String]
def recordWriter(outputStream: OutputStream, conf: Configuration): Option[RecordWriter]
val recordWriterClass: Option[String]
val schemaLess: Boolean
final def synchronized[T0](arg0: ⇒ T0): T0

Definition Classes
AnyRef
def toInspector(expr: Expression): ObjectInspector

Map the catalyst expression to ObjectInspector, however, if the expression is Literal or foldable, a constant writable object inspector returns; Otherwise, we always get the object inspector according to its data type(in catalyst)
Map the catalyst expression to ObjectInspector, however, if the expression is Literal or foldable, a constant writable object inspector returns; Otherwise, we always get the object inspector according to its data type(in catalyst)
expr
Catalyst expression to be mapped
returns
Hive java objectinspector (recursively).

Definition Classes
HiveInspectors
def toInspector(dataType: DataType): ObjectInspector

dataType
Catalyst data type
returns
Hive java object inspector (recursively), not the Writable ObjectInspector We can easily map to the Hive built-in object inspector according to the data type.

Definition Classes
HiveInspectors
def unwrap(data: Any, oi: ObjectInspector): Any

Converts hive types to native catalyst types.
Converts hive types to native catalyst types.
data
the data in Hive type
oi
the ObjectInspector associated with the Hive Type
returns
convert the data into catalyst type TODO return the function of (data => Any) instead for performance consideration Strictly follows the following order in unwrapping (constant OI has the higher priority): Constant Null object inspector => return null Constant object inspector => extract the value from constant object inspector Check whether the data is null => return null if true If object inspector prefers writable => extract writable from data and then get the catalyst type from the writable Extract the java object directly from the object inspector NOTICE: the complex data type requires recursive unwrapping.

Definition Classes
HiveInspectors
def unwrapperFor(field: StructField): (Any, MutableRow, Int) ⇒ Unit

Builds specific unwrappers ahead of time according to object inspector types to avoid pattern matching and branching costs per row.
Builds specific unwrappers ahead of time according to object inspector types to avoid pattern matching and branching costs per row.

Definition Classes
HiveInspectors
final def wait(): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long, arg1: Int): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
def wrap(row: Seq[Any], inspectors: Seq[ObjectInspector], cache: Array[AnyRef], dataTypes: Array[DataType]): Array[AnyRef]

Definition Classes
HiveInspectors
def wrap(row: InternalRow, inspectors: Seq[ObjectInspector], cache: Array[AnyRef], dataTypes: Array[DataType]): Array[AnyRef]

Definition Classes
HiveInspectors
def wrap(a: Any, oi: ObjectInspector, dataType: DataType): AnyRef

Converts native catalyst types to the types expected by Hive
Converts native catalyst types to the types expected by Hive
a
the value to be wrapped
oi
This ObjectInspector associated with the value returned by this function, and the ObjectInspector should also be consistent with those returned from toInspector: DataType => ObjectInspector and toInspector: Expression => ObjectInspector Strictly follows the following order in wrapping (constant OI has the higher priority): Constant object inspector => return the bundled value of Constant object inspector Check whether the a is null => return null if true If object inspector prefers writable object => return a Writable for the given data a Map the catalyst data to the boxed java primitive NOTICE: the complex data type requires recursive wrapping.

Definition Classes
HiveInspectors
def wrapperFor(oi: ObjectInspector, dataType: DataType): (Any) ⇒ Any

Wraps with Hive types based on object inspector.
Wraps with Hive types based on object inspector. TODO: Consolidate all hive OI/data interface code.

Attributes
protected
Definition Classes
HiveInspectors

Related Docs: object HiveScriptIOSchema | package execution

Instance Constructors

Type Members

implicit class typeInfoConversions extends AnyRef

Value Members

final def !=(arg0: Any): Boolean

final def ##(): Int

final def ==(arg0: Any): Boolean

final def asInstanceOf[T0]: T0

def clone(): AnyRef

final def eq(arg0: AnyRef): Boolean

def finalize(): Unit

final def getClass(): Class[_]

def initInputSerDe(input: Seq[Expression]): Option[(AbstractSerDe, ObjectInspector)]

def initOutputSerDe(output: Seq[Attribute]): Option[(AbstractSerDe, StructObjectInspector)]

val inputRowFormat: Seq[(String, String)]

val inputRowFormatMap: Map[String, String]

val inputSerdeClass: Option[String]

val inputSerdeProps: Seq[(String, String)]

def inspectorToDataType(inspector: ObjectInspector): DataType

final def isInstanceOf[T0]: Boolean

def javaClassToDataType(clz: Class[_]): DataType

final def ne(arg0: AnyRef): Boolean

final def notify(): Unit

final def notifyAll(): Unit

val outputRowFormat: Seq[(String, String)]

val outputRowFormatMap: Map[String, String]

val outputSerdeClass: Option[String]

val outputSerdeProps: Seq[(String, String)]

def recordReader(inputStream: InputStream, conf: Configuration): Option[RecordReader]

val recordReaderClass: Option[String]

def recordWriter(outputStream: OutputStream, conf: Configuration): Option[RecordWriter]

val recordWriterClass: Option[String]

val schemaLess: Boolean

final def synchronized[T0](arg0: ⇒ T0): T0

def toInspector(expr: Expression): ObjectInspector

def toInspector(dataType: DataType): ObjectInspector

def unwrap(data: Any, oi: ObjectInspector): Any

def unwrapperFor(field: StructField): (Any, MutableRow, Int) ⇒ Unit

final def wait(): Unit

final def wait(arg0: Long, arg1: Int): Unit

final def wait(arg0: Long): Unit

def wrap(row: Seq[Any], inspectors: Seq[ObjectInspector], cache: Array[AnyRef], dataTypes: Array[DataType]): Array[AnyRef]

def wrap(row: InternalRow, inspectors: Seq[ObjectInspector], cache: Array[AnyRef], dataTypes: Array[DataType]): Array[AnyRef]

def wrap(a: Any, oi: ObjectInspector, dataType: DataType): AnyRef

def wrapperFor(oi: ObjectInspector, dataType: DataType): (Any) ⇒ Any

Inherited from Serializable

Inherited from Serializable

Inherited from Product

Inherited from Equals

Inherited from HiveInspectors

Inherited from AnyRef

Inherited from Any

Ungrouped