feature

Feature transformers

The ml.feature package provides common feature transformers that help convert raw data or features into more suitable forms for model fitting. Most feature transformers are implemented as Transformers, which transform one DataFrame into another, e.g., HashingTF. Some feature transformers are implemented as Estimators, because the transformation requires some aggregated information of the dataset, e.g., document frequencies in IDF. For those feature transformers, calling Estimator!.fit is required to obtain the model first, e.g., IDFModel, in order to apply transformation. The transformation is usually done by appending new columns to the input DataFrame, so all input columns are carried over.

We try to make each transformer minimal, so it becomes flexible to assemble feature transformation pipelines. Pipeline can be used to chain feature transformers, and VectorAssembler can be used to combine multiple feature transformations, for example:

import org.apache.spark.ml.feature._
import org.apache.spark.ml.Pipeline

// a DataFrame with three columns: id (integer), text (string), and rating (double).
val df = sqlContext.createDataFrame(Seq(
  (0, "Hi I heard about Spark", 3.0),
  (1, "I wish Java could use case classes", 4.0),
  (2, "Logistic regression models are neat", 4.0)
)).toDF("id", "text", "rating")

// define feature transformers
val tok = new RegexTokenizer()
  .setInputCol("text")
  .setOutputCol("words")
val sw = new StopWordsRemover()
  .setInputCol("words")
  .setOutputCol("filtered_words")
val tf = new HashingTF()
  .setInputCol("filtered_words")
  .setOutputCol("tf")
  .setNumFeatures(10000)
val idf = new IDF()
  .setInputCol("tf")
  .setOutputCol("tf_idf")
val assembler = new VectorAssembler()
  .setInputCols(Array("tf_idf", "rating"))
  .setOutputCol("features")

// assemble and fit the feature transformation pipeline
val pipeline = new Pipeline()
  .setStages(Array(tok, sw, tf, idf, assembler))
val model = pipeline.fit(df)

// save transformed features with raw data
model.transform(df)
  .select("id", "text", "rating", "features")
  .write.format("parquet").save("/output/path")

Some feature transformers implemented in MLlib are inspired by those implemented in scikit-learn. The major difference is that most scikit-learn feature transformers operate eagerly on the entire input dataset, while MLlib's feature transformers operate lazily on individual columns, which is more efficient and flexible to handle large and complex datasets.

Source: package.scala
See also: scikit-learn.preprocessing

Linear Supertypes

AnyRef, Any

Type Members

final class Binarizer extends Transformer with HasInputCol with HasOutputCol with DefaultParamsWritable

:: Experimental :: Binarize a column of continuous features given a threshold.
final class Bucketizer extends Model[Bucketizer] with HasInputCol with HasOutputCol with DefaultParamsWritable

:: Experimental :: Bucketizer maps a column of continuous features to a column of feature buckets.
final class ChiSqSelector extends Estimator[ChiSqSelectorModel] with ChiSqSelectorParams with DefaultParamsWritable

:: Experimental :: Chi-Squared feature selection, which selects categorical features to use for predicting a categorical label.
final class ChiSqSelectorModel extends Model[ChiSqSelectorModel] with ChiSqSelectorParams with MLWritable

:: Experimental :: Model fitted by ChiSqSelector.
class CountVectorizer extends Estimator[CountVectorizerModel] with CountVectorizerParams with DefaultParamsWritable

:: Experimental :: Extracts a vocabulary from document collections and generates a CountVectorizerModel.
class CountVectorizerModel extends Model[CountVectorizerModel] with CountVectorizerParams with MLWritable

:: Experimental :: Converts a text document to a sparse vector of token counts.
class DCT extends UnaryTransformer[Vector, Vector, DCT] with DefaultParamsWritable

:: Experimental :: A feature transformer that takes the 1D discrete cosine transform of a real vector.
class ElementwiseProduct extends UnaryTransformer[Vector, Vector, ElementwiseProduct]

:: Experimental :: Outputs the Hadamard product (i.
class HashingTF extends Transformer with HasInputCol with HasOutputCol with DefaultParamsWritable

:: Experimental :: Maps a sequence of terms to their term frequencies using the hashing trick.
final class IDF extends Estimator[IDFModel] with IDFBase with DefaultParamsWritable

:: Experimental :: Compute the Inverse Document Frequency (IDF) given a collection of documents.
class IDFModel extends Model[IDFModel] with IDFBase with MLWritable

:: Experimental :: Model fitted by IDF.
class IndexToString extends Transformer with HasInputCol with HasOutputCol with DefaultParamsWritable

:: Experimental :: A Transformer that maps a column of indices back to a new column of corresponding string values.
class Interaction extends Transformer with HasInputCols with HasOutputCol with DefaultParamsWritable

:: Experimental :: Implements the feature interaction transform.
class MinMaxScaler extends Estimator[MinMaxScalerModel] with MinMaxScalerParams with DefaultParamsWritable

:: Experimental :: Rescale each feature individually to a common range [min, max] linearly using column summary statistics, which is also known as min-max normalization or Rescaling.
class MinMaxScalerModel extends Model[MinMaxScalerModel] with MinMaxScalerParams with MLWritable

:: Experimental :: Model fitted by MinMaxScaler.
class NGram extends UnaryTransformer[Seq[String], Seq[String], NGram] with DefaultParamsWritable

:: Experimental :: A feature transformer that converts the input array of strings into an array of n-grams.
class Normalizer extends UnaryTransformer[Vector, Vector, Normalizer] with DefaultParamsWritable

:: Experimental :: Normalize a vector to have unit norm using the given p-norm.
class OneHotEncoder extends Transformer with HasInputCol with HasOutputCol with DefaultParamsWritable

:: Experimental :: A one-hot encoder that maps a column of category indices to a column of binary vectors, with at most a single one-value per row that indicates the input category index.
class PCA extends Estimator[PCAModel] with PCAParams with DefaultParamsWritable

:: Experimental :: PCA trains a model to project vectors to a low-dimensional space using PCA.
class PCAModel extends Model[PCAModel] with PCAParams with MLWritable

:: Experimental :: Model fitted by PCA.
class PolynomialExpansion extends UnaryTransformer[Vector, Vector, PolynomialExpansion] with DefaultParamsWritable

:: Experimental :: Perform feature expansion in a polynomial space.
final class QuantileDiscretizer extends Estimator[Bucketizer] with QuantileDiscretizerBase with DefaultParamsWritable

:: Experimental :: QuantileDiscretizer takes a column with continuous features and outputs a column with binned categorical features.
class RFormula extends Estimator[RFormulaModel] with RFormulaBase

:: Experimental :: Implements the transforms required for fitting a dataset against an R model formula.
class RFormulaModel extends Model[RFormulaModel] with RFormulaBase

:: Experimental :: A fitted RFormula.
class RegexTokenizer extends UnaryTransformer[String, Seq[String], RegexTokenizer] with DefaultParamsWritable

:: Experimental :: A regex based tokenizer that extracts tokens either by using the provided regex pattern to split the text (default) or repeatedly matching the regex (if gaps is false).
class SQLTransformer extends Transformer with DefaultParamsWritable

:: Experimental :: Implements the transformations which are defined by SQL statement.
class StandardScaler extends Estimator[StandardScalerModel] with StandardScalerParams with DefaultParamsWritable

:: Experimental :: Standardizes features by removing the mean and scaling to unit variance using column summary statistics on the samples in the training set.
class StandardScalerModel extends Model[StandardScalerModel] with StandardScalerParams with MLWritable

:: Experimental :: Model fitted by StandardScaler.
class StopWordsRemover extends Transformer with HasInputCol with HasOutputCol with DefaultParamsWritable

:: Experimental :: A feature transformer that filters out stop words from input.
class StringIndexer extends Estimator[StringIndexerModel] with StringIndexerBase with DefaultParamsWritable

:: Experimental :: A label indexer that maps a string column of labels to an ML column of label indices.
class StringIndexerModel extends Model[StringIndexerModel] with StringIndexerBase with MLWritable

:: Experimental :: Model fitted by StringIndexer.
class Tokenizer extends UnaryTransformer[String, Seq[String], Tokenizer] with DefaultParamsWritable

:: Experimental :: A tokenizer that converts the input string to lowercase and then splits it by white spaces.
class VectorAssembler extends Transformer with HasInputCols with HasOutputCol with DefaultParamsWritable

:: Experimental :: A feature transformer that merges multiple columns into a vector column.
class VectorIndexer extends Estimator[VectorIndexerModel] with VectorIndexerParams with DefaultParamsWritable

:: Experimental :: Class for indexing categorical feature columns in a dataset of Vector.
class VectorIndexerModel extends Model[VectorIndexerModel] with VectorIndexerParams with MLWritable

:: Experimental :: Transform categorical features to use 0-based indices instead of their original values.
final class VectorSlicer extends Transformer with HasInputCol with HasOutputCol with DefaultParamsWritable

:: Experimental :: This class takes a feature vector and outputs a new feature vector with a subarray of the original features.
final class Word2Vec extends Estimator[Word2VecModel] with Word2VecBase with DefaultParamsWritable

:: Experimental :: Word2Vec trains a model of Map(String, Vector), i.
class Word2VecModel extends Model[Word2VecModel] with Word2VecBase with MLWritable

:: Experimental :: Model fitted by Word2Vec.

Value Members

object Binarizer extends DefaultParamsReadable[Binarizer] with Serializable

Annotations
@Since( "1.6.0" )
object Bucketizer extends DefaultParamsReadable[Bucketizer] with Serializable
object ChiSqSelector extends DefaultParamsReadable[ChiSqSelector] with Serializable

Annotations
@Since( "1.6.0" )
object ChiSqSelectorModel extends MLReadable[ChiSqSelectorModel] with Serializable

Annotations
@Since( "1.6.0" )
object CountVectorizer extends DefaultParamsReadable[CountVectorizer] with Serializable

Annotations
@Since( "1.6.0" )
object CountVectorizerModel extends MLReadable[CountVectorizerModel] with Serializable

Annotations
@Since( "1.6.0" )
object DCT extends DefaultParamsReadable[DCT] with Serializable

Annotations
@Since( "1.6.0" )
object HashingTF extends DefaultParamsReadable[HashingTF] with Serializable

Annotations
@Since( "1.6.0" )
object IDF extends DefaultParamsReadable[IDF] with Serializable

Annotations
@Since( "1.6.0" )
object IDFModel extends MLReadable[IDFModel] with Serializable

Annotations
@Since( "1.6.0" )
object IndexToString extends DefaultParamsReadable[IndexToString] with Serializable

Annotations
@Since( "1.6.0" )
object Interaction extends DefaultParamsReadable[Interaction] with Serializable

Annotations
@Since( "1.6.0" )
object MinMaxScaler extends DefaultParamsReadable[MinMaxScaler] with Serializable

Annotations
@Since( "1.6.0" )
object MinMaxScalerModel extends MLReadable[MinMaxScalerModel] with Serializable

Annotations
@Since( "1.6.0" )
object NGram extends DefaultParamsReadable[NGram] with Serializable

Annotations
@Since( "1.6.0" )
object Normalizer extends DefaultParamsReadable[Normalizer] with Serializable

Annotations
@Since( "1.6.0" )
object OneHotEncoder extends DefaultParamsReadable[OneHotEncoder] with Serializable

Annotations
@Since( "1.6.0" )
object PCA extends DefaultParamsReadable[PCA] with Serializable

Annotations
@Since( "1.6.0" )
object PCAModel extends MLReadable[PCAModel] with Serializable

Annotations
@Since( "1.6.0" )
object PolynomialExpansion extends DefaultParamsReadable[PolynomialExpansion] with Serializable

The expansion is done via recursion.
object QuantileDiscretizer extends DefaultParamsReadable[QuantileDiscretizer] with Logging with Serializable

Annotations
@Since( "1.6.0" )
object RegexTokenizer extends DefaultParamsReadable[RegexTokenizer] with Serializable

Annotations
@Since( "1.6.0" )
object SQLTransformer extends DefaultParamsReadable[SQLTransformer] with Serializable

Annotations
@Since( "1.6.0" )
object StandardScaler extends DefaultParamsReadable[StandardScaler] with Serializable

Annotations
@Since( "1.6.0" )
object StandardScalerModel extends MLReadable[StandardScalerModel] with Serializable

Annotations
@Since( "1.6.0" )
object StopWordsRemover extends DefaultParamsReadable[StopWordsRemover] with Serializable

Annotations
@Since( "1.6.0" )
object StringIndexer extends DefaultParamsReadable[StringIndexer] with Serializable

Annotations
@Since( "1.6.0" )
object StringIndexerModel extends MLReadable[StringIndexerModel] with Serializable

Annotations
@Since( "1.6.0" )
object Tokenizer extends DefaultParamsReadable[Tokenizer] with Serializable

Annotations
@Since( "1.6.0" )
object VectorAssembler extends DefaultParamsReadable[VectorAssembler] with Serializable

Annotations
@Since( "1.6.0" )
object VectorIndexer extends DefaultParamsReadable[VectorIndexer] with Serializable

Annotations
@Since( "1.6.0" )
object VectorIndexerModel extends MLReadable[VectorIndexerModel] with Serializable

Annotations
@Since( "1.6.0" )
object VectorSlicer extends DefaultParamsReadable[VectorSlicer] with Serializable

Annotations
@Since( "1.6.0" )
object Word2Vec extends DefaultParamsReadable[Word2Vec] with Serializable

Annotations
@Since( "1.6.0" )
object Word2VecModel extends MLReadable[Word2VecModel] with Serializable

Annotations
@Since( "1.6.0" )

package feature

Feature transformers

Type Members

final class Binarizer extends Transformer with HasInputCol with HasOutputCol with DefaultParamsWritable

final class Bucketizer extends Model[Bucketizer] with HasInputCol with HasOutputCol with DefaultParamsWritable

final class ChiSqSelector extends Estimator[ChiSqSelectorModel] with ChiSqSelectorParams with DefaultParamsWritable

final class ChiSqSelectorModel extends Model[ChiSqSelectorModel] with ChiSqSelectorParams with MLWritable

class CountVectorizer extends Estimator[CountVectorizerModel] with CountVectorizerParams with DefaultParamsWritable

class CountVectorizerModel extends Model[CountVectorizerModel] with CountVectorizerParams with MLWritable

class DCT extends UnaryTransformer[Vector, Vector, DCT] with DefaultParamsWritable

class ElementwiseProduct extends UnaryTransformer[Vector, Vector, ElementwiseProduct]

class HashingTF extends Transformer with HasInputCol with HasOutputCol with DefaultParamsWritable

final class IDF extends Estimator[IDFModel] with IDFBase with DefaultParamsWritable

class IDFModel extends Model[IDFModel] with IDFBase with MLWritable

class IndexToString extends Transformer with HasInputCol with HasOutputCol with DefaultParamsWritable

class Interaction extends Transformer with HasInputCols with HasOutputCol with DefaultParamsWritable

class MinMaxScaler extends Estimator[MinMaxScalerModel] with MinMaxScalerParams with DefaultParamsWritable

class MinMaxScalerModel extends Model[MinMaxScalerModel] with MinMaxScalerParams with MLWritable

class NGram extends UnaryTransformer[Seq[String], Seq[String], NGram] with DefaultParamsWritable

class Normalizer extends UnaryTransformer[Vector, Vector, Normalizer] with DefaultParamsWritable

class OneHotEncoder extends Transformer with HasInputCol with HasOutputCol with DefaultParamsWritable

class PCA extends Estimator[PCAModel] with PCAParams with DefaultParamsWritable

class PCAModel extends Model[PCAModel] with PCAParams with MLWritable

class PolynomialExpansion extends UnaryTransformer[Vector, Vector, PolynomialExpansion] with DefaultParamsWritable

final class QuantileDiscretizer extends Estimator[Bucketizer] with QuantileDiscretizerBase with DefaultParamsWritable

class RFormula extends Estimator[RFormulaModel] with RFormulaBase

class RFormulaModel extends Model[RFormulaModel] with RFormulaBase

class RegexTokenizer extends UnaryTransformer[String, Seq[String], RegexTokenizer] with DefaultParamsWritable

class SQLTransformer extends Transformer with DefaultParamsWritable

class StandardScaler extends Estimator[StandardScalerModel] with StandardScalerParams with DefaultParamsWritable

class StandardScalerModel extends Model[StandardScalerModel] with StandardScalerParams with MLWritable

class StopWordsRemover extends Transformer with HasInputCol with HasOutputCol with DefaultParamsWritable

class StringIndexer extends Estimator[StringIndexerModel] with StringIndexerBase with DefaultParamsWritable

class StringIndexerModel extends Model[StringIndexerModel] with StringIndexerBase with MLWritable

class Tokenizer extends UnaryTransformer[String, Seq[String], Tokenizer] with DefaultParamsWritable

class VectorAssembler extends Transformer with HasInputCols with HasOutputCol with DefaultParamsWritable

class VectorIndexer extends Estimator[VectorIndexerModel] with VectorIndexerParams with DefaultParamsWritable

class VectorIndexerModel extends Model[VectorIndexerModel] with VectorIndexerParams with MLWritable

final class VectorSlicer extends Transformer with HasInputCol with HasOutputCol with DefaultParamsWritable

final class Word2Vec extends Estimator[Word2VecModel] with Word2VecBase with DefaultParamsWritable

class Word2VecModel extends Model[Word2VecModel] with Word2VecBase with MLWritable

Value Members

object Binarizer extends DefaultParamsReadable[Binarizer] with Serializable

object Bucketizer extends DefaultParamsReadable[Bucketizer] with Serializable

object ChiSqSelector extends DefaultParamsReadable[ChiSqSelector] with Serializable

object ChiSqSelectorModel extends MLReadable[ChiSqSelectorModel] with Serializable

object CountVectorizer extends DefaultParamsReadable[CountVectorizer] with Serializable

object CountVectorizerModel extends MLReadable[CountVectorizerModel] with Serializable

object DCT extends DefaultParamsReadable[DCT] with Serializable

object HashingTF extends DefaultParamsReadable[HashingTF] with Serializable

object IDF extends DefaultParamsReadable[IDF] with Serializable

object IDFModel extends MLReadable[IDFModel] with Serializable

object IndexToString extends DefaultParamsReadable[IndexToString] with Serializable

object Interaction extends DefaultParamsReadable[Interaction] with Serializable

object MinMaxScaler extends DefaultParamsReadable[MinMaxScaler] with Serializable

object MinMaxScalerModel extends MLReadable[MinMaxScalerModel] with Serializable

object NGram extends DefaultParamsReadable[NGram] with Serializable

object Normalizer extends DefaultParamsReadable[Normalizer] with Serializable

object OneHotEncoder extends DefaultParamsReadable[OneHotEncoder] with Serializable

object PCA extends DefaultParamsReadable[PCA] with Serializable

object PCAModel extends MLReadable[PCAModel] with Serializable

object PolynomialExpansion extends DefaultParamsReadable[PolynomialExpansion] with Serializable

object QuantileDiscretizer extends DefaultParamsReadable[QuantileDiscretizer] with Logging with Serializable

object RegexTokenizer extends DefaultParamsReadable[RegexTokenizer] with Serializable

object SQLTransformer extends DefaultParamsReadable[SQLTransformer] with Serializable

object StandardScaler extends DefaultParamsReadable[StandardScaler] with Serializable

object StandardScalerModel extends MLReadable[StandardScalerModel] with Serializable

object StopWordsRemover extends DefaultParamsReadable[StopWordsRemover] with Serializable

object StringIndexer extends DefaultParamsReadable[StringIndexer] with Serializable

object StringIndexerModel extends MLReadable[StringIndexerModel] with Serializable

object Tokenizer extends DefaultParamsReadable[Tokenizer] with Serializable

object VectorAssembler extends DefaultParamsReadable[VectorAssembler] with Serializable

object VectorIndexer extends DefaultParamsReadable[VectorIndexer] with Serializable

object VectorIndexerModel extends MLReadable[VectorIndexerModel] with Serializable

object VectorSlicer extends DefaultParamsReadable[VectorSlicer] with Serializable

object Word2Vec extends DefaultParamsReadable[Word2Vec] with Serializable

object Word2VecModel extends MLReadable[Word2VecModel] with Serializable

Inherited from AnyRef

Inherited from Any