stat

Value Members

1. object ChiSquareTest

   

   :: Experimental ::

   :: Experimental ::

   Chi-square hypothesis testing for categorical data.

   See Wikipedia for more information on the Chi-squared test.

   Annotations

   @Experimental() @Since( "2.2.0" )

2. object Correlation

   

   API for correlation functions in MLlib, compatible with DataFrames and Datasets.

   API for correlation functions in MLlib, compatible with DataFrames and Datasets.

   The functions in this package generalize the functions in org.apache.spark.sql.Dataset#stat to spark.ml's Vector types.

   Annotations

   @Since( "2.2.0" ) @Experimental()

3. object Summarizer extends Logging

   

   Tools for vectorized statistics on MLlib Vectors.

   Tools for vectorized statistics on MLlib Vectors.

   The methods in this package provide various statistics for Vectors contained inside DataFrames.

   This class lets users pick the statistics they would like to extract for a given column. Here is an example in Scala:

   import org.apache.spark.ml.linalg._
   import org.apache.spark.sql.Row
   val dataframe = ... // Some dataframe containing a feature column and a weight column
   val multiStatsDF = dataframe.select(
       Summarizer.metrics("min", "max", "count").summary($"features", $"weight")
   val Row(Row(minVec, maxVec, count)) = multiStatsDF.first()

   If one wants to get a single metric, shortcuts are also available:

   val meanDF = dataframe.select(Summarizer.mean($"features"))
   val Row(meanVec) = meanDF.first()

   Note: Currently, the performance of this interface is about 2x~3x slower then using the RDD interface.

   Annotations

   @Experimental() @Since( "2.3.0" )

4. package distribution