org.apache.spark.mllib.stat

Class MultivariateOnlineSummarizer

java.lang.Object

org.apache.spark.mllib.stat.MultivariateOnlineSummarizer

All Implemented Interfaces:

java.io.Serializable, MultivariateStatisticalSummary

public class MultivariateOnlineSummarizer
extends java.lang.Object
implements MultivariateStatisticalSummary, scala.Serializable

:: DeveloperApi :: MultivariateOnlineSummarizer implements MultivariateStatisticalSummary to compute the mean, variance, minimum, maximum, counts, and nonzero counts for instances in sparse or dense vector format in a online fashion.

Two MultivariateOnlineSummarizer can be merged together to have a statistical summary of the corresponding joint dataset.

A numerically stable algorithm is implemented to compute the mean and variance of instances: Reference: variance-wiki Zero elements (including explicit zero values) are skipped when calling add(), to have time complexity O(nnz) instead of O(n) for each column.

For weighted instances, the unbiased estimation of variance is defined by the reliability weights: https://en.wikipedia.org/wiki/Weighted_arithmetic_mean#Reliability_weights.

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
MultivariateOnlineSummarizer()

Method Summary

Methods

Modifier and Type	Method and Description
MultivariateOnlineSummarizer	add(Vector sample) Add a new sample to this summarizer, and update the statistical summary.
long	count() Sample size.
Vector	max() Maximum value of each dimension.
Vector	mean() Sample mean of each dimension.
MultivariateOnlineSummarizer	merge(MultivariateOnlineSummarizer other) Merge another MultivariateOnlineSummarizer, and update the statistical summary.
Vector	min() Minimum value of each dimension.
Vector	normL1() L1 norm of each dimension.
Vector	normL2() L2 (Euclidian) norm of each dimension.
Vector	numNonzeros() Number of nonzero elements in each dimension.
Vector	variance() Unbiased estimate of sample variance of each dimension.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

MultivariateOnlineSummarizer

public MultivariateOnlineSummarizer()

Method Detail

add

public MultivariateOnlineSummarizer add(Vector sample)

Add a new sample to this summarizer, and update the statistical summary.

Parameters:

sample - The sample in dense/sparse vector format to be added into this summarizer.

Returns:

This MultivariateOnlineSummarizer object.

merge

public MultivariateOnlineSummarizer merge(MultivariateOnlineSummarizer other)

Merge another MultivariateOnlineSummarizer, and update the statistical summary. (Note that it's in place merging; as a result, this object will be modified.)

Parameters:

other - The other MultivariateOnlineSummarizer to be merged.

Returns:

This MultivariateOnlineSummarizer object.

mean

public Vector mean()

Sample mean of each dimension.

Specified by:

mean in interface MultivariateStatisticalSummary

Returns:

(undocumented)

variance

public Vector variance()

Unbiased estimate of sample variance of each dimension.

Specified by:

variance in interface MultivariateStatisticalSummary

Returns:

(undocumented)

count

public long count()

Sample size.

Specified by:

count in interface MultivariateStatisticalSummary

Returns:

(undocumented)

numNonzeros

public Vector numNonzeros()

Number of nonzero elements in each dimension.

Specified by:

numNonzeros in interface MultivariateStatisticalSummary

Returns:

(undocumented)

max

public Vector max()

Maximum value of each dimension.

Specified by:

max in interface MultivariateStatisticalSummary

Returns:

(undocumented)

min

public Vector min()

Minimum value of each dimension.

Specified by:

min in interface MultivariateStatisticalSummary

Returns:

(undocumented)

normL2

public Vector normL2()

L2 (Euclidian) norm of each dimension.

Specified by:

normL2 in interface MultivariateStatisticalSummary

Returns:

(undocumented)

normL1

public Vector normL1()

L1 norm of each dimension.

Specified by:

normL1 in interface MultivariateStatisticalSummary

Returns:

(undocumented)