org.transitime.statistics

Class Statistics

java.lang.Object

org.transitime.statistics.Statistics

public class Statistics
extends java.lang.Object

Some miscellaneous statistics functions such as determining average values from a list and for determining average after filtering out of outliers from a list. Also contains simple statistical functions for determining mean and standard deviation. Could use Apache Commons math3 library instead but this calculations are so simple that seems unnecessary.

Author:

SkiBu Smith

Constructor Summary

Constructors

Constructor and Description
Statistics()

Method Summary

Modifier and Type	Method and Description
static int	biasedFilteredMean(java.util.List<java.lang.Integer> values, double fractionalLimit, double stdDevBias) Returns a filtered mean that is biased to be conservative.
static int	filteredMean(java.util.List<java.lang.Integer> values, double fractionalLimit) Gets the mean value of the values passed in.
static double	getSampleStandardDeviation(double[] values, double mean) Returns the sample standard deviation for the values passed in.
static void	main(java.lang.String[] args) Just for testing
static double	mean(double[] values) Returns the mean of the array of doubles
static double	mean(int[] values) Returns the mean of the array of ints
static int	mean(java.util.List<java.lang.Integer> values) Gets the mean values of the values passed in, without doing any filtering.
static int[]	toArray(java.util.List<java.lang.Integer> list) Converts the List to a array of ints.
static double[]	toDoubleArray(int[] array) For converting array of ints into array of doubles so that it can be used in the statistics functions.
static double[]	toDoubleArray(java.util.List<java.lang.Integer> list) For converting List of Integers into array of doubles so that it can be used in the statistics functions.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Statistics

public Statistics()

Method Detail

filteredMean

public static int filteredMean(java.util.List<java.lang.Integer> values,
                               double fractionalLimit)

Gets the mean value of the values passed in. Filters outliers that are less than minPercentage or greater than maxPercentage of the average of all values.

Parameters:

values - The values to be averaged

fractionalLimit - If value is further away than fractionalLimit of average it is filtered out if there are more than 2 data points. Expressed as a fractional value between 0.0 and 1.0 such that a value of 0.0 means no data points are filtered out and a value of 1.0 means 100% of the datapoints are filtered out. Therefore if one uses a value of 0.5 than values below 50% or above 200% of the average are filtered out. A value of 0.7 is likely reasonable since that would filter out values below 70% or above 143% (1/0.7) of the average.

Returns:

The average value, after outliers have been filtered

biasedFilteredMean

public static int biasedFilteredMean(java.util.List<java.lang.Integer> values,
                                     double fractionalLimit,
                                     double stdDevBias)

Returns a filtered mean that is biased to be conservative. First, loops through the data to determine the average and to throw out any data points that are outliers. This is important since outliers are not part of a normal distribution since they represent non-stochastic data (such as a breakdown, accident, etc). Then instead of just returning the average, the remaining datapoints, as long as there are at least data points remaining, are considered to be of a normal distribution and the value that is returned is mean - stdDevBias * standardDeviation. If stdDevBias is set to 0.0 then the average is returned. If stdDevBias is 1.0 then the value returned is 1.0 standard deviations below the average, which means that 68% + (100% - 68%)/2 = 84% of the time the actual value would be at least as large as the value returned by this method.

This method is useful for when need to be somewhat conservative, such as when determine expected stop time at a terminal. For such a situation want to take into account when vehicle really leaves the terminal so want to determine the average time. But it is really problematic if the time varies which could mean that the passenger misses the vehicle.

Parameters:

values - The values to be averaged

fractionalLimit - If value is further away than fractionalLimit of average it is filtered out if there are more than 2 data points. Expressed as a fractional value between 0.0 and 1.0 such that a value of 0.0 means no data points are filtered out and a value of 1.0 means 100% of the datapoints are filtered out. Therefore if one uses a value of 0.5 than values below 50% or above 200% of the average are filtered out. A value of 0.7 is likely reasonable since that would filter out values below 70% or above 143% (1/0.7) of the average.

stdDevBias - How much to bias the mean. For a normal distribution (after outliers filtered out) using a value of 1.0 standard deviation means that will return mean - stdDevation. A value of 0.0 would return the mean.

Returns:

The mean value, after outliers have been filtered and the average is biased down by specified stdDevBias with respect to the standard deviation.

toArray

public static int[] toArray(java.util.List<java.lang.Integer> list)

Converts the List to a array of ints. This can be useful when amount of memory used is critical.

Parameters:

list -

Returns:

toDoubleArray

public static double[] toDoubleArray(int[] array)

For converting array of ints into array of doubles so that it can be used in the statistics functions.

Parameters:

array -

Returns:

toDoubleArray

public static double[] toDoubleArray(java.util.List<java.lang.Integer> list)

For converting List of Integers into array of doubles so that it can be used in the statistics functions.

Parameters:

list -

Returns:

mean

public static int mean(java.util.List<java.lang.Integer> values)

Gets the mean values of the values passed in, without doing any filtering.

Parameters:

values - The values to be averaged

Returns:

The average

mean

public static double mean(double[] values)

Returns the mean of the array of doubles

Parameters:

values -

Returns:

the mean, or NaN if there is no data

mean

public static double mean(int[] values)

Returns the mean of the array of ints

Parameters:

values -

Returns:

getSampleStandardDeviation

public static double getSampleStandardDeviation(double[] values,
                                                double mean)

Returns the sample standard deviation for the values passed in. The "sample" standard deviation is used because only looking at a subset of data and the sample size is expected to be relatively small. To determine the "sample" standard deviation the variance is determined by dividing by N-1 instead of N. Since most clients will also need the mean it is passed in to this method so that it is not calculated twice.

Parameters:

values -

mean -

Returns:

the sample standard deviation. Returns NaN if sample size is 1.

main

public static void main(java.lang.String[] args)

Just for testing