Cfrtjryk

I am having trouble thinking of a function that operates on a set of samples, that is, single-valued random variables between zero and one, $x_i in (0,1), iin{1,2,...I}$, and provides a measure of "non uniformity" of the samples:

$$
F({x_i}) : [0,1]^I to R
$$
This measure would be uniquely minimized (in expectation) when the samples are from a uniform distribution.

My first reaction was to bin these samples by cutting $[0,1]$ into equal-sized peices. Then the standard deviation of the counts across the bins would be my measure, minimized when the counts in each bin are equal.

But the binning allows for "cheating", that is, the samples may not actually be uniformly distributed, but only appear that way due to the binning procedure; a different choice of bins would show the problem.

Any better ideas?

The motivation is to evaluate the quality of a CDF fitting procedure. The fitting procedure is optimal when the CDFs (different CDF for each sample) maps all the "real" samples $y_i in R$ uniformly to the [0,1] interval.

Like @Chris said, you can use some kind of divergence or information theory ideas. KL divergence like he said is good, or more generally f-divergence, etc., but more simply just use entropy. You said minimized when uniform, so entropy would be maximized, so do negative entropy. You said maps to real numbers, maybe you want for easier comparison, [0,1] interval, so use (negative) normalized entropy (divide by logN):

https://math.stackexchange.com/questions/395121/how-entropy-scales-with-sample-size

Or since you said the end goal is to compare some empirical CDF's, then depending on your problem maybe there is a more direct way you can do this like:

KS test:
https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test

Anderson-Darling:
https://en.wikipedia.org/wiki/Anderson%E2%80%93Darling_test

Cramer von Mises
https://en.wikipedia.org/wiki/Cram%C3%A9r%E2%80%93von_Mises_criterion

etc.

Take your samples and form a random variable $X$. To measure the non-uniformity of the samples measure the KL divergence between $X$ and $X'$ as $KL(X||X')$.