Sample Usage

>>> import thoth.thoth as thoth
>>> import numpy as np
>>> a_prob = thoth.prob_from_array([2,3,12,5,7])


>>> thoth.entropy(a_prob)

“Naieve” (plug-in) entropy

>>> thoth.entropy_nsb(a_prob)

NSB estimate of entropy (note that for large cardinalities, you should consider the nsb-entropy code).

>>> thoth.entropy_ww(a_prob)

Wolpert and Wolf estimate of entropy.

>>> results = thoth.calc_entropy([2,3,12,5,7], 10000)
>>> results
array([ 2.1694808 , 2.00301591, 2.33896117, 1.89535524, 2.57598497])

Bootstrap returns estimate, +/- 1 sigma ranges, +/- 2 sigma ranges; 10000 refers to the number of Monte-Carlo samples.

Jensen-Shannon Divergence

>>> results = thoth.calc_jsd([1,2,3,4,5],[5,4,3,2,1], 0.5, 10000)
>>> results
array([ 0.05104706, -0.09282951, 0.19567123, -0.2672938 , 0.28200758])

Here alpha was set to 0.5; as before, bootstrap returns estimate, +/- 1 sigma ranges, +/- 2 sigma ranges. thoth.jsd can provide the naieve JSD (when given a prob object). Note that JSD can not actually be negative—this example shows the limits of zeroth-order bias correction.

Mutual Information

>>> arrays = np.array([[12,2,3], [3,16,2], [3,2,40]])
>>> results = thoth.calc_mi(arrays, 10000)
>>> results
array([ 0.58479914, 0.45267706, 0.71551216, 0.30732072, 0.83644323])

Mutual information is symmetric, so you don’t need to worry about transposing arrays!

>>> m_prob=thoth.jprob_from_array([[12,2,3], [3,16,2], [3,2,40]])
>>> thoth.mi_nsb(m_prob)
>>> thoth.mi(m_prob)
>>> thoth.mi_ww(m_prob, 1.0/9.0)

To use the NSB, WW and naieve estimators, you will need to make “jprobs” (THOTH’s internal representation). For MI, you get to choose the beta parameter; here we have chosen the IUV (see paper).

Additional Questions & Responses

Dear Simon,

I'm a student from USC, now I'm using your THOTH estimator.
I'm now estimating the mutual information between two binary variables. Say observed samples of (X,Y) are:
(1,0), (1,0), (1,0), (0,1), (0,1), (0,1), (0,0)

How to use your bootstrap mutual information estimator?
Is it written as:
array = np.array([[1,0],[1,0],[1,0],[0,1],[0,1],[0,1],[0,0]]);
results = thoth.calc_mi(array, 10000)
or written as:
array = np.array([[1,1,1,0,0,0,0],[0,0,0,1,1,1,0]]);
results = thoth.calc_mi(array, 10000)

Another question is that if using your estimator for calculating the mutual information between two variables, is it possible to lead to negative results?

Best Regards,
Shuyang Gao


Dear Shuyang --

Glad to hear from you, and glad to hear that you're giving THOTH a workout.

The mutual information estimator takes the joint probability, not the list of samples.

So your example would be passed to THOTH as:

[[1, 3], [3, 0]]

(one observation of X=0, Y=0; three of X=1, Y=0; three of X=0, Y=1; none of X=1, Y=1).

This turns out to be a much more efficient way of representing the data, of course.

To answer your second question: zeroth order bias correction can produce negative results. I recommend looking at the Entropy paper for more on this question, including Figure 1. Higher-order corrections can ameliorate this problem, at the cost of affecting the coarse-graining properties.

Best wishes,



Hi Simon,

I've used ANACONDA for all my python packages, and for some reason this installation is not seem to be working out.

1. I have installed GSL
2. I have altered the folders in "" to the GSL installation

Still gives me an error when I "python install"

gcc -fno-strict-aliasing -I/Users/rionbr/anaconda/include -arch x86_64 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/rionbr/anaconda/lib/python2.7/site-packages/numpy/core/include -I/usr/local/include -I/Users/rionbr/anaconda/include/python2.7 -c src/nsb.c -o build/temp.macosx-10.5-x86_64-2.7/src/nsb.o
gcc -bundle -undefined dynamic_lookup -L/Users/rionbr/anaconda/lib -arch x86_64 -arch x86_64 build/temp.macosx-10.5-x86_64-2.7/src/bstrap_bc_wrap.o build/temp.macosx-10.5-x86_64-2.7/src/characterization.o build/temp.macosx-10.5-x86_64-2.7/src/prob.o build/temp.macosx-10.5-x86_64-2.7/src/prob_util.o build/temp.macosx-10.5-x86_64-2.7/src/nsb.o -L/usr/local/bin -L/usr/local/bin -lgsl -lgslcblas -o build/lib.macosx-10.5-x86_64-2.7/thoth/ -fpic
ld: library not found for -lgsl
clang: error: linker command failed with exit code 1 (use -v to see invocation)
error: command 'gcc' failed with exit status 1

Any idea of what might be?


<<< [some excitement here, not shown]


Now using gsl-1.6 (instead of 1.9) seems to work, I changed the GSL folder to:

156-56-95-122:~ rionbr$ gsl-config --libs
-L/usr/local/lib -lgsl -lgslcblas
156-56-95-122:~ rionbr$ gsl-config --cflags

But with not -L or -I in front.

[Simon note: I find THOTH works just fine with gsl-1.15 (where 1.15 > 1.6 and 1.9) and installation via MacPorts.]