statistical test question

MangoJ

Well-Known Member
#1
I'm not confident enough with statistical tools, but I'm looking for suitable statistical test. As there are many profound people here, I ask for advice.
My original problem is different from the one below, but I will use the game of roulette to give an accurate example of what test I'm looking for.

So, let's say I have a roulette game where I believe the wheel is biased, and the bias will depend on an external parameter A. I cannot control this parameter which is different each time, but I can observe the parameter before each acting on each spin. (Such a parameter could be the the wheel's last result, the choice of ball or whatever).
Let's also say that I have a model for the roulette game incorporating this parameter A, and it will give me probabilities of each result depending on this parameter.

My question would be: How can I test whether my model does in fact produce the correct probabilities for this game, depending on the observation of both the external parameter and the outcome of the game ?
Let's say a first test of simple correlation between the parameter and the outcome turned positive significantly. But how do I quantify the quality of the model ?
 

assume_R

Well-Known Member
#2
K-S Divergence, or Chi Squared

MangoJ said:
I'm not confident enough with statistical tools, but I'm looking for suitable statistical test. As there are many profound people here, I ask for advice.
My original problem is different from the one below, but I will use the game of roulette to give an accurate example of what test I'm looking for.

So, let's say I have a roulette game where I believe the wheel is biased, and the bias will depend on an external parameter A. I cannot control this parameter which is different each time, but I can observe the parameter before each acting on each spin. (Such a parameter could be the the wheel's last result, the choice of ball or whatever).
Let's also say that I have a model for the roulette game incorporating this parameter A, and it will give me probabilities of each result depending on this parameter.

My question would be: How can I test whether my model does in fact produce the correct probabilities for this game, depending on the observation of both the external parameter and the outcome of the game ?
Let's say a first test of simple correlation between the parameter and the outcome turned positive significantly. But how do I quantify the quality of the model ?
The best and simplest would be to use a paired student's t test but that assumes you have one set of single measurements (without "A"), and another set of single measurements (with "A").

But that's not what you have. You have 2 distributions which you want to compare. These websites will show you a few different tests you can use: http://www.mathwave.com/articles/goodness_of_fit.html , http://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test

Remember, though, that cumulative distributions means that you sum up to the value. So for example, if your distribution is:
00: 2.4%
0: 2.8%
1: 3%
2: 4%
3: 2%
...

your empirical cumulative distribution is:
00: 2.4%
0: 5.2%
1: 8.2%
2: 12.2%
3: 14.2%
...
36: 100%

The tests in the websites will take a "reference distribution" which you claimed to have, and see if your empirical (measured) distribution is significantly different. The more samples you have the more accurate the tests!!!

Hope this helps, and you can probably find excel templates online which will help you compute all those tests listed above!
 

MangoJ

Well-Known Member
#3
The Kolmogorov–Smirnov test looks promising on reading, but I'm not sure I understand the test well enough. As I read it, one can test (independent and identical distributed) samples against a given probability distribution. Then there will be a coefficient tabulated somewhere, telling how (un)likely the samples fits the probability distribution.

So far so good, I'm not sure but my problem seems slightly different: If my model were correct, each sample would follow a different probability distribution, so I don't have i.i.d. samples available. Maybe a better illustration (better than Roulette) is the performance of a sports team. I might have a model for the probabilities of the number of goals in the team's next game, depending on parameters like their opponent or environment conditions. My only sample is the actual game, which is never repeated. Of course a single sample will tell me nothing, but if the team is playing a whole season I have more samples to test my model - however I need to test each sample against a different distribution.

Maybe I should just simulate bets according to the models predictions, and see if I get consistent profits according to the sample results.
 

assume_R

Well-Known Member
#4
Well, the K-S test says if your given distrubution fits an "expected" distribution.

For the sports bet I'm not actually sure which test would be best. Could you give a more specific example with numbers? Or something? I could better help then.
 

MangoJ

Well-Known Member
#5
Ok, let's do a (ridiculous wrong ^^) model of a fictitious soccer team. My model prediction is, that my team will score 0/1/2 goals with probabilities 20%/30%/50% if they are the Home team, and with probabilities 60%/20%/20% if they are the Away team.

In such a simple scenario I would sample all games they play in a given season, sort them by Home and Away team, and apply the K-S test to each of those two subsets of samples.

But what if the possible parameters of the models are not simply Home or Away, say the model probability estimates depend on the referee's phone number. This way I cannot categorize those games, and test each category with K-S against each subsets. There will be likely more subsets than samples available.

My naive test would be to just "bet" always on zero goals on each game for the season, with model-implied payouts (i.e. 5to1 if model probability is 20%). If I break even regardless which result I chose beforehand, then the model would seem reasonable accurate.

Maybe there is a more systematical test available, maybe there isn't. Not much of a problem..
 

assume_R

Well-Known Member
#6
MangoJ said:
Ok, let's do a (ridiculous wrong ^^) model of a fictitious soccer team. My model prediction is, that my team will score 0/1/2 goals with probabilities 20%/30%/50% if they are the Home team, and with probabilities 60%/20%/20% if they are the Away team.

In such a simple scenario I would sample all games they play in a given season, sort them by Home and Away team, and apply the K-S test to each of those two subsets of samples.

But what if the possible parameters of the models are not simply Home or Away, say the model probability estimates depend on the referee's phone number. This way I cannot categorize those games, and test each category with K-S against each subsets. There will be likely more subsets than samples available.

My naive test would be to just "bet" always on zero goals on each game for the season, with model-implied payouts (i.e. 5to1 if model probability is 20%). If I break even regardless which result I chose beforehand, then the model would seem reasonable accurate.

Maybe there is a more systematical test available, maybe there isn't. Not much of a problem..
Okay, so with so few samples (small sample size!) versus subsets, you will be hard pressed to find any worthwhile statistical test. What you can do is determine if the results are the expected 20%/30%/50% distributions with some degree of accuracy. But when you only have 1 or 2 samples per "set" of distributions, small sample size says you won't find any statistical significance outside of expected variance.
 
Top