Do you trust your simulator
- Thread starter psyduck
- Start date
Although my instincts told me that I was getting legitimate results with CVCX, since I had never designed a strategy before, I began having doubts that perhaps I just got lucky. I ran the simulation on several other machines, and was beginning to feel more confident when my results were within 0.01-0.02% IBA for any given count, and the SCORES were within 1-2 cents for any given penetration when comparing the results. Then a fellow lister on this forum ran my UBZ strategy and was able to reproduce my results.
Although I had confidence in CVCX before, I trust it completely now. Can't speak to the reliability of the other simulators, but CVCX is the real deal.
Although I had confidence in CVCX before, I trust it completely now. Can't speak to the reliability of the other simulators, but CVCX is the real deal.
CVData and CVCX are tops. Don't leave home without them. Well...better to leave them at home...but you know what I mean.
Still the question of "sim trust" is interesting. We run sims of 100's of millions of hands to understand EV, and yet none of us will live long enough to play that many hands. So, even if the sim is telling the "truth" in the long run, we will not live to see the long run, and therefore our results can't be reliably compared with what comes out of the sim. So is the sim right, and can we trust it? How would we know??
Which then brings us to the question...what is the value of simulation if it can't reliably tell us what results we can individually expect. I think the answer is that the sim gives us confidence to keep playing, even when we are losing big time, because we can know that there is no better strategy than the one we are following.
Still the question of "sim trust" is interesting. We run sims of 100's of millions of hands to understand EV, and yet none of us will live long enough to play that many hands. So, even if the sim is telling the "truth" in the long run, we will not live to see the long run, and therefore our results can't be reliably compared with what comes out of the sim. So is the sim right, and can we trust it? How would we know??
Which then brings us to the question...what is the value of simulation if it can't reliably tell us what results we can individually expect. I think the answer is that the sim gives us confidence to keep playing, even when we are losing big time, because we can know that there is no better strategy than the one we are following.
hmm i dunno about a FAQ.
thing is doesn't the answer simply reside in an understanding of what EV, SD and ROR is with respect to actual results along with an understanding of why a simulation needs enough data to be statistically significant.
then it's just the idea that a simulation gives us a range of results we might expect with varying degrees of confidence, and a means by which we can gauge degrees of advantage expected relative to the count, sort of thing.
:whip:
thing is doesn't the answer simply reside in an understanding of what EV, SD and ROR is with respect to actual results along with an understanding of why a simulation needs enough data to be statistically significant.
then it's just the idea that a simulation gives us a range of results we might expect with varying degrees of confidence, and a means by which we can gauge degrees of advantage expected relative to the count, sort of thing.
:whip:The only way I found errors in my simulator was to do the same sim but using different ways to set it up. When the result was different just because of a different setup, then you knew something was wrong.
Other than that, I have no other way to verify the result. Therefore I do not easily trust or distrust a sim. Even Microsoft programs have bugs. Humans do make errors when they make things. Without a benchmark to compare to, I think it is hard to "test" a simulator.
Other than that, I have no other way to verify the result. Therefore I do not easily trust or distrust a sim. Even Microsoft programs have bugs. Humans do make errors when they make things. Without a benchmark to compare to, I think it is hard to "test" a simulator.
I found running hands manually (i.e. enter them one by one for testing) was pretty effective, especially borderline cases or any weird hand combinations. But you're right, you do have to test it thoroughly.
Another way is to build a database of hands with known correct plays, feed them into the sim, and make sure they're being played correctly. Sort of an automated version of the above.
Another way is to build a database of hands with known correct plays, feed them into the sim, and make sure they're being played correctly. Sort of an automated version of the above.
Hmmm . . . . good point. It also raises the profound question of the benefit of knowing the EV, and what good that does? It seems to me that knowing the long term EV of a ruleset is essential in that it provides a basis for comparing different games OTT - and we can decide on the basis of that which ones we will play, or not. It can also be a basis for setting the parameters for when the alarm bells go off - if losses fall outside of 3 or perhaps 4 Std Devs, perhaps something is wrong? But even if you know the EV for a game it won't have any bearing on results.
Take the case of two players - one has just started blackjack and is about to visit the casino for the first time, and the other is a long term player with two million hands under this belt. Both play the same game and with perfect basic strategy. Both lose $1,000 playing the same number of hands using the same bet. The only difference I can see is that for one his loss will represent relatively high variance (relative to the other player), being a proportion of his total bets made during that first session, and for the latter it will represent relatively insignificance variance, and probably leave his long term results hovering around the EV line, when compared to the total amount he has bet over his lifetime of play. But they're both a grand out of pocket for that session, although the overall results of one is nearer to the EV than the other.
So what?
Take the case of two players - one has just started blackjack and is about to visit the casino for the first time, and the other is a long term player with two million hands under this belt. Both play the same game and with perfect basic strategy. Both lose $1,000 playing the same number of hands using the same bet. The only difference I can see is that for one his loss will represent relatively high variance (relative to the other player), being a proportion of his total bets made during that first session, and for the latter it will represent relatively insignificance variance, and probably leave his long term results hovering around the EV line, when compared to the total amount he has bet over his lifetime of play. But they're both a grand out of pocket for that session, although the overall results of one is nearer to the EV than the other.
So what?
The most common errors in simulations starting with the worst:
1. Poor random number generator
2. Unrealistic true count calculation
3. Not correctly following the specified strategy
4. Errors handling splits
5. Not handling out of cards condition
6. Not handling seen vs. unseen card in pitch games correctly
7. Not handling cover play adequately
8. Implementation of rule variations
As you add features, of course possible errors multiplies dramatically. Simulation code is usually written very tightly making it prone to error when variations are added. That is, you may test a new feature and see it work, failing to see that you broke another combination of features.
Testing functionality not present in other simulators is very difficult as there is no back of the book to look up the answer. This is further complicated as, unlike combinatorial analysis, there is no exact answer.
1. Poor random number generator
2. Unrealistic true count calculation
3. Not correctly following the specified strategy
4. Errors handling splits
5. Not handling out of cards condition
6. Not handling seen vs. unseen card in pitch games correctly
7. Not handling cover play adequately
8. Implementation of rule variations
As you add features, of course possible errors multiplies dramatically. Simulation code is usually written very tightly making it prone to error when variations are added. That is, you may test a new feature and see it work, failing to see that you broke another combination of features.
Testing functionality not present in other simulators is very difficult as there is no back of the book to look up the answer. This is further complicated as, unlike combinatorial analysis, there is no exact answer.
Ideally you should verify the sim results mathematically to make sure it is working properly and giving accurate output. For example, calculate the frequencies of each starting hand total and compare that to the output of the sim program. If your program does not have the correct distribution of hands then you know something is wrong with your dealing procedure (or the deck, or the shuffle, or the hand total calculation, etc.). Then compare the final hand frequencies to your sim output. If the output is not accurate then there is a problem with the playing strategy, hand total calculation, or something like that. Then check the EV and variance (from a combinatorial analysis) with the sim output. Again, you should be able to find any major problems very quickly.
Another way is to have the computer cycle through every possible hand and check the results manually to verify that it is making the proper plays and payouts. This brute force approach can work for simpler games like Roulette or 3CP, but for something like BJ it is much easier to verify your sim program mathematically.
So I agree with you – If you do not properly debug your sim program then you should not trust it’s output. Sometimes writing the program is the easy part.
-Sonny-
Another way is to have the computer cycle through every possible hand and check the results manually to verify that it is making the proper plays and payouts. This brute force approach can work for simpler games like Roulette or 3CP, but for something like BJ it is much easier to verify your sim program mathematically.
So I agree with you – If you do not properly debug your sim program then you should not trust it’s output. Sometimes writing the program is the easy part.
-Sonny-
I was under the impression that you created the simulator and tested it against another similar program. If you got it from someone else then it should have been properly tested already. If you are getting inconsistent results then you should contact the developer. It is very easy to make a mistake setting up the parameters of the simulation, which can account for a big difference between two different programs. It is also very easy for the developer to make a mistake in the code. I’m sure the developer can figure out what went wrong if you explain the problem to them.
-Sonny-
-Sonny-