When Dice Attack

I once sat down to do the math on how many rolls would be needed for a proper distribution of rolls with a d20. And by “did the math” I meant “plump formulas into Excel until the numbers looked right”. Even when generating a few hundred rolls, the average could be swayed by small random spikes in local probability: with 200 rolls you expect 10 rolls of each number, so a single extra roll is a 10% variation. It was several thousand rolls before the numbers hit a reliable average. I’m not sure how reliable Excel is as a random number generator, but it is bound to be as accurate as a every so slightly warped or imbalanced d20. 

Now, assuming you roll a d20 an average of five times per combat encounter and extra five times between combat encounters. With 10 encounters per level, this means 100 rolls per level, and 2000 rolls over a course of a level 1 to 20 campaign. If you have a single d20 that is used for all of those rolls, it will take the duration of an entire campaign to have an accurate gauge of its probability, to tell if the dice is imbalanced, rolling cold, of if you simply suck at D&D. A sample size smaller than a single campaign is too short, with too few rolls, to accurately gauge the probability of a die.