In an article about the legal defenses for a drug possession charge, Jon Katz mentions a case in which a suspect was caught with ten pills, all allegedly methadone. The state had a chemist test one of the pills to determine it was methadone, then the chemist testified that the other pills looked similar. This was enough to convict, and the decision quoted favorably an earlier ruling that

"Random sampling [of controlled dangerous substances] is generally accepted as a method of identifying the entire substance whose quantity has been measured." 

Katz argues:

The chemist had the alleged drugs available to test; it is not too much to insist that a possession with intent to distribute conviction for methadone be precluded without testing each pill, or at least over half the pills.

My prob-and-stats classes were long ago, but I'm going to try to figure out the math behind sampling a bunch of pills. I think I can get some useful results fiddling with the hypergeometric distribution function in Excel. (Although I could certainly be screwing this up.)

I believe the situation in the methadone case is similar to sampling the output of a production process and trying to set an upper limit on the defect rate, where a defect is equivalent to a pill that is not actually methadone.

For example, if our production process manufactures a batch of 1000 lightbulbs and we sample 5 random bulbs to test, and none of them are defective, can we conclude that the entire batch has no defects? What if we test 100 bulbs?  500?  999?

The short answer in all these cases is no. As long as we do not test every bulb, there is always a chance that one of the untested bulbs is defective. And there is always the chance that one of the defendant's pills is not methadone unless we test them all.

So, before we can answer questions about the scientist's methadone tests, we need to answer a policy question: How confident do we need to be in our results?

It would be easy to say we need 100% perfect confidence, but that would be hyperbole, not reality. 100% confidence implies an error rate of exactly zero, which is unrealistic. We routinely accept larger risks, such as the inherent error rate of the test for methadone or the risk of assigning a lab sample to the wrong case. We need to pick a more realistic number.

They say it's better for 10 guilty people to go free than to convict one innocent person, which kind of implies about a 90% confidence would be acceptable. Let's use that to keep the numbers simple.

So, if the scientist tests 1 pill out of a batch, and it's methadone, what percentage of the pills can he conclude are methadone with 90% confidence? Well, 1 sample is a degenerate case: He can only conclude that 10% of the pills are methadone. Since he has actually sampled 1 of the 10 pills, he already has empirical evidence that 10% of them are methadone. Statistical sampling is useless in this case.

How many samples would he need to be 90% confident that more than 90% of the pills---i.e. all of them---are methadone? He would need to sample more than 90% of them---i.e. all of them. Again, statistical sampling is useless.

The problem here is that a total population of only 10 pills is an absurd use of statistical sampling. A better use would be if you recovered a bag of 300 little glass bottles which you suspected to contain crack, and you wanted to know if your suspect had enough crack to reach the sentencing enhancement at 100 bottles.

How many straight successful tests for crack in a bottle would be necessary to ensure with 90% confidence that at least 100 of the bottles held crack? It only takes about 3. If you tested 13 straight bottles and all of them were positive for crack, you could assert with million-to-one confidence that at least 100 of the bottles in the batch were crack.

(It's important that the sampling be truly a random sample of the bottles. Merely taking the first 13 could lead to mistakes if the initial bottles differ from the others in some way.)

So what was really going on in the Williams case? Well, the court's opinion that the scientist was using random sampling is nonsense. As we've seen, random sampling of a single pill out of 10 tells him nearly nothing.

However, one of the crucial assumptions behind these statistical calculations is independence: The calculations assume that the contents of any pill (or bottle) is in no way related to the contents of any other pill (or bottle). This is often not the case in the real world. If you find an unlabled bottle of identically shaped and colored pills in your medicine cabinet, and you see that one of them has "aspirin" written on it, you can fearlessly conclude that you have a bottle of aspirin.

That works because you assume someone (the manufacturer, the pharmacist, you) put all the pills in the same bottle because they are the same kind of pills. The prosecutor in the drug case just asked the court to infer the same thing about the defendent's pills, which it did.

The court apparently made this inference based on common sense. However, the prosecution also used testimony from a police detective whom the court described as "an expert in the packaging, use, and distribution of narcotics." I'm sure he would have been willing to back up the claim that drug users and distributors don't usually mix pills.

(Lord knows, he testified to all kinds of other amazing things. It's a wonder these police drug experts aren't just allowed to testify that it would be very unusual for a young black or hispanic male to be standing on a street corner for reasons other than selling illegal drugs... Sheesh.)

In short, the scientist's sole contribution was identifying the one pill as containing methadone. The rest was inference by the fact finder, which did not involve statistics in any meaningful way.