Judge Grimm, Sampling and eDiscovery Process Quality

Statistics for eDiscovery: Judge Grimm And Others Recommend Using Sampling To Evaluate The Quality Of Our eDiscovery Process. How Do We Do That?

There are a number of sampling techniques that can be used for quality control. Here is one method that can be used to determine whether you have missed an unreasonable number of responsive documents. A highly accurate system will correctly identify most of the responsive documents. Therefore, there should be few documents that are incorrectly identified as nonresponsive. A quality measure can address the question: Of the documents that have been marked nonresponsive, what percentage of those documents is actually responsive? This measure is called “elusion,” reflecting the number of documents that have eluded identification.

Rather than estimate this value directly, elusion leads to a quality control measure with a very simple statistical test to determine whether the elusion rate exceeds a reasonable criterion. What percentage of documents could you leave behind and still find your process to be reasonable? In the Roitblat, et al. study (2010), using the original review as a standard, two new review teams showed elusion rates of around 6 – 7% (Team A: 6.6%, Team B: 6.3%). In other words, of the documents identified as nonresponsive by Team A, 6.6% of them had been identified as responsive by the original review.

To use the elusion test, we need to specify a level of confidence and a criterion percentage, called “prevalence.” We want to say that responsive documents were no more common in the set identified as nonresponsive than our specified prevalence rate. To do that, we draw a random sample of documents from those that have been identified as nonresponsive. The size of the sample depends on our prevalence rate and desired confidence level. If none of the documents in this sample is found to be responsive, then we can say with confidence that there were no more responsive documents left behind than the prevalence rate. With 95% confidence and a prevalence of 0.5%, you would have to review 598 documents. If none of the documents in the sample was responsive, then you could say with 95% confidence that no more than 0.5% of the documents we rejected were actually responsive.

Source:  Statistics For eDiscovery (PDF) - Used by Permission of Herbert L. Roitblat, Ph.D.

Series:  Statistics for eDiscovery – Blog Posts – Orange Legal Technologies

Orange Legal Technologies helps corporate legal departments and their outside counsel conduct the critical electronic discovery task of document review by providing advanced predictive coding technologies and expert reviewer assistance to accelerate the electronic discovery process. To learn more about our automated predictive review services, click here.

This entry was posted on Friday, February 10th, 2012 at 11:24 am. It is filed under metadata, resources and tagged with resources, services. You can follow any responses to this entry through the RSS 2.0 feed.