Item exposure during an exam in the testing world is often viewed as a bad thing, because it seems obvious that item exposure leads to item over-use which in turn leads to item compromise. It is common for psychometricians to limit item exposure, defining it as either a too-high absolute number of presentations of the items in a test, or a too-high rate of the items presented on tests. Unfortunately, there is no scientific research or even unscientific guidelines, or even reasonable casual suggestions, about how many exposures are too many, or which rate of exposure is too high.
It does not follow that item exposure is the same as item compromise. In fact, I’ve seen items compromised with an extremely small number of presentations. Some items have even been compromised prior to the first test being administered!
In my opinion, the notion that item compromise results from item exposure—as defined above—leads to improper conclusions, decisions, and ineffective procedures. I have a few reasons for this opinion, a couple of which I’ll give here. First, item exposure is absolutely necessary. It is obvious that no test can be effective unless its items are exposed during the exam. Test designers even let examinees view an item multiple times encouraging them to return to and review previous items again and again. Second, item compromise has very little to do with the definitions of item exposure given above. Consider this simple example: Suppose that an item was shown to one million test takers and was presented on every exam administered. This would be considered a very high number of exposures along with a 100% exposure rate. But, suppose that none of those examinees were able to share the item with others. In this simple example, the item remains uncompromised and perfectly secure, and can be continued to be used on the exam.
If we wish to reduce item compromise, the example illustrates that limiting the number of presentations or rate of presentations of an item is not as important as the methods used to secure the items, to protect them from theft, and to keep them from being used for cheating. For this reason we need improved item security, which means better ways to keep items from being stolen and used for cheating on subsequent exams. We need methods to detect when an item is truly compromised and then immediately to take it out of service. Instead, we often see stubborn adherence to a century-old model of relatively unsecure test administration, and believing that keeping an item from being presented on a test is a sensible way to secure it.
It is certainly possible to improve the way we secure items. As examples, there are protective item and test designs available, and certainly better test monitoring procedures, that we can use. And perhaps we can learn a little from other industries as well. Consider the problem with the theft of music over the Internet. No one would suggest that music is stolen because it was listened to by too many people. Instead, we see serious efforts to protect the music, to keep it from being stolen, to detect when it is stolen, and to punish those that are responsible. We should be doing the same.
We welcome comments below!
Dave,
Thanks for this article. You raise a very important topic and make a number of really good points. Compromise and exposure are definitely not the same thing. As you point out, compromise is not necessarily dependent on exposure (e.g., an item can be compromised without having been exposed). However, even under the best of circumstances, it’s also not independent of exposure. Therefore, while I agree with your fundamental premises that (a) compromise rules based on exposure rates are arbitrary, (b) we need to develop tools to allow detection of items when they are compromised, and (c) we need to use better methods to keep items from being stolen, as of yet, such tools do not exist to an adequate degree. Until we are better positioned to prevent item harvesting, I do believe that limiting item exposure is a reasonable strategy towards mitigating the effects of item preknowledge.
Thanks for your comments, Jim. We very much appreciate your expertise on this subject.