Big Myths about Knowledge and Skills

After much experience in the testing industry, I have come to believe that we have embraced some false conceptions with respect to the concepts of knowledge and skills. With the desire of improving how we talk about and measure knowledge and skills, I would like to share three myths with you.

1. We are able to compare the concepts of knowledge and skill in a meaningful way.
2. Knowledge is less important than having skills; knowledge is passé or something relatively useless in the world we live.
3. Multiple-choice questions only measure knowledge. If I use them in a test, my test will be silly and people will criticize it.

Myth #1: We are able to compare the concepts of knowledge and skill in a meaningful way.  Knowledge is included in the general concept of KSA’s (Knowledge, Skills and Abilities) as a popular way to organize what we are trying to measure with a test. Loosely defined, knowledge is a collection of what we have learned since we were babies. It is stored in our brains. The ability to retrieve knowledge quickly when we need it is an important skill, and not always easy to do. Cognitive scientists have studied the factors that affect a person’s ability to recall stored information in a given situation (e.g., anxiety, distractions, relevant cues) for more than a century. Because retrieval of knowledge is a skill, what we need to do is simply lump that skill with any other skills we are trying to measure in a test, thereby removing the false distinction.

This is the most important of the three myths, because the separation of knowledge and skills has contributed to the development of the next two myths and a lot of wasted time and effort.

Myth #2: Knowledge is less important than having skills; knowledge is passé or something relatively useless in the world we live. Retrieving knowledge is considered by many to be a “lower-order” thinking skill, although this is not supported by cognitive science research and theory. I’ve heard the statement often from people that extol the virtues of so-called performance tests that “knowledge is not as important as doing”. It’s a thought expressed in marketing literature and blogs. This statement is supported by rather absurd comparisons such as: (1) To know that postage stamps go on letters in order to pay for their delivery is not as important as being able to launch a nuclear missile when the time comes, or (2) Knowing that 3+3=6 is not as important as being able to drive a car. Can you see how easily a fallacious comparison may be created? And false comparisons apparently still work for some when the value is obviously reversed: Knowing that a copy of Assassins Creed for the Xbox One costs $59 is not as important as being able to dial a rotary phone.

It only takes a little bit of introspection for each of us to realize that without our knowledge, along with our ability to retrieve it, we would be functionally helpless in this world. No amount or quality of other skills would save us. This same introspection provides the explanation: that it is impossible to separate the efficient retrieval of knowledge from the performance of any other skill, the latter being so completely dependent on the former.

The criticism of knowledge then naturally leads to the criticisms of popular methods of measuring it.

Myth #3:  Multiple-choice questions only measure knowledge. If I use them in a test, my test will be silly and people will criticize it. Okay, the last 5 words are true because critics are everywhere, but the rest is nonsense. With only a slight effort, I can easily provide hundreds of examples of multiple-choice questions that assess cognitive processes other than the retrieval of knowledge, and multiple-choice questions that reliably measure the performance of overt behavioral skills. I believe that most of those who hesitate to use multiple-choice question varieties don’t understand what they can do and are overly-influenced by the current political climate critical of their use. The critics, almost without exception, have something to sell. There is simply too much objective, scientific research as well as a mountain of evidence from operational testing programs to justify the appropriate use of multiple-choice questions in every testing program in the world.

Now, there is no doubt that multiple-choice questions, along with any other question types we can use in a test, have been used improperly at times, perhaps too often. I believe it is this improper use that has fueled the more generalized criticisms. It would be proper and appreciated if a neighbor would criticize me if I ever used, or kept using, a garden rake to vacuum my living room carpet. But he would be wrong to tell me and everyone else that the garden rake is a bad tool. When creating tests, we often use the wrong tool. And sometimes we don’t know it’s the wrong tool because we don’t really know what skill we are trying to measure. It is the easiest thing in the world, almost the default situation, to create a test not knowing what skills we are trying to measure and then throw items at it because they are what “we have always used” (e.g., multiple choice and such) or because they are “innovative” (e.g., simulations and such). Even if you enjoy taking huge risks, the end result will almost surely be a disaster.

It is time for us to put away unsupported, unhelpful and damaging fads like the surface-level comparison of knowledge and skill. They lead directly to the myths above, and to lengthy and expensive wrong paths.