Help me!

So, Mom sent me a clipping that includes this very interesting editorial about apparently little things with big impacts in education.

There's lots of intriguing stuff in there, but let me draw your focus to one alluring comment:

Black students also perform better on an exam when it is presented as a puzzle rather than as a test of academic achievement or ability, another study has shown.

Unlike most of the other cited research, this is all the detail Professor Nisbett gives about this.

Your task, should you aspire to SUPER COOLNESS: find me this study, or a citation of it, or really ANYTHING more about it. How do you present an exam as a puzzle? Did nonblack students perform better on puzzles than exams? What, in short, is he talking about here?

Richard Nisbett, the author of the editorial, has written lots of intriguing books, and has his own website, including a list of selected publications. I've looked over these, but none of them say something like "Narrowing the Achievement Gap through Assessment Restructuring", let alone "Puzzle Tests".

And thus, my first web puzzle, which I pose to you. Find me the puzzle exam study, and treasures will be yours. If you don't, the princess will be lost! Help!

Update: Waiting to hear back from Joshua Aronson, one of the "stereotype threat" researchers featured in the editorial (Nisbett forwarded my email to him.) Therefore, there is STILL TIME to submit your candidates for puzzle style examinations (Raven matrix tests seem possible) and other puzzle-based education studies. Find the research, save the world!

Update II: Aronson promptly and helpfully got back to me, with copies not only of the source for Nisbett's comment, but also another similar paper. And the results will SHOCK YOU! (Well, maybe.)

The referred paper is "The Difference Isn’t Black and White: Stereotype Threat and the Race Gap on Raven’s Advanced Progressive Matrices" by Ryan Brown and Eric Anthony Day, published in the Journal of Applied Psychology, 2006, Vol. 91, No. 4. The study contrasts, as Nisbett said, the way a test (in particular, "Raven's Advanced Progressive Matrices") is presented. When the participants were told it was an IQ test, African Americans underperformed whites. But when they were told it was a set of puzzles, and the researchers wanted the participants' opinions, African Americans performed just as well as whites. The test itself was unchanged.

The related paper is the seminal work on stereotype threat by Claude M. Steele and Joshua Aronson: "Stereotype Threat and the Intellectual Test Performance of African Americans", published in the Journal of Personality and Social Psychology, 1995, Vol. 69, No. 5. This paper is the source for many of the other results in Nisbett's editorial, including the striking result that asking participants to specify their race on a form before administering a test created an achievement gap between blacks and whites; without the race question on the pre-test questionnaire, there was no achievement gap.

Here's some answers to questions that came up (from colleagues and myself) regarding these studies:

• These were psychological studies; in a classroom setting, claiming a test is just a set of "puzzles" (that constitute part of the class grade) wouldn't have the same effect.
  
• On the other hand, if I were to present students with "math puzzles" that begin as not counting for a grade, on which students achieve well, they will surely exhort me to count them (after the fact) for a grade. This would achieve the aim of assessment disguised as non-assessment, as well as students thinking I'm super-nice and giving me good evaluations.
  
• "Raven's Advanced Progressive Matrices" are not nearly as exciting or puzzle-like as they sound. The appropriate Wikipedia page includes a link to a website that mimics these test questions; they are pattern-matching, nonverbal questions that might be familiar to you from IQ tests you may have taken in the past. Disappointingly for me, one can't present typical algebra content in this form.
  
• White students performed better (though not very significantly) on tests described as measuring their ability, while black students performed better on tests that were described as puzzles or research not meant to assess their ability. (You can imagine the white students "slacking off" when there's no pressure to achieve.) However, black students' performance on "puzzles" was as good as white students' performance when the test was presented as measuring IQ (their best category.)
  
• It seems to me the biggest take-home message is that we can, perhaps unwittingly, draw students' attention and promote their anxiety, which can have a big (negative) effect on their test performance.
  
• As Rebecca pointed out, there is a vicious assessment cycle; the people writing tests are those who did well on prior tests, and thus see nothing wrong with the conventional test structure. Students that might be otherwise good at math, but who perform poorly on tests, suffer the consequences, and do not continue in math far enough to be in a position to write different tests for the next generation.
  

This does not mean the challenge is over, although the KNOWN sources have been found. I welcome ANY citations of research on the topics of puzzle tests and games, if such things exist. (And yes, you should feel free to write your own paper on the efficacy of "puzzle tests", then send me the citation and demand a gold star.)

Click here to return to Gnomicon home page