Global test scores don’t tell it all
Norman Matloff, Star Tribune Commentary, December 10, 2013 – There was a lot of manufactured hand-wringing last week about the middling performance of American 15-year-olds on a global measure of reading, mathematics and science skills. Yet if we look at the scores on the Program for International Student Assessment (PISA) in the context of another international competition, we get a clearer picture.
Every year the Association for Computing Machinery holds its International Collegiate Programming Contest. In the old days, U.S. teams consistently dominated the event. Recently, however, the top teams have tended to be from Asia and the former Soviet-bloc nations. Jiaotong University, or Jiaoda, in Shanghai — the same city whose students topped this year’s PISA rankings — has been especially strong, winning gold medals on several occasions.
In other words, it would appear that not only are Shanghai’s 15-year-olds sharper than their American peers, but Shanghai’s geeks also are smarter than our geeks. So, the sky is falling, right? Appearances are deceiving, it turns out.
The Jiaoda contestants are essentially student-athletes, spending all their time training for the event, according to a Jiaoda public information officer, Xu Jun. And the skills needed for the competition are indeed trainable. Although the problems posed each year are unique, their solutions usually fall into a handful of mathematical patterns.
This gives a huge benefit to those who can devote themselves to full-time, year-round practice. By contrast, most top U.S. computer-science students have better things to do with their time, including founding start-ups that might become billion-dollar companies.
The PISA results appear to be analogous. In China, test-taking dominates education, and preparation for the zhong kao (high school entrance exam) and gao kao (for admission to university) reaches Herculean levels. Thus the Shanghai students were well-practiced for the PISA. The PISA questions are good ones, but, as in the ACM programming contest, they boil down to a few patterns.
In the United States, a complicating factor is socioeconomic class. A 2001 analysis by Arizona State University’s David Berliner found that test scores in states such as Utah, Iowa and Nebraska, which don’t (yet) have a large underclass, are similar to those of the top Asian countries. A 2011 compilation by the Atlantic of the PISA data showed that the white (read middle-class and up) students in Washington, D.C., scored much higher in math than did students in Hong Kong and Taiwan, two of the highest international scorers. A forthcoming Economic Policy Institute study shows that after socioeconomic data are accounted for, the adjusted ranking for the United States is much higher than the raw scores indicate.
Critics of such analyses point out that kids in some poor countries, such as Vietnam, do well on the PISA. A Chinese University of Hong Kong study found that the 2006 PISA data “suggests that Hong Kong students perform equally well regardless of their socioeconomic background.” The difference is that in Asian countries, even impoverished students tend to be highly motivated in school. Sadly, this isn’t the case for the long-neglected U.S. underclass.
Other than as a testimonial to extraordinary hard work and dedication, however, the top PISA and ACM results don’t tell us much.
Even former Premier Wen Jiabao has complained about China’s rote-memory approach to education. Chen Lixin, an engineering professor at Northwestern Polytechnic University in Xian, has warned that China produces students who can’t think independently or creatively, and have trouble solving practical problems. He wrote in 1999 that the Chinese education system “results in the phenomenon of high scores and low ability,” an observation germane to PISA results. In the 2009 tests, “students scored low in independent reading strategies, meaning they rely on teachers’ instruction on what to read,” according to the Shanghai Daily.
My hat is off to those 15-year-olds and Jiaoda contestants in Shanghai. But this isn’t the direction the United States should take. Yes, we need to bring up the proficiency of our weakest students — a social challenge that goes far deeper than the harrumphing about “fixing our schools” would indicate. Yet we shouldn’t bring down the level of the stronger students just to win international contests.
Norman Matloff is a professor of computer science at the University of California, Davis. He wrote this article for Bloomberg News.