J. Tyson Tildon, president of the Baltimore school board, saw red recently when he came across a "finding" by Harvard University researchers that good reading instruction can overcome the home conditions of impoverished children.
Total number of children tested to reach the conclusion: five.
"This is really bad," says Tildon, a biochemist and professor of pediatrics at the University of Maryland. "In medicine, we would never put out anything like that."
But it happens thousands of times a year in education research, as Tildon says he's discovered in seeking policy guidance in his two years as city school board president.
Tildon and many experts say much of what passes for research in education is of poor quality, lacks authority and is inaccessible to the people who need it most: teachers in classrooms, professors in education schools and those who set education policy.
Reading research -- a politically hot topic these days -- is particularly suspect.
Congress was so concerned about childrens' poor reading skills that it ordered a National Reading Panel in 1997 to sift through the 100,000 known studies on reading since 1966 and report on their quality and usefulness.
The group's report won't be out until next year. But Donald N. Langenberg, University System of Maryland chancellor and chairman of the reading panel, says "large chunks of research literature aren't making it through our sieve."
Reading research "suffers from a lack of rigor and a lack of investment," says Christopher Cross, former president of the Maryland state Board of Education and now head of the Washington-based Council for Basic Education.
One of the primary causes is the decades-long reading wars between proponents of the phonics and whole-language approaches. The two sides argue bitterly over whether children should first learn the sounds that make up words or be immediately immersed in literature.
Many reading studies "are driven more subjectively, more philosophically than they are scientifically," says G. Reid Lyon, who directs reading research at the National Institutes of Health (NIH) in Rockville. Researchers stake out positions in the reading wars and then conduct studies to back up their opinions.
There's plenty of education research going on, but much of it isn't valid, Lyon says.
Children in studies aren't chosen at random, or the number of pupils studied is too small. Control groups aren't established for comparison. Often, there's no replication after a one-shot experiment. And educators, unlike most medical researchers, often don't submit their findings to peer review.
The result is that several of the major instructional programs and initiatives in American education are based on scanty research or they run counter to seemingly valid research:
States and school districts are rushing to end "social promotion" by retaining poor readers in grade. Estimates of retention rates range from 17 percent to 30 percent nationally. Yet most studies show retention hurts children academically and results in higher dropout rates.
The charter school movement is sweeping the states despite a paucity of evidence to show that these schools are superior to run-of-the-mill public schools.
Though there's conflicting research evidence on the benefits of smaller class sizes, California, Texas and several Maryland districts, among many others, are rushing to reduce class sizes. "One of the things we know the least about is the effect of lowering class size," says Cross, "and California is finding out that when you lower class size across the board, there are unforeseen negative consequences."
The respected American Institutes for Research in Washington studied 24 schoolwide reform programs used in thousands of U.S. schools. Only three, including Towson-based Success For All, showed strong evidence of positive effects on student achievement.
After President Clinton vowed two years ago to recruit a million volunteers for tutoring in reading -- and pledged $2.75 billion to the effort -- Johns Hopkins University research scientist Barbara Wasik evaluated 16 volunteer tutoring programs. Only three could demonstrate effectiveness, and only one used a control group of children who did not have tutors.
"I'm not at all against voluntarism," says Wasik, "and I'm sympathetic to those I spoke to who said they didn't have the money to do a thorough evaluation. But on the other hand, you shouldn't make national policy based on a feeling.
"If you have cancer, do you want to go to a volunteer who's going to help try to cure you? I think not being able to read is equivalent to having cancer, so trying to solve the reading problem on the cheap is ridiculous."
Up and down the line, there is what Wasik calls "disconnectedness" between researchers and practitioners.