The time has come! To talk of carpenters, of ceiling wax, and KINGS!
Or in this case, that afore mentioned study. Before I do, however, let me note that I will address J’s question after my analysis of the analysis of the literature in the field.
Now, Genesis, when critically approaching a study one should ask three questions first, above all others:
1) Who wrote it?
2) Is the individual capable of telling the truth (that is, do they know it)?
2) Is the individual willing to tell the truth?
The article, “A Win-Win Solution: The Empirical Evidence on How Vouchers Affect Public Schools,” was written by Dr. Greg Forster and sponsored by the Friedman Foundation for Educational Choice. Notably, Forster had written in support of school vouchers previously, and the Friedman Foundation is pro-school voucher as well. If you’ll look at the endnotes of the article, you will also notice that Forster and his colleague Jay Greene make up 10 of the 25 citations. About a third of the material that he is reviewing is his own or that of his collaborator.
Do you see anything suspicious about that? He might know the truth, but he is also heavily invested in presenting his prior stance, rather than critically examining the evidence in this meta-analysis if such an analysis would undermine that stance. This doesn’t mean his analysis is bunk (after all, lead researchers in a field will often cite themselves, because anyone writing a similar article would cite them), but it does mean that one must approach it carefully.
Let us look at the article itself. On page 10, paragraph 2, Forster states that “defenders of the government monopoly on education” make various claims (the claims are not here important), but he fails to provide a citation to back this claim up. Citations are necessary to verify his summary of the opposition’s claim. Without it, the reader cannot determine if his summary is a strawman fallacy. Now, his claims in this regard are fairly benign, and so lacking proper citation here isn’t particularly a sticking point. However, this displays very bad form on the author’s part (and, indeed, on the organization that would let that slide). One MUST support one’s claims with arguments and citations if one is writing an academic paper. Citations, however, are scant (but the research done on the topic is also a bit scant).
In his defense, one might balk at the information contained in paragraph 3 of the same page. At the time the study was performed there were 24 programs in 14 states plus D.C., but on page 5, in the Executive Overview, right hand column, first full point, it is stated that only 6 states have been studied plus D.C. An analysis of less than half of the states involved might seem dubious, but 6+1 is actually a fairly good sized cohort, assuming the statistical analysis is sound; many scientific studies make do with small samples… though in turn, the more variables there are, the larger the sample size needs to be and the more impressive the results need to be before they are notable.
First full paragraph on page 11 is an appeal to emotion fallacy. The second, third, and fourth full paragraphs, on the other hand, are quite interesting and informative (though one might wonder if money gets left behind when school vouchers are used, saving schools money… where are those extra funds coming from? The school where the student is going to? Or is there just more money in the system, in which case one would need to divorce vouchers from simple increased funding). You’ll note that he cites his sources there. How lovely of him not to expect us to take his word for it. Unfortunately right on the next page, full paragraph 2, he is right back at making claims that he doesn’t support: do American’s know that monopolies provide poor quality?
Do all monopolies provide poor quality? The government monopoly on Fire Protection seems to be a good bit better than private competition, for example, so his statements do not seem to be on par with saying that water is wet (a statement that one could reasonably get away without citing). He thus needs citation and arguments if he is going to bother saying this at all.
As a bit of an aside, maybe I am just not perceiving things correctly, but I am fairly sure all science is empirical, and I can guarantee you that random assignment is not the “gold standard” for the entire shebang. It is very important, don’t get me wrong, but that isn’t what “gold standard” refers to. The gold standard, in science, is that which everything else can be measured to in order to approximate worth. If you want to know if a particular type of neuron is absent from a particular section of a brain, you will use whatever the field considers to be the “gold standard” (laser-capture microscopy, for example, though I don’t believe that is actually the gold standard in that field, but one of the many standards used). Randomization is a necessary part of science, but it is more along the lines of the wood standard. As in, don’t take any wooden nickels.
Simple randomization is great, but randomized double-blind studies (in which both the researchers and the patients aren’t sure who is in the variable group and who is in the control) are where it’s at. It is a little hard to do in this situation, mind you (presumably there is no way for a student not to realize they are not at a different school), but without it, observer bias is something that one has to take into account. Forster brings up randomization but doesn’t address observer bias; that is something that has to be addressed. While the studies he is reviewing might address the concern, such information should be echoed in the reviews as well.
Unfortunately, the first study of Milwaukee schools (page 16, 4th paragraph) is of limited value. The study compared different levels of involvement, not if involvement is beneficial. Simply put, as Forster presents it, the study had no controls (later it sounds like they did have controls, just not positive and negative controls, which is nearly the same thing). However, Hoxby’s study seems essentially to have served as a pilot study, setting the foundation for later research. Pilot studies generally aren’t as useful as other studies, but they can get away with things like poor controls. They’re just supposed to show that something is worth studying more into. And people did, so it served a fine scientific purpose.
Forster’s analysis of Carnoy study in Milwaukee is, unfortunately, flawed. Carnoy argued that the voucher programs did produce benefit but that this benefit was in doubt since the benefits did not continue to increase. Forster discards that objection because the benefits didn’t go away, either. Now I haven’t read Carnoy’s study, but for Forster to be correct in his rejection of the former’s objection, Forster must essentially claim that total market saturation was achieved in two years. Assuming that the program was K-12, total market saturation couldn’t be achieved until 13 years had passed (though significant saturation would be reached at the 4 year mark, with diminishing returns until 13 years). Not only must individuals become aware of the program, they must be convinced of its merits, and they must have no reasons for remaining at a school that a voucher would not address. One should expect the benefits to have increased (albeit slowly) after 2000. That they did not indicates that the vouchers themselves were not producing the benefits but that some unidentified side effect, which was independent of student participation, was the cause.
The second objection that Carnoy raised is also more significant than Forster estimated. Areas with fewer private schools did not see different benefits than areas with more private schools. The problem there is that the vouchers are supposed to improve schools by increasing choice. It is a greater expression of choice to move from a public school to a private school, or vice versa, than to move from a public to public school. I would also like to see an analysis based on institutional density in an area (more schools in the area should have increased the benefit, fewer schools should have decreased it), but I probably need to look into Carnoy’s study directly for that one.
Greene’s Florida analysis (pg 18, starting with paragraph 4) is very interesting, but I question his methodology. The focus of the analysis is on schools that received D and F grades (F’s being necessary for vouchers to come into effect). That is good, but it needs to be put into a larger context. Did the grading system influence all schools? Did C schools on average increase their scores, even though they were in no particular threat from vouchers (Chakrabarti’s analysis on pg 19 indicates yes)? If so, then it indicates that the threat of vouchers was not the only thing motivating schools. While F schools increased their grades more than D schools, was this a predictable trend? If F schools increased by X amount, did D schools increase by 1/2 X, and C by 1/4 X, and B by 1/8 X? Again, if so, this would indicate that the grading system, not the threat of vouchers, was a useful tool for encouraging improvement. This generally gets at the problem of only using positive controls; a negative control (ideally a school not involved in vouchers at all, but alternately schools not subject to the threat of vouchers could also work) is needed to properly frame the information.
Forster’s 2008 study (pg 20, starting paragraph 3) is interesting since it implies that both were important factors. Merely getting a grade helped, while the threat of vouchers helped more. Of course, if vouchers ever became normal (and not something to fear, as Forster noted in his introduction), one might wonder if the threat would retain its significance. But that is admittedly outside of the scope of his investigation.
Forster’s study of Texas, on the other hand, (pg 21, full paragraphs 2and 3) is dubious since more money was being pumped into the school system via a private voucher program. Were the gains the result of vouchers or simply more money? The study should have compared it to a publically funded, similar voucher program elsewhere.
The alternate theory section is interesting, but underdeveloped. Forster didn’t consider (page 24, starting paragraph 2) the possibility that merely moving individuals to a new setting could result in increased test scores (I believe in another thread I had even suggested that moving students around could break up undesirable social clichés and promote a desirable learning atmosphere). The analysis he did provide was relatively good (minus the snide remarks), just incomplete. The problems with it are the same problems I noted in the individual analysis sections.
His analysis of the stigma effect, however, (pg 25, 2nd column, 3rd paragraph) is shoddy. That dissimilar programs produced benefits without the stigma effect does not comment on the stigma effect in the program where it might be present. It is enough, by itself, to argue that the gains in Florida are not due entirely to stigma, but it doesn’t discount it. He also discounts the stigma effect because there was a single-year backslide for schools that moved out of the threatened range. A single year does not a trend make. One would need to analysis the schools themselves to see what they did differently between their years threatened and this one year non-threatened. Did any policies or practices change? If not, a backslide could be due to random fluctuations. Changes in data need to be investigated and explained. To note, if these were high schools, then that last year also represented a full changing of the guard; no student there had ever been stigmatized. The stigma effect would have been removed, then.
Chakrabarti’s response to the stigma effect (pg 26, first full paragraph) is interesting, but needs more analysis before it can be properly interpreted. One would need to go to her report, it would seem. Florida had a ranking system prior to the A+ program, but was it a similarly structured program with publically recognized indicators of failing? An F is far more visible score to the public than a score of Purple, for example. Forster again overlooked the influence of stigma, however. If vouchers become socially acceptable, will the gains of the A+ program be lost? Indeed, is it even a prudent course of action to make vouchers a punishment?
Forster then goes on to note that another study, Figlio & Rouse’s, had results that contradict Chakrabarti’s. She said the old program had no affect on bottom ranking schools (it appears she did not look at higher level schools), Figlio and Rouse say that there was an affect but that it was smaller. A third study, then, is needed to identify which study is correct.
To note, Forster’s comment on page 26, third full paragraph, that private schools boosted voucher improvements is at odds with his dismissal of Carnoy’s similar but opposite arguments regarding his own Milwaukee study. In that study, the location and density of private schools did not influence voucher success, but in this study it did. Why is there this difference? What factors explain it? Why is that one the outlier and this one the accepted.
His conclusion to the discussion on the Stigma effect is poor in itself. He concludes that either the stigma effect doesn’t exist or it exists but has a small impact. That is an unacceptable and (almost) entirely un-useful conclusion. Either something exists or it does not. If the data is not in agreeance, then more data is necessary to establish what is real and what is the result of confounding factors. Outliers need to be accounted for.
Forster’s discussion on the regression of the mean was interesting.
The author does make a reasonably convincing argument that voucher programs are beneficial, but the quality of his analysis is far from ideal or commendable. His own objectivity is questionable, his style contains fallacious rhetoric, and his itself is hasty. The article needs a section on methods specifically (it is one of the more boring parts of a paper to read, but also one of the most important), and it needs the help of a statistician to frame the math in a significant manner, rather than having it as almost an after-thought (and even then, not much of one).
As I originally said, it isn’t that a close examination of the article should make one discount it, but it should lead one to be careful of the author’s claims and analysis. In a fledgling field, such as voucher-research appears to be, this isn’t a disgraceful review of the literature, but neither is it a paradigm establishing one. It is really more of a systematic review of the literature, rather than a critical review, which subtracts from its potential use.
I see a lot of antipathy toward the school of thinking which emphasizes memorization and rote as an educational vehicle. Given that our decline in global literacy and numeracy rankings is due largely to our de-emphasis on memorization while other nations have continued to emphasize it, would anyone like to explain in greater detail why they are opposed to this system, and explain what they think a viable alternative would include?
First, I call your basic supposition into question: has memorization, in general, been de-emphasized?
Second, I call your basic conditional supposition into question: has a de-emphasize in memorization resulted in a sliding of the U.S. (for I assume you are participating in the standard Amerocenterism of these sorts of discussions) rank as compared to other nations which have emphasized it, as opposed to other unrelated factors?
Memorization is a very useful tool when information is not easily accessible. Pre-modern educational structures focused on it, since reference sources were nearly impossible for the average elite-individual to access. Memorization, then, allowed an individual to see the book, commit it to memory, and pass it along. In an age, however, where information is freely flowing and easy to access, memorization loses its value. The more omnipresent access to information is, the less important it is to memorize it (though the more important it becomes for us to be aware that the information exists).
It may be that though memorization is not as wide spread, it hasn’t been de-emphasized. Rather, it may have been refocused onto those points were technology has not made it obsolete.
That doesn’t seem to be the case, but it is a worthwhile consideration.
May I be anecdotal for a moment? A friend of mine was recently (as in, the last two years) diagnosed with diabetes. Doctors insisted that she was type II despite the fact that she was insulin sensitive (which alone indicates type I or a variation). The reason for this is that she did not fit into the mold of the memorized information of the M.D.s. Type II is X, Type I is Y. Memorization failed this friend as she was misdiagnosed for about 6 or 8 months. Critical thinking skills, however, finally revealed that she had Type I (well, latent autoimmune diabetes in adults, or LADA, actually). Memorization works well enough when nothing unusual happens.
Of course, then there is the opposite; Ph.D.s with critical thinking skills overlooking the obvious because they don’t have the raw knowledge base necessary to connect to important points of information. No matter how good you are at thinking, you can’t solve a problem you don’t even know that the information you need exists.
If you want the best doctor around, you want an M.D./Ph.D. You want memorization and critical thinking.
Thus I argue against memorization because, currently, it takes the place of thinking. Not that we enforce memorization much, either; really busywork is where our education spends most of its time, though it is busywork intended to aid memorization, not comprehension. We memorize names and dates in History, books and themes in English, equations in Math, random disconnected facts in Science, and rules in P.E. None of these teach individuals how to approach something, they all teach individuals to accept what other people have already said. It is a means of indoctrinating the population into preconceived notions without needing to justify them.
History is the perfect example. At least with science, people can’t (usually) leave High School without at least hearing about the Scientific Method. Most people will need to get to the graduate level before they’ll hear about the Historical Method. This makes history a dead thing; it is sort of like someone who has been fed beef all their life but never having known that beef comes from cows. Beef is tasty, but the individual gains a greater appreciation for it by knowing its origin. Likewise with history, if individuals understand the process of conducting history (since history really is more of a verb than a noun), history becomes more meaningful. Because we keep it at the level of memorizing facts, history is dying out in the United States (well, ancient history; modern history is alive and well).
Does this mean that historical facts shouldn’t be memorized? Not at all, but rather that thinking is important too and needs to be included.
I would propose, briefly here, that the reason other nations are doing better than the United States in global education rankings is actually two. One, other nations have a more efficient educational system. Certain individuals of pre-specified educational merit automatically get college paid for, others go onto other sorts of education and career options (in some places with Trade Schools even being respectable options). This creates an environment in which individuals can excel to their capabilities, and it also adds much needed bite to educational drive. If you ever hope of being ambitious, that ambition needs to be realized at young age. Good or bad, this is a motivation that Americans lack. Two, other nations have increased their educational systems over past years while the American system has not been significantly altered (especially not in desirable ways). It is little surprise that a 2006 Toyota Camry could out perform a 1986 Toyota Camry. Some slippage from wear and tear occurs, but a lot of it also has to do with improvements over the intervening years. The American educational system is getting to be outdated. Indeed, if we do not seriously address the issue, the entire thing is likely to collapse in the next few generations due to educational inflation (HS diplomas aren’t worth anything now-a-days, and Bachelor degrees are starting to become the same).
A viable alternative to the current system, J, is one that emphasizes both thinking and memorization. Along with having 1st graders memorize the spelling of various words, we should be teaching them why those words are spelled the way they are spelled and mean what they mean (yes, we should be teaching linguistics in English classes). We should make Jr. Highers memorize geometrical equations by forcing them to discover at least a few of them on their own; by helping them discover why some of these equations are such wonderful things and how they can be useful in our daily lives.
I hope to become an educator myself one day, but unless I can teach historical processes and well as historical facts, I won’t make a difference.
Oh, as an aside, I suspect (though have not studied) that the decline of the quality of education may coincide with the rise of educaters who are trained in education instead of their subject matter. Back in 2002-2004 I worked in a Teacher Credentialling office in California. You could get a liscense to teach a Math Class having never taken a math class yourself (dramatically speaking, since a college degree was required and math was part of the GE requirements). But, you could not get a liscense to teach a math class if you had never taken a class on teaching practices. Some of the credentially program made sense, but other parts of it were bunk. Maybe if we spent more time ensuring that we were teaching the subjects and less time worrying about how we were teaching, then we might do better.
