Saturday, May 13, 2006

Americas "Best" High Newsweek

Are you out there as disturbed as I am about the yearly list that Jay Matthews puts out in Newsweek purporting to list America's best high schools? I really don't know where to begin to state why it is so inaccurate, distrubing and damaging. The list ranks high schools by the ratio of the number of AP tests taken to the size of the school.

1) There is no evidence whatsoever that there is a correlation between simply taking AP courses or tests and students' post high school performance: Klopfenstein and Thomas in The Link between Advanced Placement Experience and Advanced Placement Experience and College Success (2005) find, in a rigorous study, that participation in "the three most popular categories of AP classes, math, English and history, do not consitently improve college retention or GPA... Students are no better prepared for the academic rigors of college than their non-AP taking counterparts."

2) There might be some justification (llittle in my mind) for ranking high schools by their participation rate AND their performance on the tests. But to 'reward' participation only is like praising that kid in little league to never gets a hit but gets a walk from not swinging. "Good eye, Jonnie, good eye."

3) There seems to be some misguided attempt at social engineering here that I don't get. I do not think there is enough evidence out there to state that increasing participation in AP courses is a social good. Yet this list is clearly an attempt to do just that. As with the case of the US News list of colleges, this is another case of the tail wagging the dog.

4) I don't really have any idea what this list is really measuring, except maybe a school or state's emphasis on testing and appearance over learning. Florida is by far number one on this list with 21 of the 'top' 100 schools. My father, who lives in the state and is a PhD. in psychology and a highly trained psychometrician, complains constantly of the state's obsession with standardized tests. I look at the state of Florida's students' scores on tests like the AP's, the PSAT's and other national tests and they are no where near the top in the nation. Yet those states and schools who do score the highest are almost absent from the list. Alabama, Oklahoma, Arkansas and Tennessee each apear as many times on the list as Massachettes, New Jersey and Connecticut COMBINED.

5) The school list does note the percentage of free lunch kids, but does not use it in the formula. If the formula were something like (Mean AP Performance) X (Participation Rate) X (free/reduced percentage), there might be improvement from both a predictive and social engineering point of view.

6) This is simply, in my mind, bad science and lazy and sloppy journalism. It is taking numbers and pretending they mean something without doing the work that such a activity should entail: controlled study and expert analysis.

That the journalism community has been allowed to define educational quality in this way is an indictment of both the education and journalism professions. It is irresponsible reporting for which the Washington Post (the parent company), Newsweek Magazine and reporter Jay Matthews should all take responsibility. It is also the responsiblity of the educational community to re-take the reins of answering the question of educational quality on the secondary and post-secondary level.

The educational community's response to this issue seems to be similar to the Democrat's response to the Republican's. The republican's come up with pithy, simplistitic and deceptive catch phrases ("death tax", "soft on terror" "waffler") that the Democrats respond to with long diatribes that few understand.

The educational consumers want information and it is the responsibility of the educational community to provide that informantion is a way that is meaningful and useful. Just criticizing the lists (as I am...guilty) is not working. On the secondary and post-scondary level it is necessary to take the mound of data that is being collected and make it that students and parents can find and use it easily. Does it mean we should be developing our own lists using data like the NSSE? I don't know. But it is time to begin a discussion on how to transform the way students and parents look at the quality of schools or we are doomed to have ever more simplistic and unsicientific lists.

Joy Matthews Response

I saw this posting and was very sad i have never shared with you my rationale on this. Below are the faqs that summarize what is the result of 23 years of reporting, thinking and writing about high schools, but i know you like to go deep, so i would recommend you read two of my books, Escalante, which sets up my mind set for trying to change the way we think about high schools, getting away from the standard "rich kids good, poor kids bad", and then Class Struggle, which introduced the index and my reasons for it. I get a lot of very kind emails from AP teachers like the one at the end of the faqs who say it helps them a lot. It is less popular, and harder to understand, in more blessed schools like yrs, but i think once you read this you will get what i am talking about. i would love yr comments after you read this. you have been so helpful to me, and if i can get a dialogue going, that would be even more help. maybe we could even debate it in my column. ---jay

My Response:

I read the full text of your faq as well as the recent web posting on what makes a great high school and it only strengthened the feelings i originally wrote in the posting. 1. I didn't see a shred of evidence that you are basing your list on anything backed up by evidence, either through a controlled study or a review of literature. I quoted the study by klopfestein and thomas that found that participation in ap courses does not improve retention or gpa's in college. is there any evidence you have to the contrary. 2. I do think that this is an attempt at social engineering. your goal is to increase participation in ap courses, particularly at schools with less advantaged students, with the assumption that there is a value to exposure to higher level teaching, expectations and material. this is your perogative and it may be a laudable goal, if there were evidence that it was productive, other than a hunch on your part. I still think it is inappropriate to produce a list such as this which purports to identify strong schools when you have no evidence whatsoever on the quality of instruction. This is why so many like me, who are no great fans of the college board, are lauding their decision to begin auditing ap programs. 3. You seem to contradict yourself on the use of performance statistics. First you say that school's restrict ap participation and thus have skewed statistics and then state that the college board has an 'equity and excellence' rate for schools which takes this into account but still choose not to use it in you computations. 4. I know you want to redefine quality in a way that does not reward wealth and advantage, and I have to say that this is an interesting and provacative approch. but i think it is misguided. anything the journalism or education community does to promote participation without quality has great risks, not the least of which is treating taking an ap course or test as an end in itself. I do not believe this is a worthy goal. It can lead to students to set the bar to the level of participation only. This is like comparing the olympics to the special olympics. aren't they both competing in olympic events....but you and i know there is no similarity between the two other than participation... 5. I might be convinced that your methods or your goals were sound if either - they were based on established, corroborated, controlled studies whose conclusions supported your assumptions or - you did some study of your results which supported your position..i.e that students who participated in ap courses, no matter the result, performed better in college, graduated from high school at a higher rate, had higher sat scores or overall high school records, had higher college graduation rates, etc. or that schools that were higher on your list, in aggragate, had higher college gpa, graduation rates, etc. as schools that were similar but did not have high ap participation rates. 6. There are few suggestions that i might make -use a formula that uses participation and performace in equal measure -multiply the result by the free/reduced percentage so that disadvantage was taken into account.

And Jay's response:

Thanks scott. the multi-pillars of my confidence that this is the right way to go are: ---the many emails and conversations from great educators like you in low income schools that say this helps them, the letter at the end of the faqs being a sample. ---23 years of watching kids struggle and sometimes fail AP tests but hearing from them that it has helped them survive in college. ---and most particularly, as far as research goes, the UC and NCEA reports of AP in Calif and Tex that show significant correlation between getting a good grade on an AP test and higher graduation rates. You can't get a good grade on an AP test, except in a tiny number of cases, unless you take the course and the tests, so schools that restrict access are standing in the way of higher college grad rates, as least based on that research.; the NCEA tried to look at the kind of kids i have covered, marginal students at risk of failing an AP exam, and could not reach a conclusion. it looked like blacks did better on college graduation even if they flunked the AP test, and for others it was a wash, but they could not reach any conclusion. My problem with the Klopfenstein study is that she looks at kids who have taken an AP class without any distinction between those who took the test and those who did not. AP kids who dont take the test are not getting a full dose of AP, and my reporting indicates that such classes with few kids taking the test are rarely taught at an AP level, since there is little incentive to do so. Lastly, given all this, and i really want to hear yr thoughts because i think this is the heart of yr argument, but i am not getting it yet, what harm can come from removing the barriers to AP and IB participation and letting all students, as Garfield did, take AP if they want to? What harm can come from letting motivation be the entry ticket? All the kids I have followed, and that is a lot of kids in two decades and three books, say that it helped them to make that struggle. And yet the vast majority of US high schools restrict access, and for reasons that they rarely even review or subject to debate. If you have a chance to skim through Class Struggle, you will see my points made in more detail. But i do want to hear you on the harm factor. as for the contradiction between my distrust of using grades and using the e and e rate, you are right. i have a problem even with e and e because i think any focus on passing scores inhibits efforts to start AP or IB in low income schools. but as long as i dont use the e and e in the rankings, i think it is a useful number, just as is free and reduced lunch percentage. ---jay

Counselor comments:

*I am the principal at _____________High School, one of California's top academically performing schools. I am absolutely appalled that Mr. Matthews can gain an audience with this ridiculous way to rank schools, and I think it's even worse that Newsweek picks it up and makes it a national pronouncement. I would be happy to talk to you in more detail, and I might even be able to get my Principal's Collaborative, a group of principals from ten of the highest performing public schools in Los Angeles County, to make a joint statement, if that would be of any use. Thanks for taking up this issue. My former assistant principal carried on some dialogue with Matthews about this, and he even invited her to speak against AP's at a forum he moderated (our school would like to get rid of AP's completely, but we play the game). If we can help, let me know.

*Bravo to you for this response! I couldn't agree more, and only wish there was a way to communicate this to all of the students and parents who take stock in these kinds of listings!

* It seems this entire debate over the challenge index is centered around attempting to find one rule for all, which is destined for failure. School culture and community expectations are perhaps two of the most significant influences in public schools. In one school if a student struggles in AP and earns a 75 as a final average, the student and parent may understand that this was a challenging course that they attempted and were happy to have the experience. The parent may also not fully understand AP curriculum or what it means. Given the family's total picture, the level of classes in which the student is enrolled may be the least significant issue in their lives. In another school, the culture can be one where if a student is struggling with a 75 in an AP, the onus and intense focus is on the teacher not simply to try to bring the student up to a 'B', but that anything less than a 'B' is an indictment of the teacher's abilities. Questions arise as to how the counselor had allowed a student to take on a program that was beyond his/her abilities. Parents may also look in retrospect at how the school could not have been aware of how this would negatively impact admission to a highly competitive college. And issues inevitably arise from parents of students who see their own child's learning being impacted, whether true or not, by those students who are, "slowing down the class." Subsequently, what is school administration doing about this? These are two different worlds. Each one carries their pluses and minuses, but the most important thing to acknowledge is that you can't necessarily go about business the same way at each school, or be critical of one for not abiding by the same rules as the other. Given that schools are an extension of our homes and families to a large degree, I envision prescribing one standard of operating as no different than walking into one family's home and instructing them that they must now abide by another family's standards, which has been deemed by a national publication to be the better way of living. Hopefully we can keep that idea contained in the realm of reality television.

*I'm on your side of the aisle. Have been since Jay started doing this about 5 years ago. Your summary of what's wrong with it is masterful (really) and I wish I had written it. ACT, in the last year, has come out with two reports showing that students aren't prepared for college. Now, one way of putting the ACT report and the findings about AP together is to say, that there is no contradiction – AP isn't good preparation for college, except that it teaches students to focus on their schoolwork in ways that will serve them well as students in college. But that's like saying that students who score well on the SAT or ACT tend to do better in college than those who don't. To claim a predictive relationship because of the tests is one possible way of understanding that relationship. Another, simpler, way is to recognize that students who do well in college generally are those who have done well in high school and they tend to do well on the tests. In other words, it's like saying the people with certain characteristics tend to be good at certain things – or not (all you have to do is look at me to see that I'll never be a good pole vaulter). But I wonder if the ACT reports won't give ammunition to those who argue that AP is good preparation for college in the sense that there does seem to be an easy, natural association of the rigor of AP (well, mostly – we all know about the exceptions; why else did CEEB begin the "audit" process?) and the rigor of curriculum that ACT seems to be talking about. No conclusions here, but I wonder if we aren't going to see a continuing rush to AP as the standard for the "college track" in high schools. I'm coming more and more to the view that high school generally is lousy preparation for college, and not just academically. The more AP is shoved down the students' throats, the more we will see a tiered high school program and the more emphasis there will be on taking the "right" courses and getting the "right" grades so that you have a chance at college. It goes on and on. The machine is broken, the emperor has no clothes, we all know it but none of us seem to be able to do anything about it. Maybe the post-consumer society, when it comes, will think differently

*Scott, Not to be overly political, but how can false pretense with testing not be a focus in Florida when you consider who their esteemed governor is? You might appreciate this article on good 'ol Jeb and his push for numbers and less creative learning. <>

* I have to say that Jay truly believes he is one of the nation’s authorities on AP and education. I am continually appalled by his lack of understanding and his continual reliance on feedback from a small handful of self-promoting VA public high school teachers, who he deems to be experts. Having read essays on the AP exam during the grading period, that have sometimes been written on an elementary school level, no one can tell me that the number of students taking the AP reflects anything about the quality of a school. If you want my take on this, which I have shared with ______________, I believe College Board is positioning itself to take over the US public school system in the future. Privatizing American education is already on the horizon. I don’t think this is a case of the tail wagging the dog – it’s a case of the media hog-tying educators and disallowing us to educate the public. Makes me wish I had enough pocket cash to take out a full page add in the NY Times. *As ever, you're right on target. I agree with your dad.

*I really appreciate your statements on the NACAC list regarding the use of that list to somehow rank US high schools. Our parents here in _________ are always looking at the list as if it were some type of gospel. It just grates on my nerves when I have to be part of a conversation about why a school was or was not included on the list. I shared your comments with our curriculum person since our school is seemingly getting more and more involved with the AP frenzy, so much so that teachers are wondering how we got to this point. *Amen to your thoughts.

*It's just as wacko as rating colleges... *Instead of making a personal comment, let me just direct people to Jay Mathews' preemptive rebuttal (of sorts) in his weekly on-line column. It is available at < <> * While I understand your concerns over these rankings, I would like to point out one way in which they can be helpful. Despite the simplistic formula for determining this proxy for the level of educational rigor in a high school, the list does in fact include the vast majority of strong high schools in my state and others. In my opinion, the Advanced Placement program (along with IB) has been one of the true success stories of educational reform in the US. The expansion of AP (and IB) curricula has served as a catalyst for pushing the academic bar higher in many high schools--and for many kids who have long suffered the effects of low expectations. As you may recall, I have worked for many years as a member of my local school board. By pushing for a more robust and accessible AP program in our high schools, we have seen substantial gains in academic performance and college-going rates. Every year, I use the Newsweek "Best High Schools" issue to challenge my board and our administration to push for ever higher levels of academic challenge. In fact, I just cited this year's rankings in a public statement Monday evening. I noted that I was pleased to see (on the Newsweek website) that 3 of our 5 high schools made the list--they weren't in the top 100, but they were on the larger website listing of some 1,200 schools. I challenged our school division to strive to have all 5 of our schools on next year's list and to work to move up in the rankings by further expanding our students' AP participation. It may be that "ranking" or calling them the "best" high schools smacks of too much elitism for you. Or it may be that the false precision of the rankings gives too little acknowledgment of the myriad other factors that constitute "quality." However, I just wanted to offer another perspective and suggest that these things can be used to spur positive actions on the part of some of us who are toiling in the vineyards of school improvement. Hope all is well with you. I miss our frequent interactions through NACAC service but I do keep up with your listserv postings. Take care.

*You are so on target , again.

* Scott, thanks for your comments. It is one of those things like US News and Reports that will hit once a year and you always shake your head and say "why"?....your thoughts were right on....we have neighboring schools with 70% in AP courses but low percent taking the tests and admission that the curriculum is not fully taught....maybe this is why College board is developing the AP police. I actually was writing on another topic. I am mentoring a graduate of DHS who left here in 1990 and went to Northwestern. In his junior year he and his younger brother went to med school in Hungary. They returned 6 years later (brother went from hs to med school directly...all taught in English) and passed the USMLE and did a residency and became docotrs. A few years ago the other doctor decided he wanted to give back and formed a company to take students direct from high school to med school in Slovakia and now Bulgaria. No MCATs required. I have helped him connect up with the Detroit Public schools and develop an Urban Physician's initiative with Marygrove College where inner city kids take the first 2 years at Marygrove and 4 years in med school in Bulgaria..He has done the same with two colleges in Illinois. He is actually coming to your NJACAC meeting and I would love to have him have a chance to visit you at your high school after the conference and perhaps a few other high schools. However, I need guidance as I do not know the schools in NJ. Can you provide any guidance. His name is Dr Mark Kaushal and his company is Source America. Thanks Scott.

*Well, Scott, I couldn't agree with you more. The death knell, to me, was when George Bush mentioned AP in the State of the Union.If he likes must be bad!!! Funny how his brother's state gets the good press (Florida also pays for all testing), and yet as you noted, the scores on national tests are awful. There just has to be some big "kick back" for this relationship.

*I also was shocked to see what the criteria was. I am an independent consultant in ______________ (right down the road) and I am more impressed by schools that received Benchmark Awards and things of that nature because they take into account the diversity of the school and many other pieces of info. Thanks for your comments.