Thursday, July 06, 2006

SAT Scoring Errors

1) The decision to make known only the scoring errors that resulted in lower scores and to suppress scores for those who were too high was a political not an educational decision. Of the seven students we had who had scoring errors reported to us, only two were truly affected for the other 5 already had previous scores which were higher than the re-scored test. There is likely a higher percentage of students who scores were artificially inflated who are affected than those with aftificially low scores (if our percentages are typical).

2) There may be students who received scholarhips or admissions based on a numerical selection index who received a benefit that they were not qualified for. I cannot see a college withdrawing a scholarship or offer of admission from these students, but some organizations or schools might expand opportunities for other students to take into account "unqualified" students receiving awards or admissions (whether using scores in this way is appropriate or sound is another whole topic of conversation).

3) By only reporting those students whose scores were originally too low makes public relations sense but not educational sense. Obviously there would be less of an outcry from students, parents and ultimately lawyers from students who received corrected high scores than those who received corrected low scores. Yet if the purpose of the SAT is to help students and colleges in a matching process, then the decision to only report the high scores makes a mockery of this test. It supports the notion of college admissions as a beauty contest more than finding the best match. Obviously, if the College Board has reported scores for a student which were artificially too high and the SAT helps improve, with the transcript, a prediction of who will be successful in college, than that student with artificially high scores is being done a disservice. Both the student AND the college may be basing decisions on incorrect information. If there is any credibility of the validity of the SAT, the College Board would released all incorrect scores, high or low. Not doing so makes a mockery of the claim that the SAT is only a tool for colleges to aid in selecting students and for students to select appropriate colleges and further promotes higher SAT's as a goal unto itself, a notion all to common with students and parents.

4) On the comment of the scoring error being a "stone in the shoe"...I think is should be viewed more a club to the head. It should be a wake up call to the College Board that there are serious qualms about the credibility of the College Board. With ill advised decisions to become a for profit company to doing a total reversal on SAT prep (at first claiiming its lack of effectiveness to then actually developing its own SAT prep materials), the College Board is losing the hearts and minds of the college admissions community and to students and parents. It is not just one more blow to blow to their reputation but it could not come at a worse time for the SAT I Writing Test. I believe this may be the death knell for wider acceptance of this test. Many colleges have been sitting on the fence about whether to use this score in admissions and more and more seem to be deciding to not give it the same weight as the reading and math sections and many more are rejecting its use altogether. It is time to do as the ACT does and make this test section optional so that market forces can determine its use rather than it being shoved down the throats of colleges and students.
Q: Is there any student who is harmed by being in an AP class?

A: That's a rather leading question. For one, I said nothing about there being a problem with encouraging students participate in AP classes. In my mind, there are two sides to the argument to having more kids participate in AP classes for their enrichment value. On one hand, there is an established, high level and standardized curriculum which can benefit a large number of students, not just the most talented. I can certainly see the argument that all students benefit from high expectations. On the other hand, is participation without performance really meaningful. Here we had one "AP" class whose students' AP scores were so low that we took away its AP designation. Is a course where almost all the students receive 1's or 2's really an AP course?

The other side of this question is not whether individual students are hurt by being in an AP course but whether other students are hurt by students who may be unprepared for the course being in the course. This is a tough question. On some level, it is like having students audit a course. Even if they are not at the level of other students, they probably gain from being in the course without affecting the other students. But there probably is a tipping point where too many unprepared students bring down the whole level of the course. At what point does the course have the AP disignation without being a true AP course? I don't really have an answer for this, but it is obviously a question the College Board is trying to answer by its AP auditing.

But this is all beside the point. My argument with the Newsweek list is its manipulative quality. If Jay Mathews has an agenda he want to promote, let him do what Lloyd Thacker has done: be straightforward and passionate about it and convince others of the merit of your position by the facts as you and others present them. I see the use of this spurious "Best High Schools" to be using the pulpit Jay's been given in an inappropriate manner. Whether it is good for individual students, schools, the validity of the AP program, or the society as a whole for more sutdents, particularly those who would not traditionally be taking them, to be in AP courses is an open question. To pretend that there is some value or merit to a list of school's with a high AP participation rate is another issue altogether.

So to Bob Turpa's question, I would say he is asking the wrong question. The question he should be asking is there harm creating and publicizing a list such as the one that appeared in Newsweek. I think there is. Lists masquerading as meaningful information are harmful. Promoting an agenda by creating a well-read list in a national publication is harmful. It uses the public's thirst for simplicity to promote a viewpoint.

Some have asked why I am so focused on this list and not all the others that are out there. For one, I have no inherent problem with lists. I produce my "list of lists" which I think is quite useful in context. I use list of majors quite often. The list most often villified is the US News and World report list. How is this different from the Newsweek list? For one, there has been plenty of outcry about it from groups like the Education Conservancy who are much more articulate then I have ever been. Both lists certainly use pseudo-science masquesaring as science. I am bothered that the US News list is so heavily weighted by peer's measurement of "quality". How many among us are truly versed in the strengths and weaknesses of the thousands of colleges out there. It is an absurd notion. I am troubled that is is a rating of input statistics without, like the NSSE, measuring value added. I am bothered by the fact that the highest correlation to the standing on the list is the founding year of the institution.

Maybe I am naive here, but I see the agenda of the US News list as selling newspapers. And maybe I am splitting hairs here, for both lists are clearly cases of the tail wagging the dog. And one might argue that there is an advantage to the Newsweek list because the desired outcome (having students who might not otherwise take AP course take them) is preferable to the outcome of the U. S. News list (have more colleges put emphasis on incoming statistics than improving the student experience). But I am troubled by the use of the media to create a list to promote an agenda, no matter how worthy.
College Board Schools

I have a good degree of familiarity with the College Board. I have been an SAT test center supervisor for 20 years (1984-88 and 1991 to present), was a paid member of an advisory board for College Board publiscations (Book of Majors, College Counseling Sourcebook, etc) and was on two panels organized by the College Board (one on student's perceptions of the New SAT and one on advising first generation college students) at the College Board National Forum. I write this as perspective on my comments. I think there is some merit to what they are trying to do. The goal to work with first generation minorty kids to prepare for college is an admirable one. Yet I have all too often seen College Board as a lumbering, bureaucratic organization which lacks much creativity. I got to read and review many of their publications and this came out again and again: out of touch and off the mark. Then they decided to become a profit making organzation, further muddying the already murky waters. I would question the wisdom of taking an organization with so many misteps to serve in this capacity. I know of no one who describes this organization as innovative or concerned with the welfare of students. They are pretty good at doing what they do and I would question the wisdom of them running schools. If there is any momentum for the college counseling community, it is toward less focus on tests and more on learning. There is clearly a backlash toward NCLB. The one state that has a true obsession with testing is Florida, so there is little surprise that College Board Schools are thinking of expanding there. College Board running schools? Talk about the tail wagging the dog!
How has the thinking about and use of EA changed in the last few years? Do you think the changes are good or bad? Why?

The implication here is that the system has remained the same and the students are acting differently in relation to early action. That is partly true, but this does by no means capture the whole story. Colleges continue to be rather opaque with any information they give to students about admissability and there are major changes going on with admit rates. But colleges who feel they may be left out of this wave are acting to prevent this.

I think we are going through such a period of anomie in college admissions that, for now, EA is a good thing. It allows for some degree of assurance in a rapidly changing landscape. There was a quote in The Chosen that there is not a democratization of oppotunity in college admissions, just a democratization of anxiety. The landscape that is changing the fastest is in urban areas.

Like the double deposit discussion this is another red herring to attempt to blame the victim. I have had only one kid double deposit that I know of in a career directly counseling over 2000 kids and this year did not have a singel kid apply to more than one school early action. Have I had kids apply EA, ED I and then ED II? Sure. But the system is designed to encourage this kind of thing. Would I have a problem with a kid applying to 6 EA schools. Not at all, as long as the kid is not choosing those schools BECAUSE they were EA. That is gaming the system. But if they were six schools that he would have applied to whether the schools were EA or not, then I would not discourage him,

I guess you have to look at the school culture and the behavior of the students. If there is a highly anxious student/parent body and a culture of "means justifying ends", then you might feel that too many kids are choosing colleges by their admissions plans rather than what the colleges have to offer. And that certainly is a pernicious thing. We happily do not see that going on. We have almost every parent and student on an e-mail list and send things out daily (you can log onto then to high school, then to guidance and scroll down to the end and sign up for any group and you can see the archives of what is sent out). Parents and kids begin to feel that they have a sense of what is going on despite what I believe are the intentions of the colleges and the media, for very diffenent reasons, to distort what is going on. They also, at the same site
have a full pdf version of my book which I think gives them a lot of info that they need. They need information without hype, which is what I believe we give them and that is the greatest antdote to anomie.

Durkheim noticed greater suicides during times of economic change, which he called anomie, even when economic conditions were improving. The fluidity of the college landscape is very damaging. A lot of the things may settle out. The intense marketing of colleges, beginning in the 80's, is still only relatively new in societal terms, with the most intense models of enrollment management probably emerging most in the last decade. We have had widley cyclical numbers of students applying to college, from a decline in the 80's, to a rapid and steep climb in the 90's. Things are starting to level off and will begin declining in the next decade. Will this decline help students due to increasing openings in the more selective colleges? Not according to Durkeim. It will still make for changing expectations and high uncertainty. Not until there is a stable base of applicants and some change in how colleges do business (such as all colleges going rolling admissions or have a preference system like that for medical schools) will there be a change in the huge anxiety and thus game playing by students, parents, schools and colleges.