Thursday, September 25, 2008

For Those Who Would Better Understand the ACT

KSN&C, and I suspect a lot of folks, got a note from Prichard Committee honcho Bob Sexton today regarding the ACT test results recently kicked around in the press.

The media coverage, and occasionally comments from school officials, badly confused what the ACT is and is not and how scores should be used and should not be used.

To bring some scholarly understanding to this misinformation about ACT, Ben Oldham, Distinguished Service Professor at Georgetown College, has written the attached statement.

Here's Ben's article:
September 24, 2008


Adding Understanding to the ACT scores


Ben R. Oldham
Distinguished Service Professor
Georgetown College


The recent release of statewide ACT scores has created a lot of discussion about the quality of Kentucky schools. It has been reported that approximately 43,000 Kentucky juniors earned an average of an 18.3 composite score out of 36 on the ACT. Kentucky is one of just five states that requires the ACT for all high school juniors. It should be noted that the American College Test (ACT) is a highly regarded test developed by a cadre of some of the best measurement professionals in the world and is used by a number of colleges as one selection-for-admission measure.

Extensive research has been conducted that suggests that the ACT is a significant predictor of freshman college grades. The ACT is designed to predict college success. My research suggests that high school grade point average is a similar predictor of college success. Since the ACT is administered to all Kentucky juniors, there is a tendency to over-interpret the results as a measure of the success of Kentucky schools. The successes of Kentucky education reform are inevitably brought into question.


Other evidence tells a different story. More students in Kentucky are taking AP exams and more students are earning college credit through the Advanced Placement program than ever before. These are standards-based exams. The purpose of these tests is to determine, by following a tightly structured curriculum, if students earn high enough scores to earn college credit while still in high school. Teachers know precisely what should be taught and through their excellent instruction more high school seniors earned college credit.


The Kentucky Core Content Test (KCCT) is administered throughout the grades of Kentucky’s public schools. Like AP tests, the KCCT assessments are standards-based exams. Teachers in Kentucky’s schools teach from a core content that defines what Kentucky students should know and be able to do as they progress through school. Like the AP test, the KCCT is designed to precisely measure how well the students have mastered the defined curriculum. The categories of novice, apprentice and distinguished are used to define the achievement of students. Its purpose is to monitor the growth of schools toward a Commonwealth goal of the average student achieving at the proficient level by 2014.


It is desirable for large numbers of students to achieve at the highest levels of achievement on both the AP and KCCT tests. Having small and reducing numbers of students at the lowest levels of achievement is also desirable. Here is where the difference with the ACT and standards-based tests lies. The ACT is a norm-based test. It is not designed to determine what students know and are able to do like the AP test or KCCT test. A norm-based test compares a student’s performance on a bank of test questions with students in a comparison group; in this case a national but not nationally representative comparison group since its purpose is for the college-bound. By design, the ACT spreads student scores to assist colleges and universities in making admission and scholarship decisions. When the ACT was developed, the average score was set at 20 regardless of the academic achievement of those in the norm group. If it were the case that everyone in the national comparison group scored at a high level, the mean score would be 20. If nearly all scored at a low level on the ACT, the mean would be 20. The purpose of the ACT is to assist colleges and
universities in making admission decisions. By design it separates students into a range of scores from the 1st percentile to the 99th percentile regardless of the pure academic achievement.


Because it is administered to students across the country, the ACT is designed to be insensitive to curriculum to not give an advantage to any particular curriculum. This is another major difference. Both the AP and the KCCT are built around a tightly defined curriculum. Because the ACT is insensitive to school curricula many employ the test-taking strategies to artificially inflate test scores. It should not be quick and easy process to improve test scores because it does not reflect true improvement. However, given a defined curriculum, public school teachers have done and will continue to do an exemplary job educating students toward a common goal. Are there improvement strategies that can be employed? Absolutely, but the ACT does not contribute to these strategies because the ACT must, by design, separate students to assist colleges in selection decisions.


Teachers, parents, principals, superintendents, board of education members, and most importantly students must not overlook this purpose. While it is important, schools should not be evaluated using a tool that is insensitive to the core content and is designed to differentiate between the higher-achieving college-bound.


If the purpose of Kentucky public schools is to prepare all elementary and secondary school students for college, then the core content followed by schools needs to be adjusted with significant input from college professors to include things like the thoughtful analysis of data and ideas, explaining and demonstrating math solutions and a solid foundation in the college general education curriculum. Regardless, the evaluation of the achievement of the core content must be measured by a test that determines the success in achieving that curriculum rather than a norm-based instrument, like the ACT, that merely compares a student’s performance with college-bound students nation-wide.

2 comments:

Anonymous said...

I see a great number of problems with Oldham’s analysis – too many to discuss here. But, here are a few comments.

Oldham pushes out-of-date thinking that the ACT is only a norm-referenced test. The ACT did start out more or less that way, years ago, but the addition of the benchmark scores, which are empirically developed from actual college student performance to indicate a good probability of college success, provides a criterion-referenced element today, as well.

Furthermore, the CATS scores were adjusted last year with the highly controversial concordance table process, a norming process. This norming process was necessitated by the unstable and generally inflated scoring that resulted when the standards were reset, again. KSN & C has written about this. Even the KDE admits it was a norming activity. Should we throw out CATS because it is also normed?

Even before the significant scoring standards inflation that occurred in the latest CATS biennium, scoring criterions in CATS have never been stable over time. A comparison I recently made to the NAEP achievement level scores (http://www.bipps.org/pubs/2007/CATSinDecline.pdf) shows the performance required to reach KCCT “Proficiency” in math, reading and science has actually been getting lower and lower over time. It’s been like a stealth norming process – a fairly steady regression to a fairly low Kentucky mean. Such a process isn’t a feature of real criterion referenced tests, at least not ones intended to be used over extended time periods running from 1999 to 2014 like CATS is using. The evidence is quite clear; the KCCT truly cannot be considered standards based because the “standards” have steadily eroded over time.

Oldham admits the ACT is a good predictor of college success. But, he misses a key point; many better paying non-college track jobs now require essentially the same skills that college freshmen need. This feature of our rapidly more technological society that means the ACT is more like a test of preparation for a decent life than just a college readiness exam. The ACT certainly didn’t start out like that, but it has grown into something more than it originally represented.

That leads to another Oldham implication that, somehow, KCCT is more relevant to what students need. I wonder how that can be. The KCCT is primarly created by Kentucky teachers who have limited exposure and experience in both higher education and business and industry. How, exactly, can this be a formula to create a truly relevant assessment for what kids will need in life? Perhaps I am missing something.

In contrast, the ACT is developed with considerable participation from college staff. The ACT, Incorporated also benefits from its long-standing relationship with business and industry through its experience with the Work Keys program. I don’t think the KCCT has anything remotely approaching this pool of experience in its development process.

The AP exam progress is certainly good, but the total number of students taking APs in Kentucky is still well below the number who are going on to higher education. Also, CATS offers no credit for schools that have good AP programs. CATS does not test to those educational levels, either. So, if APs are important, is it wise for CATS to ignore them?

Certainly, comparing the KCCT to the APs is a terrible stretch. APs are developed by nationwide teams of high school and college educators and feature high quality tests with commensurate quality scoring. In very sharp contrast, the KCCTs are an inward-looking, P to 12 only, Kentucky only affair which continue to suffer from far too little input from postsecondary instructors in the core college disciplines and which also suffer from continuing concerns about both the quality of questions and the scoring of same.

Richard Day said...

The ACT is a norm reference test.

More later perhaps.