As games get tougher, the tough get gamer!

There's a growing anti-videogame mentality in this country. While it's typically directed against "violent" video games (like this Tennessee effort Glenn Reynolds highlighted recently) people forget that (as Glenn reminds elsewhere, that video games can "save your life."

I think it's worth noting that familiarity with video games (and the video game mentality) might make or break your career in civilian life.

While I didn't grow up with video games, if I had a kid today, I'd want him to be thoroughly familiar with the technology and the mentality, because like it or not, it's creeping into everything.

Even something as dry and supposedly mundane as the GRE. (That's the Graduate Record Exam.) Gone are the days in which I grew up when standardized testing consisted of a group of people all taking the same test at the same time. It used to be that every single test taker got the same identical test booklet, and all you needed to to take the test was a pencil. A brain, of course, would help. But the whole idea was to compare in very linear, inherently objective terms, how well the different brains performed when faced with the same test. Obviously, few people know the answer to each question. How much time to spend on a tough question, whether to guess or leave the hard questions to return later is time permits (or if a later question suddenly triggers the memory to supply the answer to an earlier question) -- this all tests the ability of a test taker to use judgment under pressure. But still, all test takers were treated the same.

That was in the old days.

Today's GRE is a video game. They don't call it that, of course. It's called a "Computer-Adaptive Test." Instead of having a booklet full of the same questions, the test morphs into an ever-harder challenge depending on how well the test taker performs. It is designed so that no one can "ace the test" as they might have in the old days. Nor can harder items be skipped over and returned to later.

Because the CAT works like a video game, strategies have developed accordingly.

With each examinee receiving a different set of questions, there can be perceived inequities.

Examinees are not usually permitted to go back and change answers. A clever examinee could intentionally miss initial questions. The CAT program would then assume low ability and select a series of easy questions. The examinee could then go back and change the answers, getting them all right. The result could be a 100% correct answers which would result in the examinee's estimated ability being the highest ability level.

Students in Asia quickly learned that the questions could be shared later, which gave them an advantage. Apparently, this has caused ETS to rethink the game, so they're revising the test to make it more "linear":
Many graduate students have cheered the GRE board's decision to replace the computer-adaptive test format with a linear testing model. The computer-adaptive test calibrates the difficulty of each question based on how the test taker had answered the previous question. In the linear approach, every test taker receives the same questions in identical order.

ETS officials said they are no longer administering the test in the computer-adaptive format because examinees in Asia were posting test
questions from the GRE on Web sites after they had completed the test. In the computer-adaptive format, anyone who takes the test can have the same questions as someone else who took the test at an earlier date. As a result, some test takers had gained an unfair advantage over others who took the GRE before them.

Well that's a hell of a note.

Just because of some clever kids in Asia, the ETS is getting rid of Computer Adaptive Testing? No more video game? Well, not until 2007:

The biggest change will be a shift from a computer-adaptive test to a computer-based linear test. This means the test will be much more standardized.

Currently, the GRE adapts to your performance so the question pool varies by test taker. You start with a question of medium difficulty. If you answer it correctly, the testing engine serves a harder question and the more correct answers you give, the more difficult the questions get. The new computer-based format means the text would be fixed. Everyone takes the exam on the same day at the same time and receives the same questions in the same order, much like the SAT taken in high school.

While ETS claims that the test won't be changed until 2007, it hasn't happened yet, and skepticism is being voiced:
Graduating students planning to take the Graduate Record Examinations this fall will now have extra time to prepare for the new format, which was announced last year.

Educational Testing Services announced earlier this month that the revised version of the GREs will be delayed until the fall of 2007.

The implementation of the new format of the exam, which is required for admission into graduate school, was supposed to begin in October 2006.

The changes will be significant, according to Liz Wands, national director of graduate programs at The Princeton Review.

She said the new test is changing from a computer-adaptive test to a computer-based, linear test.

Current forms of the test challenge students — the difficulty levels of the questions increase — and the scores are then calculated based on the level difficulty reached.

The revised exams, however, will provide the same questions to all students.

But lovers of the old non video, "fair"-style linear test -- don't hold your breath waiting for things to hurry up and change. It might take longer than ETS is claiming. While ETS is as cagey as the NSA (and probably tougher to pin down), reading between the lines, it's clear that the test is not popular in some circles:
Despite claims by the ETS that the new exam will more accurately determine how students will perform in graduate school, Wands said The Princeton Review disagrees, adding that the changes came from graduate schools — clients of ETS — which wanted a better, more accurate test.

“The GRE is changing based on this,” she said. “But they failed to come up with a more valid test.”

She is not surprised about the delay after several changes in the last few months.

“The main reason for the delay is because their field testing has not been successful,” she said. “We [The Princeton Review] would not be surprised if it is delayed past 2007, but it is too early to tell.”

I wonder whether there might be fear of litigation, because it strikes me that there's something unfair about a test that changes depending on the taker. The theory that slower minds should get easier tests, while faster minds get harder tests sounds to me like video-based socialism. Because it's done by computer program, it invites the smarter minds to figure out how it works, and beat not the test, but the game.

I can't help but wonder whether ETS has any inside data suggesting a correlation between high scores and the video game skills of the test taker.

Philosophically, I know that all tests can be seen as games. But there's something utterly intriguing to me about seeing tests that appear to be games by their very design.

I hadn't known about the CAT GRE tests until recently, and I understand why someone who came up under the old school of thinking might be incredibly pissed off by the new way of testing.

Frankly, I don't know what I think, and I'm glad I don't have to take the test, either in its old format, the new video game format, or the new "return" to the old linear format.

I hate to say this, but I think my view of the test would probably vary according to how well I could manage to beat it.

Hmmm....

Is morality involved? I mean, if you can deliberately increase your score by pretending to be more stupid than you are, that can't be called cheating, can it?

I mean, isn't this just a game?

MORE: Commenter Euler Function explains why pretending to be more stupid than you are would not work. (And assuming that there's no way to correct earlier answers, he's probably right.)

Also, some apparent confirmation that the credibility of the GRE among admissions officials has suffered:

Timothy Blackman, associate dean of students in the Social Science Division, said that he did not think there was any consensus among graduate officials at the University of Chicago on whether or not the updated GRE will be given any more weight than the current GRE in admission to graduate school.

Blackman was reluctant to comment on ETS’s claim that graduate admission officials would perceive the new GRE as a more valid examination of an applicant’s critical thinking skills. He noted that besides GRE scores, a candidate’s transcript and letters of recommendation also take high priority in applications for graduate school.

Louis Tremante, Senior Advisor in the College, however, recalled that several years ago officials from the Biological Sciences Division had said they felt that the computer-adaptive test format of the GRE made it a less reliable examination.

“My impression was that they subsequently gave it less weight,” Tremante said.

ETS has a lot of power over a lot of people's lives.

I assume there has to be some system of accountability, but I'm afraid I'm outside my field of expertise on the field of GRE testing.

posted by Eric on 03.15.06 at 07:54 AM





TrackBack

TrackBack URL for this entry:
http://classicalvalues.com/cgi-bin/pings.cgi/3408






Comments

you can deliberately increase your score by pretending to be more stupid than you are

No so. The test essentially does a binary search to find your score. First it asks a medium level question. If you get it right, you move on to harder questions, and the test sets a lower bound on your score. If you get it wrong, you move on to easier questions and the tests sets an upper bound on your score. (This is not completely accurate because this bounding is done based on several questions). As a result, the test each person takes is shorter than the full "linear" test. The assumption is that if a person gets all the medium level questions right, he is unlikely to get the easy ones wrong, so there is no need to torture somebody who knows calculus with basic algebra. On the other hand, if somebody fails the medium questions, that person is unlikely to get the hard questions right, so no need to waste time asking those questions.

Euler Function   ·  March 15, 2006 04:43 PM

I did say "if" but assuming you're right, then does this mean it's inaccurate to say that "a clever examinee could intentionally miss initial questions" and thereby increase his score (as the site stated)?

Eric Scheie   ·  March 15, 2006 05:16 PM

I see my error.

If you can't go back and change incorrect answers, then you're right. Playing dumb would not work.

But doesn't that mean that if you got off to a slow start, you could cement yourself into a lower score range and not be able to get out of it -- even if you knew answers to some of the tougher questions?

There's something about not being able to correct errors in a test you're still taking that bothers me. Couldn't this cause a nervous or obsessive person to completely blow the test?

Eric Scheie   ·  March 15, 2006 06:04 PM

The adaptive test is definately riskier. When I was preparing for the GREs, my biggest worry was making stupid mistakes (e.g. dropping a minus sign during a calculation) rather than knowing the material which was easy. However, I found the GREs less stressful than the SATs.

First of all, you only need to get a decent score not the best score. You take the test when you want, (personally, I'm not at my test-taking best at 7am). You decide how long a break you need between sections. (I didn't take one because I find them more nerve wracking than useful). You can take the test alone - so you're not surrounded by 100 equally nervous people. And the whole test is much shorter (mine took 30 minutes, but that was before the essay).

While I'm not convinced that the GREs are really an appropriate test to determine your potential as a grad student (the math section is an insult to all math/science majors while the verbal section does the same for humanities people), I definately prefer the adaptive computer version to the old fashioned test.

Euler Function   ·  March 16, 2006 08:26 AM

I've never taken the GRE and I don't plan to. My interest in the CAT issue stems from the fact that people are not informed about how it works, and the older students get blind-sided. In one case with which I'm intimately familiar, a math whiz (math SAT scores in the upper 700s) was applying to grad school and took this test. Panic immediately set in as soon as the applicant realized the machine was behaving in a challenging manner -- as if it was deliberately upping the ante with each successfully answered question. This was a huge contrast to the linear paper SAT which had all the questions laid out. The CAT GRE score was in the low 600s -- and this student swears that the result would have been very different had it been the old fashioned paper and pencil linear test. The student accepted the ETS claim that no preparation was necessary for those already proficient at math. Proficiency in math and proficiency in old style testing means nothing if you have a problem with test that changes the game on you.

I think students from the old school should be warned accordingly.

But warned of what? An adaptive test that's being changed yet again?

Who elects the people who have so much say-so in the future directions of so many lives?

Eric Scheie   ·  March 16, 2006 09:30 AM

people are not informed about how it works

Ugh. I hate defending ETS and the GREs because it is a stupid test, but the above statement is just not true. ETS not only tells you how the GREs work on their website, they publish practice questions -- with the implication that you're supposed to practice. Everyone I know who took the GREs spent the summer before practicing. This is actually where my beef comes in.....I majored in Math & Computer Science and had to waste a summer practicing solving simple algebra and geometry problems quickly and accurately. The test is easy in the sense that the material is not complicated, but it is annoying and doesn't really say much. (It's sort of like your gaming analogy -- pressing up-down-left is not hard, but doing it quickly and at the right time takes practice). The GRE subject tests are even worse -- they pretend to test you in depth on the subject you majored in; in reality they are written by non-experts and are horribly out of date (the CS GREs required an in depth knowledge of Pascal, a language so old that I've never seen *anything* coded in Pascal).

Anonymous   ·  March 17, 2006 11:01 AM

Look, I didn't take the test, and as I said I'm no expert. I didn't need to do much checking to see that ETS is not being up-front about the nature of this test, or the special skills in test taking which are required. Quite the opposite.

The GRE test manual -- the one currently available at the GRE site --

http://www.ets.org/Media/Tests/GRE/pdf/Math.pdf

-- does not make it at all clear that this is anything but a conventional test. Read page 5.

"A general review of your college courses is probably the best preparation for the test."
Under "test-taking strategies," the test is described as one to be taken on answer sheets with pencil and paper, and with the usual opportunity to go back and change answers.

The point is that proficiency at Math (if you understand the subject material sample questions in the practice book then you can deem yourself proficient at Math) does not guarantee proficiency at taking this form of test.

I think it's deceptive advertising, and maybe more.

Eric Scheie   ·  March 17, 2006 11:26 AM

NOTE: To get to the above, I simply went to the home page at ets.org, then clicked "GRE©" then clicked "Prepare for the Test(s)" then "Prepare for a Subject Test" which took me to "Free Subject Test Preparation Materials" which then listed the .pdf file, called "Mathematics."

http://www.ets.org/Media/Tests/GRE/pdf/Math.pdf

Eric Scheie   ·  March 17, 2006 11:35 AM

The link you gave is for the mathematics subject test. This is a traditional paper exam, not a Computer Adaptive Test. It isn't what most people are talking about when they talk about the "math" GRE. The test booklet you linked for is intended for people who are entering mathematics PhD programs.

What people are talking about (and what is required by the vast majority of graduate programs) is the quantitative section of the general test. The website for it is very explicit about what format the test is in. They also send you a CD in the mail before you take the test. The CD has two sample tests that are EXACT replicas of the CAT. I really don't see how anyone could go into the test not knowing what a CAT is, to be honest.

Chris   ·  March 18, 2006 04:40 PM

You have a good point, Chris, and I am showing my lack of expertise with the GRE. (Again, I've never taken it and know little about it, but the CAT strikes me as a little weird, because of a lack of uniformity, especially for something like math.)

The student I am talking about did not practice with CDs, but only reviewed paper materials -- not realizing that strategy mattered as much as it turned out that it did (I got this second hand and have no idea whether the CDs were sent out. This was a couple of years ago.) While making the assumption that overall knowledge was all that's needed may not have been a good idea, is this the best way to test overall math knowledge? If you're a bit slow on a couple of initial geometry questions, you might never be able to demonstrate your expertise with tough calculus questions, because you could be relegated to a lower level.

Anyway, people who've never taken a CAT test before can be in for a rude awakening if they make this mistake.

I think that people should be given an option to stick to the old-style paper tests, especially for math.

Eric Scheie   ·  March 18, 2006 05:49 PM

One last point: ETS's web site also has a study file which "Contains information on the mathematical skills and concepts likely to be tested on the GRE General Test." (I believe that's the test you mentioned.)

Downloadable here:

http://www.ets.org/Media/Tests/GRE/pdf/GREmathPractice.pdf

It's another paper booklet, and if you studied it, you wouldn't be prepared for the CAT. I haven't read every word, but no CAT warning stares out at me. People who are super confident in their abilities and think that looking over these problems in a book (which is exactly what happened in the case I described) can get screwed.

Eric Scheie   ·  March 18, 2006 05:57 PM

Even if we assume that all people were properly prepared for the CAT test, does it really make sense to test people according to their ability? If my understanding of the test is correct, someone who can only answer easier questions ends up only getting easier questions, while someone who can answer the tough questions gets only tough questions.

This is fair?

How can the approach of easy questions for stupid people and tough questions for smart people be called comprehensive testing? Or standardized? What's the standard?

Different standards for different people strikes me as pretty close to no standard at all.

Eric Scheie   ·  March 18, 2006 06:59 PM


March 2007
Sun Mon Tue Wed Thu Fri Sat
        1 2 3
4 5 6 7 8 9 10
11 12 13 14 15 16 17
18 19 20 21 22 23 24
25 26 27 28 29 30 31

ANCIENT (AND MODERN)
WORLD-WIDE CALENDAR


Search the Site


E-mail




Classics To Go

Classical Values PDA Link



Archives




Recent Entries



Links



Site Credits