Why I'm Testy About Tests

Standardized online tests are bunk, and the companies that use multiple-choice tests as a sole determinant when hiring people are likely staffed with very lazy people. I don’t say this just because I am a poor test taker, and I don’t say this just because my 40-year-old brain is no longer nimble and good at memorizing facts. (However, it is full of too many useless facts already.) Standardized online tests as a sole means of screening people is problematic and is no better at determining potential for success than grades in college. (Many a C- student millionaires and leaders will agree with me here—Bill Gates, Michael Dell, George Bush, and John Kerry, included.)

A True Story

A local peer of mine recently wrote to lament a college student he hired as a developer. From his letter, he clearly was impressed by her academic record, the fact that she had earned a scholarship to an excellent university, and that she seemed articulate and bright. By all accounts, she was a superstar waiting to shine. This employer also mentioned that his technical ability—read, his ability to screen her real skills—was somewhat wanting. What was the result?

This academically successful person is failing miserably at her job. Her peers are saying she’s in over her head. Her boss is seeking advice from several corners, including researching her precise academic curriculum in detail, and doesn’t know how to proceed. Yes, maybe the boss needs a refresher on hiring strategies, but that’s not why I wrote this.

This young person should have been a star. What happened? The answer is that grades aren’t everything. Grades reflect an ability to take tests and remember facts, but many classes aren’t based on critical thinking and problem solving—they are based on timed fact recollection. This suggests that test scores may be an element for evaluating potential but not the only one—and probably not the most important one. Actual accomplishments are the best means of evaluating potential.

Another True Story

Recently, however, an agency asked me to take a C# test for a prospective customer. Being a little curious, I decided to take the test. This particular test was 40 multiple-choice questions; the time for each question was three minutes; and you could use online materials and books in the time allotted. I took the test using only extemporaneous recall. Guess what? I scored in the 53 percentile. Out of 7,000 tests takers, I scored as well as about half.

Granted, had I employed Google, Visual Studio’s help documentation, and the compiler, I probably would have scored somewhat higher, but what was really tested?

One way to look at the results is that I got half the questions right—about 20 out of 40—so I know 50 percent of the facts. Say there are 10 million facts, then according to the test results, 3,500 people and I know 5 million of them. That’s pretty good. But what good are facts alone?

Here is another problem. In practice, no time limits exist for solving problems in software—for the most part. If one spends an hour or two finding a fact or working out a problem, no one cares. If one is stuck after an hour and reaches out to online columns like mine or emails friends who may know, then the problem ultimately is solved. Who really cares what the compiler switch is for range checking when a 10-second search of the help documentation will provide the answer?

Building software is about problem decomposition, solution composition, tenacity, problem solving, good tools, organization, and having the money and time to finish. Facts are the least of anyone’s problems. Seventy-five percent of all projects in our industry don’t fail because of an ignorance of facts; they fail because of poor planning, flagging budgets, or lousy analysis, designs, and specifications. On a team of 5–40 people, the team members can rattle enough facts off the tops of their heads to fill an encyclopedia, yet projects routinely fail.

The Best Interview I Ever Had

During these interviews, you are asked about facts. You are instructed to think out loud, and if you don’t know something then you’re allowed to think and discuss the problem critically. You are asked to solve problems like why are manhole covers round? How would you move Mt. Fuji? How would you go about counting all the stars in the sky? The following was my question:

Given three barrels (one marked apples, one marked oranges, and one marked apples and oranges) that are all mislabeled, you can pull only one piece of fruit from one barrel. How would you go about determining the proper labeling of the barrels?

(The answer can be had in a Sesame Street song: [singing] one of these things is not like the other.)

During the course of the morning, I was asked hundreds of facts. But I was also asked about my accomplishments, best and worst projects, hobbies, who I was as a person, what I had done, and what I’d like to do yet—not a single multiple-choice question.

To get hired at Microsoft (at least for the job I was interviewing for), one had to get a unanimous thumbs-up. I got a thumbs-up from everyone who interviewed me but one developer. He said I seemed too much like a manager, which from developers is not a compliment. My secret belief is that he wasn’t impressed that I didn’t know what string-interning was. (Damn! Foiled by a fact again.)

The worst part about the string-interning question is that it was a pre-screening question. I’d been asked the same question during the pre-screen interview, and I forgot it when asked during the Redmond interview a week later. (String interning is how strings are stored internally to conserve memory and is why strings are immutable in .NET.) The truth: during the pre-screen I determined that string interning was beyond my control and a truly irrelevant fact except for the .NET compiler writers, so I immediately discarded the factoid.

There are so many factoids that unloading most of them daily is a necessary survival skill just to keep some room in one’s brain. Facts are one of the reasons I write books. I write all the stuff I know at a given time in a book, and then I proceed to forget most of it. If I need the fact again, I pick up one of my own books and search the index. In this way, my books become the permanent overflow storage of my brain, and it is an added bonus that I get paid to store the overflow.

Testy About Testing

I have to admit I am a little disappointed that I didn’t score higher than everyone else on that particular test, but that’s probably my ego talking. I am wise enough to know that I am not the smartest person on the planet and definitely not the best test taker. I used to believe I had a practically photographic memory, but that was many winters past.

I will definitely return to my insistence on not taking tests nor traveling for onsite interviews. (I work with too many companies to take tests and travel for onsite interviews as pre-conditions for collaboration.) The reason won’t be just poor test-taking skills. The truth is that by using the resources available during the test, my score would be improved noticeably. I won’t be taking tests because they are impersonal, impractical, inefficient for evaluating potential, and they promote laziness in personnel departments. When I work for a company, they probably won’t want me to be lazy, and I don’t want them to be lazy either.

About the Author