Are you certified or certifiable? I may be (partially) to blame.


A certification is the objective, external verification of the ability to perform a collection of tasks/activities critical to a job role. (IBM)

CertificationMany professions require practitioners to be certified—often by passing a written exam. In information technology there are many, many types of professional certifications offered by industry groups as well as product vendors (IBM alone manages just under 500 certifications; click here for a look). Certification is very serious business, as many jobs hinge on holding particular credentials, and I think it’s safe to say that certification exams generate a certain amount of anxiety for the examinee.

But what about the examiner?

That’s a question I never considered until I was asked to join the team preparing a new certification for System z Business Analytics Solution Selling. I have to say, it was quite an intense, yet very rewarding, experience. And certainly worth a blog post!

There is nothing simple or easy about developing a certification exam. Roughly speaking, the process covers the following phases.

Know your victim (er, I mean candidate)

There are no “grades” in certification; it is a pass or fail exercise. The target is what is called the minimally qualified candidate (MQC)—a practitioner that has just enough skill to fulfill job responsibilities. The test is built under the assumption that the guru will sail through and the clueless will fail; the goal is to set the test somewhere in between these two extremes and reveal the MQC.

Set the expectation

The first phase of our process, then, was setting out exactly what that level of knowledge is that enables someone to qualify as a System z business analytics solution seller. We developed an outline of the skills we felt were essential to the job, devised knowledge categories around these skills and established a baseline number of questions for each. This became the blueprint for the test question writing.

Understand A, B, C and D

I did not previously know, but in hindsight am not surprised, that there is an entire field of study, psychometrics, dedicated to the theory and technique of human measurement (and I have no doubt you must be certified to be a psychometrician!). Before we wrote our first question we had to receive a crash course in the art and the science—psychometrics—of question writing.

ExamIt has been determined that the ideal certification question contains four multiple-choice responses (less makes guessing too easy; more turns it into an exercise in logic).

Having a single, absolutely correct answer (as opposed to “all of the above” types) has proved to be the best measure of understanding level. The other three answers must be plausible, attractive and absolutely wrong.

We were reminded—many times!—that our goal was to discriminate between those who really know and those who think they know. After reading the perfect question, the MQC should already have the answer in mind and simply need to find it in the list of responses.

Write, rewrite and write some more

Writing the first third of the questions is relatively easy. The second third requires more effort. The final third is excruciatingly painful! At the end of the process we were pouring over technical manuals and IBM Redpapers to find those last few questions. Many times the hardest part of the process is coming up with the wrong answers (known as distractors). Thinking up alternatives that are plausible, attractive and defensibly wrong is a lot harder than you would think.

Of course our dedicated psychometrician was always on hand to gently guide us whenever we broke one of the cardinal rules of question writing (no negatives, no trivia or memorization, no deliberate tricks and so on—there are a lot of guidelines!).

Set a passing grade

We had to write enough questions to support multiple versions of the exam (one of the many safeguards against cheating). Unfortunately for us, the process does not end when all the questions have been written. The final phase is determining the passing grade or cutoff score.

According to psychometrics, the line between competence and incompetence cannot be arbitrarily determined; it must be empirically justified and defensible. IBM likes to use an industry-standard process called the Modified Angoff Method for this purpose. Our team was required to individually answer each question and assess, out of an imagined field of 100 MQCs, how many we thought would be able to answer the question correctly. The results were collated, outliers and anomalies were discussed and resolved, and the passing grade was set.

Mission accomplished!

CertifiedIf you need to be certified in System z Business Analytics Solution Selling, you may be tempted to buy me a drink (or two or three) should we meet, in the hope of getting an edge in taking the exam. While I will appreciate the drinks, be aware that I signed a non-disclosure agreement not to reveal any of the details of the exam, so your efforts will fail!

But I do wish you the best of luck—and whether the exam leads to your certification, or becoming a certifiable wreck, please don’t blame the writers!

Paul DiMarzio has 30+ years experience with IBM focused on bringing new and emerging technologies to the mainframe. He is currently responsible for developing and executing IBM’s worldwide z Systems big data and analytics portfolio marketing strategy. You can reach Paul on Twitter: @PaulD360.

Redbooks Thought Leader

Smarter Computing Analyst Paper - HurwitzTo effectively compete in today’s changing world, it is essential that companies leverage innovative technology to differentiate from competitors. Learn how you can do that and more in the Smarter Computing Analyst Paper from Hurwitz and Associates.

Subscribe to the Smarter Computing Blog

Recent Posts

Creating a fast-track for the hybrid cloud

Setareh Mehrabanzad

Last month, IBM Systems unveiled new solutions for creating an agile hybrid cloud architecture by enabling VMware’s vRealize Automation Platform for IBM Power Systems and IBM z Systems. Today, IBM Systems and VMware are introducing expanded capabilities this week at VMworld 2015 Europe in Barcelona.

Continue reading

Opening up new options for hybrid cloud

Setareh Mehrabanzad

The creation of hybrid clouds has become a hot topic among IT organizations. Companies have come to realize that the best systems almost always include both a public cloud and a private cloud, but that is often fraught with technical difficulties. IBM is unveiling new capabilities to bring down barriers and allow IT departments to choose the best solution, regardless of platform.

Continue reading

Leave a Reply

Your email address will not be published. Required fields are marked *

* Copy This Password *

* Type Or Paste Password Here *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>