Items are comprised of a “stem,” which is the question; and “options” that should contain a single best answer and some wrong answers. These components are divided further with the stem containing a vignette (a scenario) and a lead-in to the options. The options will contain a “key” (the right answer) and “distractors,” which should sound plausible but be incorrect:

So, proper Items will have a stem comprised of a vignette and lead-in, and options comprised of a key and distractors. Now we have enough information to start spotting some bad questions.

Shapely Form
In well-written items (questions), the stem is long and the options are short. The writers should be putting the relevant information in the stem, not in the options. This is called a front-loaded or shapely form. Long options are often a sign the stem is unfocused, or that relevant information is missing from the stem.

Cover-the-Options Rule
You should be able to answer the question just by reading the stem. You should not have to read the options to know what the question is asking. So, if you were to cover the options with your hand, does the stem still make sense and allow you to answer the question? If not, you are using poorly written questions--unlike the ones you will see in the exam.

Likewise, beware if you see practice exams with more than four options. Or options that say “All of the above” or ‘None of the above.” These are simply not used in PMI certifications. You would not practice for a tennis competition by playing squash, so you should not practice for a project management certification by using the wrong type of questions.

Instructional Design
Next, let’s dig deeper and learn a little about instructional design so we can identify some further instances of bad questions. “Blooms Taxonomy” is a framework that describes the levels of thinking used when learning new material:

Everybody goes through these stages as part of learning and being able to use new skills. The stages can be divided into two main categories, recall and application:

Good exams have a combination of recall and application questions, with a preference for the application of knowledge. An example of a question that tests recall only would be:

The communications management plan is a document that includes descriptions of:

1. Project-level performance reports
2. Activity-level status reports
3. Stakeholder communication requirements
4. Responsibility assignments

Here we just need to recall that a communications management plan contains information about stakeholder communication requirements. These questions are easier to write, so people generating hundreds of questions for online exam simulators create a lot of them. Unfortunately, PMI exams use predominantly application-type questions that first outline a scenario and then present a question. These elevate the question from recall to application. For example:

Your team planned to complete six stories in the current sprint. But as you reach the end of the sprint, only four of them are done. What should you do?

1. Return the remaining stories to the backlog for re-planning.
2. Ask the product owner to extend the sprint.
3. Work on the remaining stories when you can fit them in.
4. Schedule the remaining stories at the start of the next sprint.

Here we have to first interpret the scenario analyzing the situation, then evaluate the best response. So another warning sign to look for are too many recall-based questions. The majority should be scenario-based application questions.

As we take more and more sample tests, our ability to answer these types of questions increases, but so too does our ability to spot the right answers amongst poorly written distractors and score higher than we really should do on low-quality, poorly written sample tests. So, unfortunately, your increasing performance may not all be attributed to a sponge-like brain for project management knowledge; some might be due to an ability to quickly spot weak wrong answers. In the exam-writing world, this is called “testwiseness.”

Testwise Signs of Poorly Written Questions

1. Clanging
A word or phrase in the question also appears in the correct answer: “A Release Map is commonly used to…”

A) Manage WIP
B) Plan Releases
C) Assess Risk
D) Coordinate Users

Here, the term “Release” appears in the question and the right answer.

2. Grammatical Clueing
In the question, the word “an” signals that the answer begins with a vowel: “A short period to undertake work is called an…”

A) Backlog
B) Burndown
C) Iteration
D) Persona

Here, only C) makes the question grammatically correct. Most likely, the question writer created the question with the right answer and generated some distractors without checking the grammar.

3. Syntax Clueing
Similar to grammatical clueing, the wording of the question indicates that the answer is a noun, and one option is the only noun: “The Taskboard” versus “Prototyping” or “Decision Making.”

4. Non-Parallel
Having some distractor answers that are not the next steps in a process: “A retrospective normally happens after the:”

A) Iteration
B) Risks
C) Backlog
D) Requirements Gathering

Here, “Risks” and “Backlog” are not steps in a process and so can be eliminated.

5. Absolutes
People know few things are absolute, so it is easy to rule out “Always” and “Never” options: For instance, “All teams need coaching.”

6. Longest Option
If one option contains more detail or information, maybe the question writer was trying to better qualify the answer to make it completely correct. This can be a signal that other, less carefully crafted options are distractors.

7. Convergence
When the correct answer has the most elements in common with the other options:

A) People and Tools
B) Processes and Techniques
C) Processes and Tools
D) Principles and Tools

Here, “Processes” occurs twice and “Tools” three times, making option C) a good bet for the right answer, since “Techniques” and “Principles” only occur once and were likely added as distractors.

8. Numeric Convergence
Likewise:

A) 6 points
B) 60 points
C) 30 points
D) 15 points

The “6” shows up twice, likely because the question writer typically creates the right answer first then tries to make plausible distractors by basing them on the right answer.

9. Enemies
The answer to this question was in another question! This is what happens when people try to generate too many questions based on the same topic.

These examples above were kept simple and obvious to illustrate the flaws, and written in recall format to keep the example questions short. However, all the same mistakes are can be found in longer, situational questions with varying degrees of clarity.

For an excellent overview of question writing guidance, see this presentation from the University of Michigan Medical School. It is designed for medical doctors and includes an eight-question sample. Non-medical professionals typically score the “monkey click” average of 25% on the first pass and then a scary, passing grade of 80+% once they are aware of the question flaws. The problem is we may not realize how much we are relying on subtle clues to answer questions correctly until we attempt questions without these flaws.

Answering Situational Questions
Now that we are better equipped to identify poorly written sample questions, we can focus instead on answering well-structured questions that likely do not have these testwise flaws and are largely in a situational structure:

“You are working at a medical laboratory that traditionally tested 25 swab samples per day and typically worked on 100 swab samples at any given time. Improvements to the testing process have reduced the cycle time to 3 days and the WIP to 90 samples in progress. What is the percentage of improvement in throughput?”

This is a nasty two-part question, but it is a good example to study. First, the question provides us with a scenario to interpret and then tests our application of the relationship between Throughput, WIP and Cycle Time. It does not ask us to recall the equation but requires us to know that Throughput = WIP / Cycle Time. With our new, improved process…

WIP = 90 and cycle time = 3 days, so throughput = 90/3 = 30 samples per day

Once we’ve done this calculation, we still have to find the percentage of improvement over the old throughput of 25 samples per day. So we subtract the old throughput from the new measurement (30 – 25 = 5), and then divide the difference by the old throughput (5/25 = 0.2, or 20%). Therefore, increasing our throughput from 25 samples to 30 samples per day is a 20 percent improvement.

The process to successfully answer these tricky situational questions is:

1. Find the question in the question. Read the question through to understand the scenario, then recheck the lead-in part of the question—that’s the final part before the options are presented that asks the specific question to be answered. In our example, it is: “What is the percentage of improvement in throughput?”
2. Apply your knowledge and skills, tools and techniques to answer the question. Now that we know it relates to throughput, WIP and cycle time, we have to apply our knowledge of those topics to answer the question asked. Remember to answer the question that is specifically asked, not some other question that could be asked with this set of inputs.
3. Answer the question. Answer the question without looking at the options presented. It is easy to be drawn to true statements that do not answer the question asked and then stop thinking. Instead, create your answer to the question asked.
4. Find the option that best matches your answer. Now look at the options and identify the one that is the closest match to your answer. This sounds a long-winded, time-consuming way of answering a question when we know one of the answers is correct, but finding an option that matches your answer is good validation and means you have not been drawn to an option.

Summary
To make the best use of you study time and avoid unnecessary stress (either while preparing or in the exam), take care to ensure you are using high-quality simulation questions. Learn how to spot poorly written questions that are not indicative of real exams and use the most realistic questions you can. Get plenty of practice answering situational questions that require you to apply your knowledge to project scenarios.

Then, assuming you have the appropriate experience and knowledge, you should be well prepared to visit the test center with confidence to pass the exam and gain your next credential.

(Note: I first published this article on ProjectManagement.com Here)