The True Cost of Free Exam Prep. Questions
November 20, 2016
Most people taking a project management certification exam use sample tests. Whether it is a PMP exam, ScrumMaster, CAPM, PMI-ACP, PgMP or many others, there are plenty of online options for getting familiar with the format and determining if you are ready to sit the exam proper.
Unfortunately, like all things found online, the quality and relevance varies considerably. If we are just looking for funny cat videos, the occasional shaky video filmed in portrait mode is annoying--but easily skipped and not the end of the world. However, bad exam simulators can give a false sense of security--or a false sense of insecurity--and generally do not prepare you at all for what the actual exam will really be like.
Before getting trained and involved in question writing for PMI and professional training companies, I had no idea about the science behind good multiple choice questions. Now, I cannot help but notice poorly written questions. Even if the test is free, if it tests material not in the exam, it can generate unnecessary anxiety for people studying--and so is bad value. More frequently, people get used to poorly written questions (because these exams are free, they consume a lot of them), and then find the real exam very different--and fail.
So how do you ensure you are taking good, quality sample exams? The simplest and most effective way is to only trust questions from a reputable training company. They have writers that have been trained in how to create questions that meet ISO/IEC 17204 requirements. This is the standard that PMI and many other reputable certification bodies use, such as doctors and teachers.
Ask yourself how much your study time is worth, what are you giving up to get this certification? Given the sacrifices made so far to study, investing in an exam simulation from a reputable source makes good economic sense. However, I understand not everyone can afford or justify paid content, so let’s at least understand how to assess questions to make a judgment call on if the exam simulation is useful or a dud.
Multiple Choice Questions: A Primer
First, a primer on exam question design. This is useful information for everyone taking a test. Understanding how questions are designed helps you answer them more successfully. We will also uncover why you might be good at acing free online tests, but then trip up on the real deal. It all comes down to your online question writers often not knowing this theory.
Multiple choice questions (MCQ) are deceptively simple, so people underestimate them. It seems pretty easy--there is one right answer and three wrong answers. As a test taker, you just pick the right one; as a test creator, you just write the questions and think up a few wrong answers to catch out the guessers.
Let’s start by examining the anatomy of a question and learn the lingo. First of all, questions--along with their correct answer and incorrect options--are called “items”:
Items are comprised of a “stem,” which is the question; and “options” that should contain a single best answer and some wrong answers. These components are divided further with the stem containing a vignette (a scenario) and a lead-in to the options. The options will contain a “key” (the right answer) and “distractors,” which should sound plausible but be incorrect:
So, proper Items will have a stem comprised of a vignette and lead-in, and options comprised of a key and distractors. Now we have enough information to start spotting some bad questions.
In well-written items (questions), the stem is long and the options are short. The writers should be putting the relevant information in the stem, not in the options. This is called a front-loaded or shapely form. Long options are often a sign the stem is unfocused, or that relevant information is missing from the stem.
You should be able to answer the question just by reading the stem. You should not have to read the options to know what the question is asking. So, if you were to cover the options with your hand, does the stem still make sense and allow you to answer the question? If not, you are using poorly written questions--unlike the ones you will see in the exam.
Likewise, beware if you see practice exams with more than four options. Or options that say “All of the above” or ‘None of the above.” These are simply not used in PMI certifications. You would not practice for a tennis competition by playing squash, so you should not practice for a project management certification by using the wrong type of questions.
Next, let’s dig deeper and learn a little about instructional design so we can identify some further instances of bad questions. “Blooms Taxonomy” is a framework that describes the levels of thinking used when learning new material:
Everybody goes through these stages as part of learning and being able to use new skills. The stages can be divided into two main categories, recall and application:
Good exams have a combination of recall and application questions, with a preference for the application of knowledge. An example of a question that tests recall only would be:
The communications management plan is a document that includes descriptions of:
- Project-level performance reports
- Activity-level status reports
- Stakeholder communication requirements
- Responsibility assignments
Here we just need to recall that a communications management plan contains information about stakeholder communication requirements. These questions are easier to write, so people generating hundreds of questions for online exam simulators create a lot of them. Unfortunately, PMI exams use predominantly application-type questions that first outline a scenario and then present a question. These elevate the question from recall to application. For example:
Your team planned to complete six stories in the current sprint. But as you reach the end of the sprint, only four of them are done. What should you do?
- Return the remaining stories to the backlog for re-planning.
- Ask the product owner to extend the sprint.
- Work on the remaining stories when you can fit them in.
- Schedule the remaining stories at the start of the next sprint.
Here we have to first interpret the scenario analyzing the situation, then evaluate the best response. So another warning sign to look for are too many recall-based questions. The majority should be scenario-based application questions.
As we take more and more sample tests, our ability to answer these types of questions increases, but so too does our ability to spot the right answers amongst poorly written distractors and score higher than we really should do on low-quality, poorly written sample tests. So, unfortunately, your increasing performance may not all be attributed to a sponge-like brain for project management knowledge; some might be due to an ability to quickly spot weak wrong answers. In the exam-writing world, this is called “testwiseness.”
Testwise Signs of Poorly Written Questions
A word or phrase in the question also appears in the correct answer: “A Release Map is commonly used to…”
A) Manage WIP
B) Plan Releases
C) Assess Risk
D) Coordinate Users
Here, the term “Release” appears in the question and the right answer.
2. Grammatical Clueing
In the question, the word “an” signals that the answer begins with a vowel: “A short period to undertake work is called an…”
Here, only C) makes the question grammatically correct. Most likely, the question writer created the question with the right answer and generated some distractors without checking the grammar.
3. Syntax Clueing
Similar to grammatical clueing, the wording of the question indicates that the answer is a noun, and one option is the only noun: “The Taskboard” versus “Prototyping” or “Decision Making.”
Having some distractor answers that are not the next steps in a process: “A retrospective normally happens after the:”
D) Requirements Gathering
Here, “Risks” and “Backlog” are not steps in a process and so can be eliminated.
People know few things are absolute, so it is easy to rule out “Always” and “Never” options: For instance, “All teams need coaching.”
6. Longest Option
If one option contains more detail or information, maybe the question writer was trying to better qualify the answer to make it completely correct. This can be a signal that other, less carefully crafted options are distractors.
When the correct answer has the most elements in common with the other options:
A) People and Tools
B) Processes and Techniques
C) Processes and Tools
D) Principles and Tools
Here, “Processes” occurs twice and “Tools” three times, making option C) a good bet for the right answer, since “Techniques” and “Principles” only occur once and were likely added as distractors.
8. Numeric Convergence
A) 6 points
B) 60 points
C) 30 points
D) 15 points
The “6” shows up twice, likely because the question writer typically creates the right answer first then tries to make plausible distractors by basing them on the right answer.
The answer to this question was in another question! This is what happens when people try to generate too many questions based on the same topic.
These examples above were kept simple and obvious to illustrate the flaws, and written in recall format to keep the example questions short. However, all the same mistakes are can be found in longer, situational questions with varying degrees of clarity.
For an excellent overview of question writing guidance, see this presentation from the University of Michigan Medical School. It is designed for medical doctors and includes an eight-question sample. Non-medical professionals typically score the “monkey click” average of 25% on the first pass and then a scary, passing grade of 80+% once they are aware of the question flaws. The problem is we may not realize how much we are relying on subtle clues to answer questions correctly until we attempt questions without these flaws.
Answering Situational Questions
Now that we are better equipped to identify poorly written sample questions, we can focus instead on answering well-structured questions that likely do not have these testwise flaws and are largely in a situational structure:
“You are working at a medical laboratory that traditionally tested 25 swab samples per day and typically worked on 100 swab samples at any given time. Improvements to the testing process have reduced the cycle time to 3 days and the WIP to 90 samples in progress. What is the percentage of improvement in throughput?”
This is a nasty two-part question, but it is a good example to study. First, the question provides us with a scenario to interpret and then tests our application of the relationship between Throughput, WIP and Cycle Time. It does not ask us to recall the equation but requires us to know that Throughput = WIP / Cycle Time. With our new, improved process…
WIP = 90 and cycle time = 3 days, so throughput = 90/3 = 30 samples per day
Once we’ve done this calculation, we still have to find the percentage of improvement over the old throughput of 25 samples per day. So we subtract the old throughput from the new measurement (30 – 25 = 5), and then divide the difference by the old throughput (5/25 = 0.2, or 20%). Therefore, increasing our throughput from 25 samples to 30 samples per day is a 20 percent improvement.
The process to successfully answer these tricky situational questions is:
- Find the question in the question. Read the question through to understand the scenario, then recheck the lead-in part of the question—that’s the final part before the options are presented that asks the specific question to be answered. In our example, it is: “What is the percentage of improvement in throughput?”
- Apply your knowledge and skills, tools and techniques to answer the question. Now that we know it relates to throughput, WIP and cycle time, we have to apply our knowledge of those topics to answer the question asked. Remember to answer the question that is specifically asked, not some other question that could be asked with this set of inputs.
- Answer the question. Answer the question without looking at the options presented. It is easy to be drawn to true statements that do not answer the question asked and then stop thinking. Instead, create your answer to the question asked.
- Find the option that best matches your answer. Now look at the options and identify the one that is the closest match to your answer. This sounds a long-winded, time-consuming way of answering a question when we know one of the answers is correct, but finding an option that matches your answer is good validation and means you have not been drawn to an option.
To make the best use of you study time and avoid unnecessary stress (either while preparing or in the exam), take care to ensure you are using high-quality simulation questions. Learn how to spot poorly written questions that are not indicative of real exams and use the most realistic questions you can. Get plenty of practice answering situational questions that require you to apply your knowledge to project scenarios.
Then, assuming you have the appropriate experience and knowledge, you should be well prepared to visit the test center with confidence to pass the exam and gain your next credential.
(Note: I first published this article on ProjectManagement.com Here)