Most people taking a project management certification exam use sample tests. Whether it is a PMP exam, ScrumMaster, CAPM, PMI-ACP, PgMP or many others, there are plenty of online options for getting familiar with the format and determining if you are ready to sit the exam proper.

Unfortunately, like all things found online, the quality and relevance varies considerably. If we are just looking for funny cat videos, the occasional shaky video filmed in portrait mode is annoying--but easily skipped and not the end of the world. However, bad exam simulators can give a false sense of security--or a false sense of insecurity--and generally do not prepare you at all for what the actual exam will really be like.

Before getting trained and involved in question writing for PMI and professional training companies, I had no idea about the science behind good multiple choice questions. Now, I cannot help but notice poorly written questions. Even if the test is free, if it tests material not in the exam, it can generate unnecessary anxiety for people studying--and so is bad value. More frequently, people get used to poorly written questions (because these exams are free, they consume a lot of them), and then find the real exam very different--and fail.

So how do you ensure you are taking good, quality sample exams? The simplest and most effective way is to only trust questions from a reputable training company. They have writers that have been trained in how to create questions that meet ISO/IEC 17204 requirements. This is the standard that PMI and many other reputable certification bodies use, such as doctors and teachers.

Ask yourself how much your study time is worth, what are you giving up to get this certification? Given the sacrifices made so far to study, investing in an exam simulation from a reputable source makes good economic sense. However, I understand not everyone can afford or justify paid content, so let’s at least understand how to assess questions to make a judgment call on if the exam simulation is useful or a dud.

Multiple Choice Questions: A Primer
First, a primer on exam question design. This is useful information for everyone taking a test. Understanding how questions are designed helps you answer them more successfully. We will also uncover why you might be good at acing free online tests, but then trip up on the real deal. It all comes down to your online question writers often not knowing this theory.

Multiple choice questions (MCQ) are deceptively simple, so people underestimate them. It seems pretty easy--there is one right answer and three wrong answers. As a test taker, you just pick the right one; as a test creator, you just write the questions and think up a few wrong answers to catch out the guessers.

Let’s start by examining the anatomy of a question and learn the lingo. First of all, questions--along with their correct answer and incorrect options--are called “items”:

This article aims to dispel the myth that agile projects somehow magical manage risks for us, and outlines a couple of practical tools that can be used to start improving risk management approaches. 

Agile is Not a Risk Management Approach

Some people believe agile approaches with their short cycles and regular feedback have a risk management approach naturally built into the process. It is easy to see why, the building blocks and attachment points for plugging in an effective risk management process are certainly present, but unfortunately just building something iteratively or incrementally does not ensure risks are managed. 

It is all too easy to develop iteratively missing opportunities to actively address threats or exploit opportunities. Many agile teams also fail to actively look for risks, discuss and decide on appropriate actions, undertake those actions and reassess the risks and evaluate if the risk management process is even working. 

It is a shame because in many ways agile methods provide an ideal framework for introducing effective risk management practices. They have short timeframes, active reprioritization of work, frequent review points, high team member and business engagement in planning, etc. However, similar to having a group of people to help you find something, “a beach-party is not the same as a search-party”. We need a conscious effort, coordination and cooperation to make it effective.

Consciously Adding Risk Management to Agile Approaches

The good news is, that when organizations and their participating teams decide to layer risk management onto agile approaches there are many self-reinforcing cycles and mechanisms to make use of. For instance, the frequent consideration of change requests and reprioritization of work in the backlog makes the insertion risk avoidance or risk mitigation tasks an easier process to handle. 

Likewise, the regular retrospectives that review progress and process are great points to examine the effectiveness of risk management strategies and take corrective actions. Daily standup meetings that surface issues and blockers can also act as early warnings for potential new risks, etc. 

For anyone interested in linking agile approaches to risk management steps, here’s a White Paper on Collaborative Games for Risk Management that was presented at the 2012 Agile conference and PMI Global Congress. These ideas and their development more into Opportunity Management were explored at this 2015 Agile Conference Session. However, the mechanics of doing the work and linking it into an agile lifecycle are the easy parts, getting people to take a risk-based view to project work is where the real work is needed.

Thinking about Risk Management

Education and acceptance are the keys to successfully adding risk management to agile practices. We need to get people engaged in the process and instill a common understanding of threats as the possibility of negative value. Once people understand this they can answer the question “Where is the next best dollar spent?” more effectively. It might not be on building the next feature from the backlog, but instead avoiding a risk or exploiting an opportunity. 

Disclaimer: This article is based on my personal experience of software project development work over a 25 year period running a mixture of local projects, outsourced projects and hybrid models. The data is my own and subjective, but supported by 1,000’s of industry peers I question while delivering training courses for the PMI. I do not work for a local based or outsourcing based company, I have nothing to gain from favoring either approach, but I hope these thoughts are useful for determining some of the pro’s, con’s, true costs and circumstances when outsourcing is better or worse than local development.

To the uninitiated, outsourcing seems like a great idea. Software engineers are expensive in many countries but much cheaper in other parts of the world. So, since software requirements and completed software can be shipped free of charge via email and web sites, why not get it developed where labor rates are much lower?

Coding vs Collaboration Costs

The flaw in this plan comes in the execution of it when it becomes apparent that software development projects typically entail more than just the development of software. Writing code is certainly part of it, but understanding the problem, agreeing on a design, discovering and solving unforeseen issues, making smart decisions and compromises to optimize value and schedule are big parts of it too. This is the collaboration effort part of a project. Also, while the coding part might represent 30-50% of the overall project costs, these shrink to 20-30% when a 3-year ownership cost view is considered that includes support, maintenance and enhancements.

Sticking with just development costs for now, let’s examine a scenario. The business case pitched to executives by outsourcing companies initially seems very compelling: Project Alpha needs 9 months of software development by a team of 5 people. If you work in an expensive labor market, like North America, we can assume fully-loaded hourly rates of $100 per hour, yet highly qualified consultants from our fictional outsourcing country of TechLand cost only $25 per hour. So, the project for 9 months x 160 hours per month x 5 people at $100 per hour in an expensive market costs $720,000. For a TechLand team this would cost 9mths x 160hrs x 5pl x $25hr = $180,000, that’s a cool $540,00 saving, right?

Let’s revisit this scenario based on the acknowledgment that the actual software writing part of a project is closer to a 30-50% of the total effort. This leaves the remaining 50-70% of the work as the communications heavy collaboration part. It should come as no surprise that separating people via distance, time zones, and potentially language and cultural barriers increase communications effort and propagates issues up the cost-of-change-curve. 

So, when 50-70% of the communication-heavy collaboration work takes longer, how do we quantify that? Agile methods recommend Face-to-Face communications because it is the quickest, conveys body-language and provides an opportunity for immediate Q&A only for the issues that need it. Switching from Face-to-Face to video, conference call, email or paper create barriers and adds significant time and opportunity for confusion. A 2-3 X increase in effort likely downplays the true impact when considering the costs of fixing things that go awry because of inevitable misunderstandings, but let’s use that number.

Redoing our project Alpha costs with, say, 40% as the actual coding effort and 60% effort communications heavy collaboration work that takes 2.5 X as much effort we get: 9mths x 160hrs x 5pl x $25 hr x 40% = $72K Coding + 9mths x 160hrs x 5pl x $25 hr x 60% x 2.5 = $270K Collaboration giving $342,000 in total. However, this is less than half the costs of the $720,000 locally developed project so we are still good, right?

The Compounding Costs of Delay

An error in the logic applied so far is that this 2.5 X communication and collaboration penalty on 60% of the work can somehow be magically absorbed into the same 9 month timeline. In reality these outsourced projects take longer because of the increased communication and collaboration effort and 2.5 x 60% = 1.5 X as long is consistent with my experience from 25 years of mixed local and outsourced projects.