Innovation: Running Experiments and Learning

Experiment DesignIn my last article on Incubating Innovation, we explored the culture and mindset of accountable experimentation. This article focuses on actionable tools and approaches.

Within agile frameworks, the team retrospective is the primary workshop for planning and evaluating experiments. Yet most team retrospectives I see are broken.

Teams spend too much time recording viewpoints and information—but not enough time reviewing or planning experiments. It is common to see the majority of the time spent gathering what went well, what did not go well, and appreciations. Yet where’s the focus on experiments, the learning process and trials for the next iteration?

To make things worse, some teams do not take the retrospective seriously. Maybe after the potential stress of the sprint review, the largely internal retrospective is a relief. A chance to chill out, maybe share some food, and pat each other on the back. However, innovation and learning take conscious effort, forward planning and accountability.

As I work with organizations, I often sit in on retrospectives. Of all the regular workshops/ceremonies, these sessions are typically the least prepared for and worst executed. I often see lazy retrospectives where a basic lessons-learned format is used, but timings are not managed and the recommendations for the next sprint get skimped as they run out of time.

The pie chart below shows a typical planned allocation of time—and the reality of how time is actually spent:

R1

In these lazy retrospectives, people are slow to start, spend longer on recording what went well than what could be improved, and then try to cram the recommendations for experiments (the most important part) into the last few minutes. As a result, experimentation suffers. Few experiments are scheduled for the next sprint, and those that are run are not evaluated properly.

This is not how agile retrospectives are supposed to operate. An excellent guide to running effective retrospectives is Agile Retrospectives: Making Good Teams Great by Esther Derby and Diana Larsen. In it they describe a five-step process:

  1. Set the stage – Help people focus on the task at hand; check that people are ready and willing to contribute. Outline and gain consensus on the process we will use. Techniques we can use include: check-in, working agreements, focus-on/focus-off (see the book for full descriptions of these techniques).
  2. Gather data – Create a shared view of what happened during the sprint/iteration. When completed, we should have a common understanding of the observations and facts. Team activities we can use include: timeline, mad-sad-glad, team radar.
  3. Generate insights – This focuses on understanding the implications of our findings and discussions. We need to see the impacts of the problems we are faced with before trying to solve them. Techniques we can use include: five whys, fishbone analysis, dot voting.
  4. Decide what to do – Now we move from thinking about the iteration that just ended to what we should try next to improve things. We identify the highest-priority action items, create plans for experiments and set measurable goals to achieve the desired results. Techniques we can use include: SMART goals, circle of questions, short subjects.
  5. Close the retrospective – Here we reflect on the retrospective process and express our appreciations. We may summarize what we have decided to keep or change and what we are thankful for. Team-based activities we can use include: plus/delta, return of time invested (ROTI), appreciations.

This is a more useful format. However, despite people having access to good retrospective advice, poor implementations are still common. Teams continue to attend late, start slowly and run out of time or rush the agreement on what experiments to run.

R2

The recurring theme is poor experimentation design and restricted learning. Gunther Verheyen summarized the problem nicely in his recent post entitled “Inspection Without Adaptation Is Pointless.” Gathering data and deciding what to do is pointless if it is not acted upon. We are doing most of the preparation work but not getting any of the benefits.

Experiment Design to the Rescue
Fortunately, there are some good models we can use. We need to manage time and effort more effectively and use retrospectives to plan and evaluate more experiments. We should spend only 50% of the available time on gathering information and the remainder reviewing the results of past experiments, making wins part of our process, designing new experiments and learning from inevitable failures.

We can help the time management problem by assigning work to be done in advance. People should be thinking about issues and potential solutions independently. There are benefits of group discussion and consensus gathering on agreed experiment design, but observation and idea generation is best done individually.

The New Yorker magazine [3] describes numerous studies that show how brainstorming groups think of fewer, lower-quality ideas than the same number of people who work alone and later pool their ideas. There have been numerous reports on the downsides of brainstorming ideas as a group. Groupthink and the halo effect stifle idea generation. So, ask for people to come with ideas, then use the group setting to vet and vote for them.

Visualizing the ideas and experiments is an effective way to bring collective attention to them. Trent Hone and Andrew Jarding developed the “Ideas and Experiments board” pictured below. It shows the progression of ideas through experiments and their success or abandonment:

Experiment Board

Ideas and Experiments Board (Image Credit: Trent Hone and Andrew Jarding, MindSettlers)

As discussed in the last article, by design, 50% of our experiments should fail since we are trying to maximize our learning, not validate things we already know. So I would expect to see an equal number of abandoned experiments as successful ones.

However, this format (or a slightly modified version that represents an experiments Kanban board) is a useful tool to bring the focus for retrospectives to the experiments being run and considered. With some pre-work on idea generation and an increased focus on experiments, we can structure more effective retrospectives.

R3

This retrospective format saves some time by assigning idea generation as pre-work; this also helps avoid the groupthink pitfalls. It furthermore places emphasis on the experiments—the inputs for learning and innovation.

I have experienced pushback from teams about the goal of 50% experiment failure. People understand it optimizes learning—but say it sets people up for too much failure. I understand the sentiment but counter with two perspectives.

First, these are experiments; they should be dispassionate explorations, not evaluations of the people undertaking the work. We need to be professional and try to overcome habits of internalizing results. I know this is easier said than done, so also offer a second reason: We need to kill bad ideas early to save time and money for better ones.

In the article “The Hard Truth About Innovative Cultures,” Gary Pisano describes how killing bad ideas is critical. He profiles Flagship Pioneering, a Massachusetts-based R&D company. It uses a disciplined exploration approach to run small experiments minimizing expenditure. Instead of running experiments to validate ideas, it designs “killer experiments” to maximize the probability of exposing an idea’s flaws. The goal is to learn what went wrong early and move in a more promising direction.

Other useful ideas from the paper include:

  • Tolerance for failure, but no tolerance for incompetence – Hire the best people you can. Explain the goals clearly and let go of those that do not perform.
  • Psychologically safe, but brutally candid – Encourage frank but respectful two-way dialog. It may feel uncomfortable, but it can prevent issues or concerns from going unreported.
  • Collaboration—but with individual accountability – Encourage discussions, but avoid groupthink and hold people accountable for decisions and outcomes.

These are all great concepts and align with the frustrations I experience when I see teams not taking retrospectives seriously—or following through on conducting experiments. I realized I needed a better model for discussing the problem. Fortunately, I found the field of collaborative problem solving (CPS).

CPS is the study of how we work together in groups to solve new problems, innovate and build products. The innovation process and retrospective workshop fall squarely within its scope. CPS skills are quite separate from individual task-focused skills, meaning people can be great at working individually but poor at working together.

A good introduction to CPS frameworks can be found in the article “Advancing the Science of Collaborative Problem Solving.” One model they feature is the “PISA 2015 Collaborative Problem-Solving Assessment.” Unfortunately, like many academic models, the degree of difficulty goes downward, which may make sense as you read down through more advanced stages. However, I think graphically, so I have redrawn the model to show degrees of completeness and difficulty radiating up and out from a 0,0 origin, as shown below:

PISA 1

Along the X-axis, we see three categories of collaborative problem-solving competencies. These are:

  1. Establishing a shared understanding
  2. Taking action to solve the problem
  3. Establishing and maintaining team organization

Up the Y-axis, we have four categories of problem-solving:

  1. Understanding the problem
  2. Representing the problem
  3. Planning and executing
  4. Monitoring and reflecting

Within the body of the model, each square is labeled with the column number and row letter, and describes the tasks that occur in that space.

The model provides a diagnostic tool for identifying broken and lazy retrospectives. The poor engagement and weak follow-through I see in many Scrum teams is characterized by an incomplete execution of column 1 and only half-completion of columns 2 and 3 (as shown by the red outline below):

PISA 2

Teams are not spending time in “(D1) – Monitor and repairing the shared understanding,” nor are they getting to the “(C2) Enacting plans,” (D2), (D3) and (C3) areas to follow through on plans and hold each other accountable for actions and results.

What we want is a complete execution of all the collaborative problem-solving competencies; only then is the framework complete (along with the feedback mechanisms to keep things in check and moving in the right direction):

PISA 3

Summary
Innovation involves combining the right mindset and philosophy with tools and practical steps to ensure its execution. Motivation and attitude are paramount; people have got to want to do this work, enjoy it and create a pull demand for the tools and process that enable it. Trying to foster innovation with demotivated teams is like trying to push a rope.

When motivated and happy people create a strong pull demand for innovation, we need to be ready with the right tools to support the process and keep the momentum going. This includes designing experiments to maximize learning and killing bad ideas quickly—all while demanding competence, accountability and candor.

It is not easy to master the combination of soft skills and techniques required for successful improvement and innovation. However, organizations that succeed can respond to market changes faster and are poised to exploit new technologies and opportunities. Ideas and inventions are spreading quicker than ever. Learning how to build collaborative, innovative teams has become a critical skill.

References

  1. Book: Agile Retrospectives: Making Good Teams Great by Esther Derby and Diana Larsen
  2. Article and video: “Inspection Without Adaptation Is Pointless” by Gunther Verheyen
  3. Article: “Groupthink: the Brainstorming Myth” by Jonah Lehrer
  4. Article: “The Hard Truth About Innovative Cultures” by Gary Pisano
  5. Article: “Advancing the Science of Collaborative Problem Solving” by Arthur Graesser, et al.

[Note: I first wrote this article for projectmanagement.com here]


Incubating Innovation

InnovationIf success goes to those who can innovate the fastest, how do we nurture innovation? The basics are simple to understand—but difficult to implement and stick with in the face of adversity. We need to create an environment that encourages experimentation while also tolerating, investigating and learning from the inevitable failures.

It may sound easy, but executives and shareholders demand results, not “learning opportunities.” We need an approach that fosters experimentation and learning in a defendable way, with a bias for results. To innovate faster than our competitors, we need to maximize our learning potential. This means that by design, 50% of our experiments should fail since we are seeking knowledge expansion, not validation of things we already know. The trick is keeping people engaged and motivated when half of their experiment time is spent failing.

It starts with leadership and cascades down to a shared mindset of team members. Leaders need to be transparent about their own mistakes and learning moments. They need to model the desired behavior, share what they have learned and their new plan of action.

These can be strategic learnings (“Our European market testing has been poor, we are reworking the price options”) or personal (“Feedback on my presentation to investors indicated I was too technical; I need to find simpler ways to describe our technology”). Until team members see transparency in common use, they will be reluctant to practice it themselves for fear of reprisal or criticism.

We Are in the Maze-Solving Business

Maze 1
Developing new products or services is a maze-solving exercise. Nowadays, it is also a race. We need to find a workable solution faster than our competitors. There will be obstacles and dead-ends along the way, and that is okay. We must not let them demoralize us; we just need to learn from them, not repeat them, and keep going.

The process of learning starts with understanding the knowable and then adding to this through experiments and new learnings. So, we start with smart people who understand their industry domain, technology and the business goals. We then need to create an environment with a dual track of product development and experimental learning.

Organizations that run more experiments and iterate faster also learn faster. Scientists studying inheritance use mayflies because they reproduce and provide experimentation results so quickly. The more experiments you can run in a year, the more you can learn.

Our Mazes are Really Big

Maze 2

Developing a new product does not happen overnight. Outside of movies, rarely do lone geniuses develop a marketable product themselves. Instead, it takes teams of subject matter experts months to create proof-of-concepts and multiple iterations of tweaks to complete a viable product. These teams need support and coordination services throughout the process.

Sponsors, executives and shareholders need plans, projections and updates. Product managers, project managers and team leaders all play an important role in keeping the maze-solving teams motivated and moving in the right direction. They also need to keep funding and support going while providing inputs about changing market demands and conditions.

It sounds a difficult balancing act, but approaches such as design thinking, lean startup and agile provide stewardship models for development with inbuilt experimentation, observation and learning. What gets less attention is motivating teams to persevere despite the many failures encountered when experimenting for learning, not just validation.

The “Success Leads to Happiness” Fallacy
Most people start their lives with the mistaken view that success leads to happiness. Our internal dialog creates a series of “if/then” scenarios:

  • IF I pass this exam, THEN I will be happy.”
  • “IF I get this job, THEN I will be happy.”
  • “IF we finish this project on schedule, THEN I will be happy.”

However, the brain has a knack of moving the goalposts. We might be happy briefly, but then we quickly focus on the next exam, an even better job or a more ambitious project.

While it is good to progress in life, we should not connect achieving a goal with achieving happiness. Instead, we need to understand that happiness is only 10% extrinsic (external things that happen to us, like success) and 90% intrinsic (how we think and feel about things).

A Happy Brain is a Productive Brain

Dopamine

Happy workers are more productive and creative than stressed or unhappy workers. In our brains, the chemical dopamine is the neurotransmitter responsible for sending reward-motivated happiness signals. Put more simply, dopamine is a happiness chemical—it gets released when we are happy. Interestingly, dopamine also switches on more learning circuits in our brains.

Research [1,2,3] shows that happiness improves work performance. Happy people are 31% more productive, happy doctors are 19% more accurate at diagnosing correctly, and happy salespeople are 37% better at sales.

This intuitively makes sense. When we are unhappy or stressed, the brain prioritizes circuits for survival. If you spot a sabre-toothed tiger, it is probably best to focus on escape rather than contemplating the interplay of sunlight and shadows through the leaf canopy. Yet when searching for an innovative solution, we want all these extra brain circuits engaged. This brings us full circle on the “I’ll be happy when I am successful” logic. It turns out, being happy actually activates more of the brain to help us be successful.

Success Fallacy

Nurturing Happy Teams
So, if the smartest workers are happy workers, how do we make them happy? Well, we don’t “make them happy” at all—that would be trying to force it in externally, using weak extrinsic motivators. Instead, we equip them with the tools to help build intrinsic happiness themselves.

This might be sounding more touchy-feely than you are comfortable with. However, hard economics show that happy workers also persevere with problems longer, take less sick days, quit less and sue for wrongful dismissal much less, too…so let’s suspend the cynicism for a moment.

The good news is that with as little as 30 minutes a day, measurable increases in dopamine levels are achieved in three weeks. So, if we can encourage these behaviors and turn them into habits, we get happier, healthier, smarter and more productive workers.

These 30-minute exercises don’t even require expensive equipment or management consultants. They are simple, backed by research and include:

  1. 3 Gratitudes – a daily recording of three new things you are grateful for [4]
  2. Journaling – recording positive experiences from the past 24 hours [5]
  3. Exercise – increases blood flow to the brain and helps eliminate toxins [6]
  4. Meditation – resets multi-tasking fatigue and helps with concentration [7]
  5. Acts of Kindness - helping others and saying “thanks,” which makes us feel better [8]

Organizations spend vast sums of money hiring smart people and providing them with complex tools and training. In comparison, the cost/benefit potential in investing and encouraging team happiness is extremely compelling. As project managers, team leads and executives, we need to be conscious of the behaviors we model, because people are watching us.

If not already doing so, we can use these techniques ourselves to improve our own happiness and productivity—then share and encourage others to make use of them. A great benefit of intrinsic motivators is that they can be applied anywhere. It does not matter if you work in a toxic environment or report to a jerk. There is likely nothing stopping you making sure the first email of the day you send is to thank someone for their help or contribution. Also, no one will know if you make notes about positive experiences.

People with budgets and hiring authority do know that yoga instructors are much cheaper than labor relations consultants and HR lawyers. Longer lunch breaks and fun team activities may require some explanations, but improved problem solving, fewer sick days and better ideas are definitely worth it.

Don’t wait until project completion to celebrate team achievements. That’s too little, too late—and our brains have moved the goalposts to be thinking of the next project. Instead, celebrate the little things that recognize effort, persistence and displaying a good attitude.

Obviously, it is not as simple as “happy people are the perfect innovators.” There are some concerns that people characterized as “happy” might be less likely to spot certain types of risks. Optimism needs to be tempered with realism. However, given the variable success rates of helping people become happier (“leading horses to water,” etc.), there will still be some pessimists; but on the whole, it creates a positive change that is worth the risk.

Summary
Smart companies know the future belongs to the best innovators. Building and maintaining teams of productive innovators requires investment in tools, techniques and people. We need to have the right tools and be using today’s techniques such as the design thinking, lean startup and agile approaches. Then it comes down to our people. An appreciation of what truly makes us happy—and its effects on success—is a great starting point.

Most organizations are not R&D labs, so we need to balance innovation with everyday production and service. The mindset and changes described here may feel uncomfortable (or even unprofessional) at first. However, a quote from Eric Shinseki explains that “If you dislike change, you're going to dislike irrelevance even more.” It reminds us that the business landscape is changing faster than ever—and that we need to change with it to stay valuable.

References

  1. The Happiness Advantage
  2. The Benefits of Frequent Positive Affect: Does Happiness Lead to Success?
  3. Happiness and Productivity
  4. Counting Blessings Versus Burdens
  5. How Do I Love Thee? Let Me Count the Words, The Social Effects of Expressive Writing
  6. Behavior Matters
  7. One Second Ahead
  8. Interventions to Boost Happiness and Buttress Resilience

[Note: I first wrote this article for ProjectManagement.com here]


Government Lessons in People Over Process

CubicleMy first opportunity to create and run a large agile team did not start well. Having had good successes with small to medium sized agile teams I was keen to unleash the benefits on a bigger scale. I was working for IBM at the time and was able to persuade my account manager to pitch the approach on one of our government projects. A clean-sheet development opportunity with a smart team and engaged business group – what could go wrong? As it turns out, plenty due to my ill-advised approach.

It was the early 90’s and we were trialling techniques that would later become the agile approach DSDM (Dynamic Systems Development Method). Taking ideas like James Martin’s RAD (Rapid Application Development) and active user involvement from Enid Mumford’s Participative Design Approach, we had already dramatically reduced development time and improved acceptance rates on several projects. I was convinced collocated teams with short iterations of build/feedback cycles were the future. We were all set for a big client success and who better than the British Government for good publicity! My enthusiasm was about to be tested.

I was given a full rein of the project, or as I would later realize, just enough rope to hang myself with. Having struggled to get dedicated business input on previous projects I commandeered a large boardroom to collocate the development team and business subject matter experts (SMEs). It was awesome, everyone was together in one room and we had direct access to the business representatives for requirements elicitation, clarification, and demo feedback. We were working hard and getting lots of features built but the business representatives hated it.

At first, I thought they hated me. I think that is a common mistake, we internalize changes in behaviour as attacks or criticisms of ourselves. What have I done? What did I say to upset them? - all of them! I recall wanting to write on my internal project status report to the IBM PMO that “the business is revolting”. However, that is what occurred, starting as cordial and helpful, the business SMEs became less helpful, then uncooperative, and finally hostile. I had a revolt on my hands that I did not understand.

This was my first introduction to organizational change. Luckily for me, I had access to many people in IBM smarter and more experienced than I was. I was given a book called “How to Manage Change Effectively: Approaches, Methods, and Case Examples” by Donald Kirkpatrick that changed my career. In it Kirkpatrick outlines circumstances where people will resist change. These include:

  1. When people sense loss in: security, pride and satisfaction, freedom, responsibility, authority, good working conditions, and/or status
  2. It creates more problems than it is worth
  3. Extra efforts are not being rewarded
  4. Lack of respect for those initiating the change
  5. The change initiative and its implications are misunderstood
  6. Belief that the change does not make sense for the organization
  7. Change is misdirected, current state or alternatives are better
  8. A low tolerance for change in our lives
  9. When change violates a principle or commitment that the organization must stand by
  10. Exclusion from the change initiative
  11. Changes viewed as criticism of how things were done in the past
  12. The change effort occurs at a bad time, other issues or problems are also being handled

Something I was not aware of at the time is how the career development process works within the government. The most junior new hires work in open-plan cubical offices. Then as you get a promotion you get moved to bigger cubicles with higher walls that are more like mini-offices. Next, you get promoted to a real office, then an office with a window, and eventually a corner office. In short, your workspace defines your status, responsibility and authority.

By bringing these business representatives into a shared boardroom to work on the project I had unwittingly generated change resistance scenarios 1-3 and probably triggered many others also. Making them sit and work together like the most junior recruits had caused a loss of good working conditions, status, freedom, pride, satisfaction, and perceived authority. A bad idea when hoping to develop a productive working relationship with someone.

Luckily for me the Kirkpatrick book also lists circumstances when people do accept change, which unsurprisingly are the opposite conditions and include:

  1. When change is seen as a personal gain: in security, money, authority, status or prestige, responsibility, working conditions, or achievement
  2. Provides a new challenge and reduces boredom
  3. Opportunities to influence the change initiative
  4. Timing: the time is right for organizational change
  5. Source of the change initiative is liked and respected
  6. The approach of the change and how it is implemented appeals to us

So, equipped with these ideas we changed our approach. Instead of the business SMEs being collocated with us they returned to their fancy corner offices, long lunch breaks, and afternoons spent reading the newspaper - none of which they could do when they all sat together. Instead, we reserved their mornings for questions, review sessions, and demonstrations. This was better received because their morning calendars were blocked with important project meetings, but we rarely called on all of them at once unless it was for a business demo.

Now they had their offices back, a little more free time, and were engaged in a more respectful way. The team were sceptical at first. However, it really is much better to have one hour of someone who is cheerful, engaged, and helpful than eight hours of someone who is bitter, obstinate and causing issues. The project went much smoother after these changes and it taught me an important lesson in never trying to introduce a process or practice without considering the people elements first.

We completed the project early, largely due to the input and hard work during acceptance testing of the business SMEs, and IBM got their successful case study. I learned to temper my enthusiasm and consider other stakeholders who will undoubtedly have a different view of the project than myself. Individuals and interaction are indeed more important than processes and tools, even if they are your own pet agile processes and tools.

[I first wrote this article for the Government themed November issue of ProjectManagement.com, available to subscribers Here]


They are “Lessons to be Learned”, not “Lessons Learned”

The suggestions, observations and ideas we capture at retrospectives are not Lessons Learned. That would imply we have already learned from them and will not make that mistake again. Instead, they are Lessons-to-be-Learned which is subtly different but stresses the most important part, which is we now need to learn something.

Learning involves several steps. David Kolb, an educational theorist, describes a 4-step learning process:

  1. Concrete Experiences (What we already know)
  2. Observation and Reflection (What our retrospectives help us identify)
  3. Abstract Conceptualization (Thinking about the problems and designing potential solutions)
  4. Active Experimentation (Trying something new)

These stages act as part of an experimental learning cycle. The last step, Active Experimentation, creates new concrete experiences and builds on what we already know. Experimental Learning Cycle

It is easy to confuse the retrospective actions of Observation and Reflection (Stage 2) as gathering lessons learned. However, this is not the case, instead it is just one step in the process. We then need to determine a solution (Stage 3) and run experiments to learn from them (Stage 4). Only then might we actually learn something.

To remind us that simply gathering ideas and suggestions for improvements is not the same as learning, I suggest we stop using the term “Lessons Learned” and instead useLessons to be learned”.


New PMI-ACP Workbook

PMI-ACP WorkbookI am pleased to announce the availability of my new PMI-ACP Workbook. This new workbook focusses on a smaller subset of 50 key topics.   My original PMI-ACP Exam Prep book distilled all the relevant content from the 11 books on the PMI-ACP recommended reading list in a common voice. The workbook is also different by providing lots of exercises and many situational questions like you will find in the exam.

So, while my PMI-ACP Exam Prep book covers all the background and theory – ideal for a comprehensive coverage of everything in the exam, the new PMI-ACP Workbook is a practical, hands-on study tool that focusses on the core topics needed to pass the exam. If you already have your CSM credential or 3+ years of agile experience you likely know the agile mindset, values and principles material already. However, you may not have the lean, kanban, and team development knowledge needed to pass the PMI-ACP exam so the workbook can fill those gaps.

To help determine which book is best for you I created the following flowchart:

PMI-ACP Workbook Flowchart

Hands-on learners and people who do not want to read all about how the approaches fit together will find the 50 key topics of the new workbook a simpler way to navigate the material. Also, since the content is arranged by topic alphabetically you can easily jump around and create your own study plan based on just the topics you need.

While the workbook coverage of topics is less than the prep-book, the emphasis on exercises and situational questions is much higher and accounts for the slightly higher page count (457 pages). There is white space for writing notes and the whole thing is spiral bound so it lays flat when you are working in it. The content changes are summarized by these rough page count graphs:

PMI-ACP Book Contents

I think it fills an important need. A workbook for hands-on learners looking to build their own study plan and gain access to high-quality situational questions. It also provides access to a free online quiz. Readers can order and get an early-bird discount from RMC here.