This year we allowed community reviewing in a Wisdom of Crowds model. This brought many advantages and some challenges. The advantages include practicing what we preach and getting users involved in prioritizing our conference backlog. Plus it provided a mechanism for early feedback and adaptation. Within hours of being posted most submissions had review feedback and given the increased volume of reviewers more suggestions for improvement were provided than could have been achieved using just a review panel.

I think the net result was a much high volume of submissions and a greater number of incremental improvements to those submissions as people acted on the reviews and ratings. Also, because the submissions were open to all, people could see the level of detail and quality of content of top rated sessions and gravitate towards these higher standards.

An inevitable downside is a small degree of ‘gaming’ the numbers as people submit something, get their friends to vote for it and then down-vote competing sessions. (I know I joked about this in my last post, but we did see it happen) Anyway, by looking at whole community scores and review committee scores independently we can spot the obviously gamed sessions. Also by analyzing unusual voting trends and adding up all their -2, -1, 0, 1, 2 scores across sessions you can find the ‘down-vote trolls’ and strategic voters.

I was talking about the submission system with Michael Sikorsky from Cambrian House who has built a business around Crowdsourcing. Cambrian House has an entire chapter dedicated to them in the upcoming book Jeff Howe book Crowdsourcing: Why the Power of the Crowd is Driving the Future of Business and partnered with Don Tapscott author of Wikinomics to help spread the word about the Crwodsourcing phenomenon. He explained that most participation of crowds systems no longer allow down-voting, instead they just count distinct endorsements (up-votes). This helps avoid the down-vote troll issue and focuses on the total number of unique up-votes.

Anyway, now the submission system is closed we have the task of short listing the proposals and juggling the rooms to best meet the demand for the various stages. So there is still lots to do, but thankfully my review marathon is over. One title that made me smile and kept me going was: “Programmers may be from Mars, Customers may be from Venus, but why does everyone think that Project Managers are from Uranus?” Funny.

Incidentally, Don Tapscott author of Wikinomics: How Mass Collaboration Changes Everything will be a keynote at the Agile 2008 conference. A great choice of speaker for a conference shaped so significantly this year by the mass collaboration of its participants.