Voluntary but coordinated - release test involving the user community

The Target Management Project started its 2nd round of coordinated testing these days on its first release candidate, RSE 1.0RC1.

The interesting part of this test effort is, that it does not only involve committers but also many users who volunteered to join the testing. It looks like the users understand, that joining a coordinated test early helps each of them by achieving better quality - at little cost if lots of people join. The original test invitation E-Mail had been sent directly to all users we've had contact with, plus the tm-dev mailing list.

The test is mostly facilitated through the Eclipse Wiki as a platform for collaboration, plus daily E-Mails to all signed up testers and the tm-dev mailing list during the test period, mentioning important news.

Some things we have learned from this effort:

• Signing up each user with a 2-hour effort in advance, and asking them to change the signup on the Wiki themselves proved very effective. That way, we also signed up each user with some randomly chosen feature.
• It took some time until users learned to use the Wiki effectively. It looks like there was some unnecessary shyness of getting involved and modifying content themselves. The 2nd round of testing is going a lot better than the first one already.
• Bugzilla was used with a specialized bug entry form, remembered as bookmarkable entry and used by the testers. Having the detailed configuration info at hand with each bug report proved very helpful.
  
• We found a lot of new bugs - some of them critical - and surprisingly few duplicates, although we had asked testers to not waste time searching for duplicates before filing a bug. This seems to show how valuable it is to have a large group of testers with slightly different usage schemes, even if each of them invests only a small amount of time (we mostly asked for 2-4 hours each).

Comments from our testers showed that everybody liked the format of this test. In fact, everybody increased time commitment for the 2nd round of testing, and nobody left. At the same time, the users learned about new features and the project found new bugs.

This test is a great example showing that the Eclipse model of Open Source works really well: communities of users, contributors and committers all working together on a common goal, coordinated but voluntary. It shows that Open Source does not mean it is uncoordinated; in fact, by being Open and Transparent the Eclipse model is probably more coordinated and predictable than many proprietary projects.