The Big Picture of Skoll - Tue, 12:11 AM Jul 29 2008

In the Skoll project, we are trying to build a community-based distributed process to test MySQL. From the users perspective, the Skoll clients connect to the Skoll server and receive instructions to build MySQL in a specific configuration. The Skoll client then compiles MySQL and runs a set of about 750 standard MySQL installation tests. Finally, the client sends a summary of the test results back to the Skoll server. What the users do not see is the big picture. How does the Skoll server model the MySQL configuration space? How does the Skoll server select specific configurations from this space to be tested.

The MySQL's configuration space is large, about 48 million of them. It is impossible and impracticable to test every single configuration for a software system that is constantly changing. Therefore, the Skoll project developed techniques to explore and search the configuration space using historical test results; we call these techniques adaptation strategies. One such strategy called nearest neighbor focuses search around a failing configuration to help find additional failing configurations quickly and delineates the boundaries between failing and passing configuration subspaces. Another sampling strategy derived from computing mathematical objects called covering arrays generates a test schedule that satisfies specific coverage metrics that tests all t-way combinations of the configuration options. Strategies like these produced better classification models than equivalently-sized random samples and they scaled well to large configuration spaces.

The goal of the Skoll project is to develop and evaluate adaptation strategies to reduce the configuration space that must be tested in order to discover failing configurations. And quickly share the results with community.