SoC 2008 Beyond the End - Mon, 04:33 PM Sep 8 2008

Google Summer of Code 2008 officially ended on August 18th. Since that deadline, I have been working to solve a few non-functional problems I discovered during testing stage.

• By week 11, Skoll Client was collecting runtime information for each of the MySQL tests. During the development stage, I only tested this feature with a few MySQL tests. However, during testing stage I discovered this feature would take way too much time and resources for majority of the users. To completely test one configuration of MySQL, it took Skoll Client over 10 hours because MySQL server must be restarted after each test. Also, the runtime information collected was over 3 GB (zipped!). If the user is only interested in the coverage of the MySQL test suite, the per-test runtime information is a major overkill. The solution was to have two runtime information collection flags in the Skoll Client, --gcov and --detailed_gcov, to collect runtime information for all tests and runtime information per test. I doubt any user besides me would want to use the --detailed_gcov option.
• Generating runtime information reports from the database has always been slow due to the large amount of data collected for MySQL. Now that the database structure and the SQL queries is more concrete, I can finally spend some time optimize for performance. With a few tweaks to the SQL and a few added indices, I was able to decrease the runtime of some queries from over 2 minutes down to 0.04 seconds. I am pretty new to optimizing database performance, learning about the ?explain? command really helped me understand what was going on with a query. However, even with this huge performance improvement, certain types of reports such as "per source line" is still too slow for data collected for multiple configurations. The visualization techniques I am using is not mature enough to handle large software system like MySQL just yet.

My plans for this project after Summer of Code is as follows:

• Skoll Client is feature complete, I have no plans to add new features in the near future. Of course, bug fixes and patches will be added.
• The visualization techniques I worked on look promising, but using them on MySQL right now is costly in both time and computing resources. I will continue to research ways to visualize multi-configuration runtime data, but on smaller software systems.
• For future updates on the Skoll Project, visit http://www.cs.umd.edu/projects/skoll/contribute/mysql.html.

Week 12 - A Test Scheduler for the MySQL Build Farm Initiative - Mon, 11:04 PM Aug 18 2008

KEY ACCOMPLISHMENTS LAST WEEK

• Refactored and cleaned up source code.
• Updated JavaDoc.
• Uploaded source code to code.google.com.

KEY TASKS THAT STALLED LAST WEEK

• Tried to upload source code to launchpad, but code.google.com was easier to use.

KEY CONCERNS

• None

TASKS IN THE UPCOMING WEEK

• Keep working on the project until the semester starts (Sept 2nd), at least.
• Integrate all server side processing scripts and tools (Java, Perl, R, HTML).
• Collect and analyze MySQL runtime information.

Running Shell Commands in Java - Tue, 12:54 AM Aug 12 2008

The Skoll Client retrieves a set of commands from the Skoll Server to compile and test MySQL; these commands are UNIX shell commands. My Google Summer of Code project is to work on the Java Skoll Client, I spent a great deal of time and effort getting these shell commands to run well in Java. Running shell commands in Java is not always straightforward, here are some techniques I learned to get the job done.

Use Java's ProcessBuilder class instead of Runtime class for Java 5.0+ projects. There are two reasons. First, ProcessBuilder has redirectErrorStream() method that automatically merges the standard output and standard error together so the output of a command would make more sense and more importantly the process won't freeze if the developer doesn't clear the error output stream. Second, the environment() method returns a nice Map the developer can use to manipulate the environment in which the command should run. Unfortunately, I must support Java 1.4 and 5.0 so I had to do everything the hard way.

Modifying the PATH environment variable does not add a location for Java to locate executables, at least not the way I expected it to. I was trying to run the "bzr" command which is not in my default path, I thought setting the PATH environment variable to include the non-default path would allow Java to find "bzr". That did not work. The environment settings do not take affect until the process has started, but Java cannot start the process because it cannot find the "bzr" command. For now, I set the PATH in the shell script that starts the Skoll Client. But one possible solution is to set the environment variables, then run the command "sh -c 'bzr'" (not yet tested).

Passing a shell command to Java does not always work. The way Java parses the commands does unexpected things, at least to me at first. I found that parsing the commands myself then passing them to Java guarantees correct execution. This command parsing algorithm divides one command string into an array of command parts; the command parts are divided by whitespace, unless the whitespace appears in quotation. Here is a simple method that prepares the command for Java.

private String[] prepareCommand(String command)
{
   ArrayList commandParts = new ArrayList();
   StringBuffer currPart = new StringBuffer();
   boolean inQuote = false;

   for (int x = 0; x < command.length(); x++)
   {
      char ch = command.charAt(x);

      if (ch == ''' || ch == '"')
      {
         inQuote = !inQuote;
      }
      else if (Character.isWhitespace(ch) && !inQuote)
      {
         commandParts.add(currPart.toString());
         currPart = new StringBuffer();
      }
      else
      {
         currPart.append(ch);
      }

      // get the last part of the command
      if (x == command.length() - 1)
      {
         commandParts.add(currPart.toString());
      }
   }

   String[] coms = new String[commandParts.size()];
   commandParts.toArray(coms);

   return coms;
}

For more information and source code related to running shell commands in Java, see ShellCommandsWithJava.

Week 11 - A Test Scheduler for the MySQL Build Farm Initiative - Tue, 12:47 AM Aug 12 2008

KEY ACCOMPLISHMENTS LAST WEEK

• Finished per test runtime information collection modifications.
• Collected and analyzed per test runtime information for a few suspicious configurations and tests.
• Refactored and cleaned up source code.

KEY TASKS THAT STALLED LAST WEEK

• None

KEY CONCERNS

• None

TASKS IN THE UPCOMING WEEK

• Refactor and clean up source code.
• Update JavaDoc.

Collect Runtime Information for Each MySQL Test - Tue, 12:34 AM Aug 5 2008

In a previous post, I talked about the method Skoll used to collect MySQL runtime information for non-default configurations. At the time, the runtime information was collected after all of the MySQL tests were executed, which means the runtime information was accumulated from all tests run. There was no way to decipher how each test contributed to this accumulated runtime information. A greater degree of granularity can provide better understanding of the MySQL runtime behavior.

To collect runtime information for each MySQL test, the Skoll client still compiles MySQL as described in the previous post. Instead of running the "mysql-test-run" script with no arguments (which runs all available tests in the source tree), Skoll client calls "mysql-test-run [TESTCASE]" to run one specific test. After this test finishes, Skoll client collects the runtime information and then repeats the process for other tests. However, to make this work, two tasks must be handled:

1. Skoll client needs to know which tests are available in the MySQL source tree. The tests are located in the "mysql-test/t" directory. Each test is in a file "[TESTCASE].test", and the filename is the test name (additional test suites are in "mysql-test/suite/[TESTSUITE]/t" directories). It's pretty straight forward for Skoll client to browse through the directory to make a list of tests.
2. Runtime information from the previous test must be cleared before the next test starts, because gcov accumulates runtime information from numerous executions in data files with extension *.gcda. Therefore, after each test Skoll client must recursively go through the source directories and delete those files in order to collect per test runtime information.

Week 10 - A Test Scheduler for the MySQL Build Farm Initiative - Mon, 11:23 PM Aug 4 2008

KEY ACCOMPLISHMENTS LAST WEEK

• Fixed some left over glitches in Skoll Client related to the Bazaar switch, mostly to deal with Java Runtime exec.
• Analyzed MySQL runtime data, but did not find clear differences between different configurations of MySQL. The analysis might need more granular runtime data, see next item.
• Modified Skoll Client to collect runtime information per MySQL test. Right now, Skoll Client runs mysql-test-run script (which executes all tests) then collects gcov runtime information for all tests.

KEY TASKS THAT STALLED LAST WEEK

• None

KEY CONCERNS

• None

TASKS IN THE UPCOMING WEEK

• Finish per test runtime information collection modifications.

The Big Picture of Skoll - Tue, 12:11 AM Jul 29 2008

In the Skoll project, we are trying to build a community-based distributed process to test MySQL. From the users perspective, the Skoll clients connect to the Skoll server and receive instructions to build MySQL in a specific configuration. The Skoll client then compiles MySQL and runs a set of about 750 standard MySQL installation tests. Finally, the client sends a summary of the test results back to the Skoll server. What the users do not see is the big picture. How does the Skoll server model the MySQL configuration space? How does the Skoll server select specific configurations from this space to be tested.

The MySQL's configuration space is large, about 48 million of them. It is impossible and impracticable to test every single configuration for a software system that is constantly changing. Therefore, the Skoll project developed techniques to explore and search the configuration space using historical test results; we call these techniques adaptation strategies. One such strategy called nearest neighbor focuses search around a failing configuration to help find additional failing configurations quickly and delineates the boundaries between failing and passing configuration subspaces. Another sampling strategy derived from computing mathematical objects called covering arrays generates a test schedule that satisfies specific coverage metrics that tests all t-way combinations of the configuration options. Strategies like these produced better classification models than equivalently-sized random samples and they scaled well to large configuration spaces.

The goal of the Skoll project is to develop and evaluate adaptation strategies to reduce the configuration space that must be tested in order to discover failing configurations. And quickly share the results with community.

Week 9 - A Test Scheduler for the MySQL Build Farm Initiative - Mon, 11:11 PM Jul 28 2008

KEY ACCOMPLISHMENTS LAST WEEK

• Fixed Skoll Client to work with the new commands related to Bazaar switch. Running shell commends in Java, esp supporting both Java 1.4 and 5.0, is difficult and brittle.
• Analyzed MySQL runtime data with Weka. Most of the work is in retrieving/preparing the large amount of data efficiently from the database.

KEY TASKS THAT STALLED LAST WEEK

• Still did not collect runtime data with Skoll Client due to Bazaar switch.

KEY CONCERNS

• None

TASKS IN THE UPCOMING WEEK

• Runtime data collection.
• Continue to research tools and methods to analyze runtime information.

Week 8 - A Test Scheduler for the MySQL Build Farm Initiative - Mon, 10:45 PM Jul 21 2008

KEY ACCOMPLISHMENTS LAST WEEK

• Read the book "Data Mining: Practical Machine Learning Tools and Techniques" to learn about features of Weka.
• Played with Weka to analyze and visualize MySQL runtime data.
• As part of the effort to use push-build and Bazaar with Skoll, found a method to set $PATH in Skoll Client. Now, commands and executables in "non-standard" locations can be specified for Skoll Client (in this case bzr).

KEY TASKS THAT STALLED LAST WEEK

• Could not test or collect runtime data with Skoll Client due to Bazaar switch.

KEY CONCERNS

• None

TASKS IN THE UPCOMING WEEK

• Continue to research tools and methods to analyze runtime information.
• Runtime data collection.

Week 7 - A Test Scheduler for the MySQL Build Farm Initiative - Mon, 09:44 PM Jul 14 2008

KEY ACCOMPLISHMENTS LAST WEEK

• Worked on Skoll's data processing scripts and test result generation infrastructure.
• Modified packaging format of runtime data collection, started automated runtime information collection tasks with the new Skoll client.
• Researched tools and methods to analyze runtime information (data mining and Weka).

KEY TASKS THAT STALLED LAST WEEK

• None

KEY CONCERNS

• None

TASKS IN THE UPCOMING WEEK

• Continue to research tools and methods to analyze runtime information.
• Modify Skoll to use push-build tar balls for compilation and testing.

How Skoll Collects MySQL Runtime Information - Mon, 09:37 PM Jul 14 2008

To understand the runtime behavior of MySQL under different configurations, Skoll needs to collect runtime data while testing MySQL builds. To accomplish this, the Skoll client takes advantage of gcov, a test coverage program that's part of the GNU Compiler Collection (GCC). gcov collects runtime information such as how many times a line of code, a method or a file was executed, what was the coverage and much more.

The source tree of MySQL actually provides a few build scripts that enables gcov under the BUILD directory. However, these scripts build MySQL using the default configuration; not exactly what Skoll needs. Skoll client builds and tests MySQL in different configurations by passing compile-time and run-time flags to the configure script before compilation. For example

./configure --enable-thread-safe-client --with-extra-charsets=all --with-innodb --with-ndbcluster

To enable gcov under non-default configurations, Skoll client passes a few more compiler flags to the configure script.

CC=gcc
CFLAGS=-fprofile-arcs -ftest-coverage
LDFLAGS=-fprofile-arcs -ftest-coverage
CXX=gcc
CXXFLAGS=-fprofile-arcs -ftest-coverage

After compilation is complete, most of the source files will have corresponding .gcno files.

When Skoll client has finished running all of the MySQL tests, it will recursively go through the source directories and run

gcov <filename>

on every source file to generate gcov runtime data. These runtime data files are then zipped and sent to the Skoll server for future processing and analysis.

Week 6 - A Test Scheduler for the MySQL Build Farm Initiative - Mon, 11:35 PM Jul 7 2008

KEY ACCOMPLISHMENTS LAST WEEK

• Constructed an experimental runtime data processing program.

KEY TASKS THAT STALLED LAST WEEK

• None

KEY CONCERNS

• None

TASKS IN THE UPCOMING WEEK

• Modify Skoll to use push-build tar balls for compilation and testing.
• Modify Skoll's data processing scripts for more detailed HTML results pages.

Week 5 - A Test Scheduler for the MySQL Build Farm Initiative - Mon, 11:44 PM Jun 30 2008

KEY ACCOMPLISHMENTS LAST WEEK

• Analyzed the collected runtime data from Skoll Client. I am in the process of constructing a program to process/compare the runtime data.
• Improved Skoll Client so the it does NOT have to connect to two different databases in order to collect runtime information. This improvement makes management of MySQL configurations on the server side much easier.

KEY TASKS THAT STALLED LAST WEEK

• None

KEY CONCERNS

• None

TASKS IN THE UPCOMING WEEK

• Continue with runtime data processing, and then automate this processing with scripts on the Skoll server.
• Modify Skoll to use push-build tar balls for compilation and testing.

Week 4 - A Test Scheduler for the MySQL Build Farm Initiative - Mon, 11:18 PM Jun 23 2008

KEY ACCOMPLISHMENTS LAST WEEK

• Set up automated runtime information collection tasks on 6 machines; These machines are compiling and testing MySQL 24/7 and uploading the results to Skoll server for future analysis.
• Modified Skoll's upload manager and data processor to handle the separate log files collected by the new Skoll Client. The separate log files include a Skoll Client log and a set of logs for each command (e.g. configure, compile, test) in a test run. The command logs are zipped together by the client before uploading to the server.

KEY TASKS THAT STALLED LAST WEEK

• None

KEY CONCERNS

• Future progress of the project depends on have push-build tar balls.

TASKS IN THE UPCOMING WEEK

• Add more information from the separate log files to the HTML results pages; this task would require database changes to Skoll server.
• Analyze the collected runtime data from Skoll Client and research ways to process these data.

The "TASKS IN THE UPCOMING WEEK" from previous week do not always match "KEY ACCOMPLISHMENTS LAST WEEK" in the reports. This is mainly due to my meeting with the mentor/advisor on Weds, and my responsibilities and focus might change.

Week 3 - A Test Scheduler for the MySQL Build Farm Initiative - Mon, 11:38 PM Jun 16 2008

KEY ACCOMPLISHMENTS LAST WEEK

• Finished the integration of new data collection features in Skoll. The Skoll Client can now collect runtime information about MySQL while running the MySQL tests.
• Collected runtime information for MySQL compiled with different configuration flags and began analyzing the collected runtime data.

KEY TASKS THAT STALLED LAST WEEK

• Had connection problems with BitKeeper (MySQL's source control) servers while testing the new Skoll Client.
• Skoll currently gets MySQL source code from BitKeeper, however, not every revision in the BitKeeper can compile/run perfectly. Right now I have to find a "good" revision and manually request it with the Skoll Client. I need to get access to MySQL push-build tar balls to solve this bad revision problem.

KEY CONCERNS

• More than ever, the future progress of the project depends on having push-build tar balls.

TASKS IN THE UPCOMING WEEK

• Analyze the collected runtime data from Skoll Client and research ways to process these data.

Week 2 - A Test Scheduler for the MySQL Build Farm Initiative - Mon, 11:30 PM Jun 9 2008

KEY ACCOMPLISHMENTS LAST WEEK

• Continued with HTML testing report pages modifications, the new HTML pages report all the information collected by the current Skoll Client.
• Began integrating new data collection features into Skoll on the server side, most of these changes are made in our development database.
• Read research papers on reducing testing space.

KEY TASKS THAT STALLED LAST WEEK

• None

KEY CONCERNS

• Future progress of the project depends on having push-build tar balls.

TASKS IN THE UPCOMING WEEK

• Finish the integration of new data collection features in Skoll on the server side.
• Modify Skoll's upload manager and data processor to handle the separate log files collected by the new Skoll Client. Ideally, Skoll can generate a report page for each step of the testing process (e.g. source download, compilation, testing).

Week 1 - A Test Scheduler for the MySQL Build Farm Initiative - Mon, 11:31 PM Jun 2 2008

KEY ACCOMPLISHMENTS LAST WEEK

• I integrated some changes from the past semester into the Skoll Client source code. These changes will allow the client to collect more detailed information of the testing process. One of such change is creating a separate log file for each step of the testing process.
• I created a mock-up for the new HTML testing report pages. The goal of the new report pages is to combine the interfaces of push-build and Skoll. Push-build provides very detailed information of testing process, while Skoll is good at visualizing huge number of test results. The new interface looks promising, but lots of improvements will be added later on.
• I began to modify the module that's responsible for generating the HTML testing report pages. This week I separated test result generation from HTML generation, so in the future other reporting interfaces can be implemented for the test results. Eventually, the test results should be stored in a database for easier report generation.
• I got confirmation from MySQL team that someone is looking into providing the push-build tar balls.

KEY TASKS THAT STALLED LAST WEEK

• None

KEY CONCERNS

• Future progress of the project depends on having push-build tar balls.

TASKS IN THE UPCOMING WEEK

• Continue with HTML testing report pages modifications
• Read research papers on test space reduction techniques

Week 0 - A Test Scheduler for the MySQL Build Farm Initiative - Mon, 10:59 PM Jun 2 2008

KEY ACCOMPLISHMENTS LAST WEEK

• Even though this is the start of the SoC, I have been working on MySQL testing for some time now. My research group at University of Maryland has been working on automated testing for highly configurable software, and we have been testing MySQL for awhile. Our project is called Skoll (http://www.cs.umd.edu/projects/skoll/contribute/mysql.html), which automatically tests highly configurable software on distributed computing resources.

KEY TASKS THAT STALLED LAST WEEK

• Finished all my final exams and projects last week, ready to start this project full time.

KEY CONCERNS

• I contacted MySQL to get access to push-build source tar balls, so Skoll can test each "push release" of MySQL instead of each check-in. I am waiting for MySQL to provide a way to notify and distribute the tar balls when they are available.

TASKS IN THE UPCOMING WEEK

• Modify Skoll's html testing report pages to match push-build's
• Integrate some improvements to configuration modeling into Skoll (currently in different branches of Skoll code)
• Follow up on push-build tar balls access

Project Description - Mon, 10:53 PM Jun 2 2008

The project I am working on for Google Summer of Code 2008 is "A Test Scheduler for the MySQL Build Farm Initiative". The MySQL Build Farm Initiative seeks to create an automated environment that tests MySQL in multiple configurations over a powerful, virtual computing grid provided by community member's local machines.

Currently, MySQL developers use an internal system called push-build to test a static set of configurations across a static set of computing resources. My project will take a step towards realizing the MySQL Build Farm; modify push-build to test a dynamic, rather than static, set of configurations and computing resources.

I will do this by creating a management layer that

1. models the configuration space to be tested
2. selects specific configurations from this space to be tested based on programmable coverage criteria
3. packages and distributes build and test scripts to clients who execute them and report results

I will also investigate new visualization methods for displaying test data from large numbers of configurations. Currently, approaches simply provide tabular results that are too voluminous to use and that can hide critical patterns in the data.

Any progress and changes I make will be documented here on this blog. My plan is to provide a project update on this blog every Monday.