This is Xenofarm The Grand Design of Everything The Server Script The Web Export The Compilation Client The Web Import The Result Processor The Xenofarm project is a rather small project, if measured in code. This is because the wonders of modern computer engineering lets us reuse components to build advanced applications with ease, letting us programmers focus on important tasks as design and quality. Did you buy that? Well, normally the world isn't as perfect as we like it to be, but in this case it is, since Xenofarm is just some glue components holding together other components. The goal of Xenofarm is to aid programmers who want to build extremely platform independent applications. Most programmers would say, "If I want to do something platform independent I use Pike/Python/Perl/Java/LOGO", but there are still some guys struggling to get these programming languages to work on all strange OS on all strange platforms. It is for them (us) this tool is built. From a historical point of view, Xenofarm is a complete rewrite of the automated build and regression test tool that was used by the Pike programming language development team between 1999 and 2002. It was called AutoBuild and was created by Martin Nilsson and Johan Schön at Roxen Internet Software. AutoBuild was in turn derived from (or rather inspired by) Tinderbox. Xenofarm still sticks to the basic AutoBuild concept, but was redesigned to get rid of the few annoying flaws and shortcomings of its predecessor. Most notably the system components has been more loosly coupled and the system should scale to X different clients performing Y different projects for Z different servers. SH-client by Peter Bortas HTTP PUT by Per Hedbor The rest by Martin Nilsson Xenofarm is released under GNU GPL. The basic idea with Xenofarm is to help developers solve bugs, as compared with the Mozilla projects Tinderbox where the goal is to find the bugs as quickly as possible. We believe that it is better to get "the full picture" of what is broken, and as much information as possible, before starting to fix the code. There are essentially three ways that differs between Tinderbox and Xenofarm in this aspect; 1. Xenofarm synchronizes its build on all platforms. It creates a source package that is distributed to all clients and the build result is presented as a group. It is then easy to see if e g all Sparc computers failed or all Solaris computers failed, which eases the bug hunt. 2. Xenofarm applies CVS checkout latencies, to try to minimize mid-checkin breaks. The latency is currently based on a simple threshold value, but a more advanced solution is in the works. 3. "All" relevant information is collected from the build machines to the front end. This enables people with no login access to the computers to help in the bug hunt. Another design goal with Xenofarm was to scale better than AutoBuild did; In terms of number of clients, in terms of number of projects, in terms of number of source locations and in terms of distance. Conceptually the system has a number of source package producers and a number of compilation clients. Every compilation client then decides which packages they want to "subscribe" to. The workflow is as follows: 1. Some trigger mechanism signals that it is time to make a new build. The trigger is typically a CVS checkin. 2. The server script creates a new source package and exports it through a web server. 3. The client polls the web server and finds a new package, which it downloads. 4. The client builds/tests/whatever the package and uploads the result to a web server, typically the same web server. 5. A result processor takes the uploaded result and derives interesting facts, which it stores in a database. The result files are moved into an exported web directory. 6. The result summary and all the exported files can be viewed from a web interface. The server script is responsible for creating new source distributions when triggered. Since the composition of a source distribution varies a lot from project to project, we've taken the approach to include all the base functionality into one pike program file and then inherit that file into "project programs" that contains logic specific for that project. The rationale of course is that programming languages are better suited for this type of configurational task than any artificial configuration file language one might come up with. The base program is however useful in its own to create a "generic" source distribution from a CVS archive. The base server program is started with ./server.pike arguments project where arguments are from the list below and project is the name of the xenofarm project. You need one server per project. Many of the settings have hard coded defaults, but in some cases that was not possible, so some arguments are mandatory. These are cvs-module, db, web-dir and work-dir. Arguments: --cvs-module=<name> The CVS module the server should use. --db=<database url> The database URL, e.g. mysql://localhost/xenofarm. --force Make a new build and exit, regardless if there has been a new chekin or if we are within the min-distance quarantine or checkin latency. --help Displays a summary of the available arguments and their use. --latency=<seconds> The enforced latency between the latest checkin and when the next package is made. Defaults to 300 seconds (5 minutes). --min-distance=<seconds> The enforced minimum distance in time between to builds. Defaults to 7200 seconds (two hours). --poll=<seconds> How often the CVS is queried for new checkins, not including the time when the server is enforcing minimum build distance and when the server is enforcing build latency. Defaults to 60 seconds. --repository=<path> The CVS repository the server should use. This string is given as the -d argument to the CVS binary. --verbose Send messages about everything that happens to stdout. It might be a good idea to start the server with ./server.pike <args> --verbose <project> > server.log & --web-dir=<path> Where the outgoing source packages should be put. --work-dir=<path> Where temporary files should be put. The Xenofarm client fetches new source distributions from a specific web server URL. From that URL it expects to get a file named snapshot.tar.gz. It also expects the Last-Modified header to change to a later date and time when a new source distribution is made available. The Roxen WebServer module "Xenofarm I/O module", found in roxen_modules/xenofarm_fs.pike, is mounted as a file system module in the webservers virtual file system. By default it mounts to the path /xenofarm. The module will export the source distributions put in the path entered in "Dist search path" at the module mount point. The module expects all source distributions to be named as XXXX-YYYYMMDD-hhmmss.tar.gz, where XXXX is the name of the project, e g Pike7.3-20020509-203344.tar.gz. These files are directly accessible from the mountpoint, e g /xenofarm/Pike7.3-20020509-203344.tar.gz in this case. The latest version is exported through the stable URL mountpoint/latest, e g /xenofarm/latest in this case. This in turn returns a redirect to a snapshot.tar.gz in a subdirectory named after the latest source distribution. In this case /xenofarm/latest would redirect to /xenofarm/Pike7.3-20020509-203344/snapshot.tar.gz. The Last-Modified header is derived from the file name, not the actual mtime of the file, so the system is robust and handles dirty hacks that move the files between systems. It is of course possible to use a less complex solution for the distribution. Just put the files in a normal directory, add a symlink from snapshot.tar.gz to the latest source distribution, and ensure that the webserver handles the Last-Modified correctly when you update your files. The compilation client is the collective name of the software running on the client side, taking care of downloading, compilation, data gathering and result submission. The compilation client is written in sh scripts who mostly use standard unix components. The non-standard components required are wget for downloading source distributions and a little C utility that performs HTTP PUT, used for uploading the result. There are two reasons for why we have taken the trouble to do the client in this way: Although Pike is very portably we have to admit that sh and wget are even more so. This choice of language will enable even more platforms to become compilation clients. Since a lot of files in the Pike source code are generated by Pike scripts, it is good to ensure that the clients do not have any easily accessible Pike installed. This checks if any make rules in the source distribution target is broken so that some files are not generated. For more information about the compilation client, see the README in the client directory. When the client has built its result file, it is uploaded through HTTP PUT to a URL specified in the client configuration file. The "Xenofarm I/O module" is able to be in the receiving end and store the results in the "Result search path" directory. The module accepts all files that are uploaded to "result" in its mountpoint. Given the default mountpoint /xenofarm, the default result URL path is /xenofarm/result. The uploaded files are named "res" followed by the posix time, an underscore, a counter incremented for every upload since the module started, and finally ".tar.gz". The file name might look like res1021228036_31.tar.gz. Again it is possible to use any webserver and set up a path that accepts HTTP PUT. Possible solutions to minimize the risk of clashes is to let the clients name the result packages differently, or letting all clients have their own upload URL. Once the result packages are uploaded to the result server, which need not be the same machine as the export server, they are processed by the result parser. The result parser is a fairly simplistic script that checks the in directory for new files at given intervals, and if a new script is found does the following: 1. Unzip and untar the package into a work directory. 2. Retrieve the build id. 3. Retrieve the machine id, e g the host name and uname of the client machine. 4. Find out how many of the possible steps where completed, e g compilation and verification but not packaging. 5. Count the number of compilation warnings. 6. Store the results of the parsing in a database. 7. Move the log files to a directory reachable from the front end web server. 8. Delete the uploaded package.