Automation of daily build process with TlosLite

The aim of this paper is to describe the importance of job scheduler for build automation. This sample use case  is the daily build process of our new product Tlos Space Wide, the comprehensive workload automation solution. The build process is managed by our Open Source Free Job Scheduler TlosLite v1.8.1 which can be downloaded from here. With the help of the scheduler, the committed sources are built on daily basis and generates an email according to the build process result if success or fail. A windows based system is considered for this case.

The purpose of using a scheduler is to define the build process by steps and chain the steps to be managed by the scheduler so as to detect any failures on the relevant steps without losing the whole job flow. This is really a lifesaving process if your process is complicated and includes lots of jobs. In our case the build process does not have so many steps but if you are dealing with some kind of ETL, then the job amount is around hundreds.

For our case, the build process is a combination of three steps;

1. Export the committed sources to local folder on build platform
2. Compile and package the sources
3. Deploy the generated packages and do the commits if necessary 

For each of these three steps, we are going to prepare ant xmls and then use these xmls for creating our jobs to be defined later in TlosLite configuration files. The general configuration and use of TlosLite is not covered in this paper. Just the simple job definitions will be explained shortly. For full info please watch the video or read the documentation.



Step 1: Export the sources

The export step covers the checkout operations of the related projects from svn. This is the default target and named as checkoutCode and depends on init, checkoutSharedLibs, cleanUp and checkoutCode targets. Conceptually it is possible to divide these dependencies in to separate jobs but it is not necessary for our case.

The sample line of the checkout is like;

<echo> ${TlosSW_V1.0} </echo>
<svn username="${svnusername}" password="${svnpassword}">
<export srcurl="${exportURL}/${TlosSW_V1.0}" revision="${exportRevision}" destPath="${exportDir}/${TlosSW_V1.0}" />
</svn>

The original export.xml file can be downloaded from here.

After creating our export xml, it is time for the preparation of the first job, which will be named job1 and the batch file will be named as job1.bat.

Create job1.bat  file with your famous editor and put the line below in to the file and exit with saving.

ant -f D:\releasemanagement\RelMan\TlosSW\tlosSW\export.xml > D:\releasemanagement\RelMan\ReleaseManager\logs\TlosSW_export.log

This line tells ant that, execute the export.xml and direct all the output to the given file.

The first job is ready !


Step 2: Compile and package the sources

The second step includes the targets that are calling the related build scripts of the projects required for the product. As the product has very complex structure it consists of eight projects. If you want more control over the process you may divide this script in to eight scripts and chain together.

The default target for this file is named generateJars and depends on init.

The sample line of the build xml is like;

<echo> Generating ${TlosSW_V1.0}.jar </echo>
<ant antfile="build.xml" dir="${exportDir}/${TlosSW_V1.0}" inheritAll="false" inheritRefs="true" >
<property name="minor.number" value="${minor.number}"/>
</ant>

The original buildProjects.xml file can be downloaded from here.

After the xml is created, lets follow the job file creation process as in the first step. The job name is job2 and the script name is job2.bat.

This is the content of the second job;

ant -f D:\releasemanagement\RelMan\TlosSW\tlosSW\buildProjects.xml > D:\releasemanagement\RelMan\ReleaseManager\logs\TlosSW_buildProjects.log

The second job is ready !

Step 3: Deploy the generated packages and do the commits if necessary

This is the last step of our process. The last xml is named as deploy.xml and the default target is named as deploy and depends on clean. In this xml, the packaged jar files are copied to the deployment path and some script files are generated automatically to comply the build number hierarchy.

The sample line of the build xml is like;

<copy file="${exportDir}/TlosSW_V1.0/tlossw-server-V1.0_b${buildnum}.jar" todir="${test.dir}/Sunucu/lib/" />

The original deploy.xml file can be downloaded from here.

The last batch file will be job3.bat, and the jıob id will be job3. Here is the content of the last batch.

ant -f D:\releasemanagement\RelMan\TlosSW\tlosSW\deploy.xml > D:\releasemanagement\RelMan\ReleaseManager\logs\TlosSW_deploy.log

After creating all three scripts, it is time to define these jobs in Tloslite job definition xml to manage the process. There will be three jobs and will be chained in the same order. If any job fails, the process waits for user interaction, for the operational correction.


Defining jobs for the flow

In terms of TlosLite, three jobs means three lines in the job definition and configuration file, senaryoBilgileri.xml. Add these lines to xml and now the scheduler is ready to manage your flow.

TlosSW 1.0,D:\releasemanagement\RelMan\TlosSW\tlosSW\job1.bat,system,D:\releasemanagement\RelMan\ReleaseManager\logs\TlosSW_export.log,yok,S,30/12/2010 07:00:00,1800,false,0,1,error
TlosSW 1.0,D:\releasemanagement\RelMan\TlosSW\tlosSW\job2.bat,system,D:\releasemanagement\RelMan\ReleaseManager\logs\TlosSW_buildProjects.log,TlosSW1.0_job1,S,30/12/2010 07:00:00,1800,false,0,1,\null
TlosSW 1.0,D:\releasemanagement\RelMan\TlosSW\tlosSW\job3.bat,system,D:\releasemanagement\RelMan\ReleaseManager\logs\TlosSW_commit.log,TlosSW1.0_job2,S,30/12/2010 07:00:00,1800,false,0,1,error

The original senaryoBilgileri.xml file can be downloaded from here.


Conclusion

For the build automation, the root of complexity is not the amount of jobs to be managed but the flow of the build logic. With the help of this automation solution, it is very easy for the test department to get the latest code and deploy for the test purposes. This gives the flexibility to the test guys of not depending on the developer for the latest jar.

Hope this helps !