Moving from CruiseControl.NET to Jenkins in the PVS-Studio development team

Now it's hard to imagine software development without automated project builds and testing. There are various ready-made solutions to minimize the time expenses for the integration of the modifications into the project. In this article I am going to speak about the way PVS-Studio team changed the continuous integration server from CruiseControl.NET to Jenkins I will also be talking about the motives behind this decision, the goals we tried to pursue and the issues we had to deal with during that process.

Introduction

Continuous integration - is an automated process of building, deploying and testing software. This practice is popular both among large development teams and individual developers. There is quite a number of solutions for this practice. In this article we are going to talk about free open source projects CruiseControl.NET and Jenkins.

CruiseControl.NET(CCNet) - a tool for continuous integration of software implemented on the .NET Framework. There are also variants of the tool in Java (CruiseControl) and a version for the Ruby-environments (CruiseControl.rb). You can view and manage the information about the builds through web-interface or a desktop utility. It integrates with different version control systems. Is an open source project but, unfortunately, is not developing since 2013.

Jenkins- a tool for continuous integration with open source, written in Java. It was forked from the Hudson project after an argument with Oracle. Providing functions of continuous integration, it allows automating a part of the development process, which does not require the involvement of human developers. The abilities of Jenkins can be extended through plugins. At the moment the project is actively developed and supported by the developers and the community.

Although this article may look like a review in the "CCNet vs. Jenkins" style, the choice is already made for a server with Jenkins. The main reason why we changed the tool for continuous integration is that the CruiseControl.NET project is no longer developing. There were also other issues when working with CCNet that we are going to cover later.

Our project has quite a long history - recently we had a 10-year anniversary, you may read the story in the article "PVS-Studio project - 10 years of failures and successes". We were using CCNet for more than 5 years. We got so used to its interface, settings and functions, that Jenkins seemed to be really uncomfortable. First, we started using it with the release of PVS-Studio for Linux. The switch to the Jenkins platform was preceded by a long study of the tool. Some time was spent on looking for the similar functions from CCNet. Further on, I will describe the most interesting moments from our work.

Our claims for CCNet

• CCNet is no longer developing. It can be used, but in case you want to expand the functionality or fix existing/potential errors, you will have to do that by yourself.
• The Source Code Management mode is working unstably, namely, for the automatic launch when there are changes in the version control system. In case of problems with the network, in this mode a project gets the status "Failed", even if it wasn't run. In practice, the problem occurs so often (sadly, our office doesn't have the most stable Internet access), that this mode becomes impossible to use.
• When querying SCM on the edits, in case of an error in the version control system (for example, if some directory was removed from a repository, specified in the settings, causing a tree conflict), the execution of the project will be immediately interrupted; still, the status will be kept "Success" - a project will cease working, but its status will remain "green " in the web interface and desktop utility. In such a mode the launch of tests can be not done for weeks and there is a risk that no one will pay attention to it, thinking that tests are working correctly.
• The general server performance log is too verbose and unstructured: it's hard to understand which step of the build failed and find a log specifically for this step. When running multiple projects in parallel, the build log gets "mixed". The XML log of the build of a separate project is available in the web interface, but quite often is not really detailed and doesn't contain all the commands that were run.
• Inefficient paralleling of subtasks inside a project. The subtasks are run in parallel by groups by the number of the processor cores. If long and quick commands get to the group, then the new tasks won't be run, until all the tasks from the previous run aren't completed.

Comparison of usage scenarios

Server settings

The settings of the CCNet projects (configuration of the server) were stored in one xml file, and various passwords in a different one. Although the settings file reached the size of ~4500 lines, it was still quite convenient to use it. By pressing Alt+2 in Notepad++, the whole list of projects can be minimized and you can edit the one you need (figure 1).

Figure 1 - Editing the CCNet settings in Notepad++

Although the file had some duplicate code, there were no difficulties with the server support.

Here is how the SCM block was configured:

<svn>
  <username>&SVN_USERNAME;</username>
  <password>&SVN_PASSWORD;</password>
  <trunkUrl>&SVN_ROOT;...</trunkUrl>
  <workingDirectory>&PROJECT_ROOT;...</workingDirectory>
  <executable>&SVN_FOLDER;</executable>
  <deleteObstructions>true</deleteObstructions>
  <cleanUp>true</cleanUp>
  <revert>true</revert>
  <timeout units="minutes">30</timeout>
</svn>

Here is how the MSBuild block was configured:

<msbuild>
  <description>PVS-Studio 2015</description>
  <workingDirectory>&PROJECT_ROOT;...</workingDirectory>
  <projectFile>...\PVS-Studio-vs2015.sln</projectFile>
  <buildArgs>/p:Configuration=Release</buildArgs>
  <targets>Build</targets>
  <timeout>600</timeout>
  <executable>&MSBUILD14_PATH;</executable>
</msbuild>

Here is how the block for common tasks was configured

<exec>
  <description>PVS-Studio 2015 sign</description>      
  <executable>&PROJECT_ROOT;...\SignToolWrapper.exe</executable>
  <baseDirectory>&PROJECT_ROOT;...</baseDirectory>
  <buildArgs>"&SIGNTOOL;" ... \PVS-Studio-vs2015.dll"</buildArgs>
  <buildTimeoutSeconds>600</buildTimeoutSeconds>
</exec>

On the basis of such a project file, CCNet displays very conveniently all the executable steps (albeit only in the desktop tray utility, for some reason the web-interface didn't support this). In Jenkins we had some troubles with the "high-level" display of stages of completing an integration project, but we will speak about that later.

In Jenkins you have to store quite a number of settings files: a server configuration, settings files for some plugins, each project has its own configuration file. Although all of these files exist in the xml format, they aren't very convenient to view (at least in comparison with CCNet), because all the commands in the tags are written as plain text. The truth is, this is largely connected with the ideology of the use of the instrument. In CCNet the config is written manually, so it can be "nicely" formatted. Jenkins suggests editing project settings using its web interface and automatically generates configs.

Here is an example of how the commands look like in the Jenkins configs:

<hudson.tasks.BatchFile>
  <command>CD &quot;%BUILD_FOLDERS%\Builder&quot;&#xd;
PVS-Studio_setup.exe /VERYSILENT /SUPPRESSMSGBOXES ...&#xd;
Publisher_setup.exe /VERYSILENT /SUPPRESSMSGBOXES</command>
</hudson.tasks.BatchFile>

And this is quite a small example.

View the task statuses

As I have written before, in CCNet the projects are configured with Task blocks. Here is how a successfully completed task looks with all the steps (figure 2).

Figure 2 - viewing the status of tasks in the CCTray (a desktop client for CCNet)

An error in any of the blocks can be clearly seen in the hierarchy of subtasks. It is very convenient and illustrative visualization of the integration process. We almost never had to search for logs, it was clear looking at the description what requires checking on the local computer. We could not find such a feature in Jenkins, that's why we had to study this question really thoroughly before moving to Jenkins.

We can draw such analogy between CCNet and Jenkins: there is a project in CCNet (Project), the 'step' in this project is called a Task (as you can see on the figure above). In Jenkins there is also a project (Job), but its 'steps' are called the same - Steps (figure 3).

Figure 3 - Corresponding names in CCNet and Jenkins

Unfortunately, the web interface of Jenkins cannot visualize individual steps - the Job has only a full console log of all steps together. A big disadvantage is that you cannot see in Jenkins which of the steps failed - you need to watch the full log of the Job build. And as people get used to good things very quickly, we didn't want to abandon the old usage scenario. At that time we discovered Multijob Plugin

This plugin allowed us making the following improvements:

1. Using Jobs as Steps in other Jobs. Thus we got universal Jobs that allowed separating the log of certain subtasks and the main thing to view statuses of separate subtasks. The Jenkins web interface can visualize quite well the completion of individual Jobs within a Multijob - it's exactly what we were looking for. Figure 4 shows an example of a completed Multijob.

Figure 4 - viewing a completed Multijob.

2. We managed to get rid of duplicate code using universal Jobs. For example, there is a compilation of some utility: for the distribution, for tests, for the code analysis. In CCNet these were same Task blocks in 3 different projects. To compile this utility in Jenkins there is a Job that is used by several Multijobs.

3. During the creation of projects in Jenkins, we have the following workflow. We divide all the Jobs into Multijob "projects" and universal "steps". The names of the universal Jobs have a prefix "job_" and do not suggest using as a separate (stand-alone) project. They also don't have loading the source code from the repository. The names of Multijobs have the "proj_" prefix and include loading of the source code and running only other Jobs. (We try to avoid Steps, because they are not visualized).

The universal Jobs are run with the following parameter:

WORKSPACE=$WORKSPACE

It means that the Job will be run in the working directory of the Multijob.

Thus we can get a log of the source files updates and logs of all the steps of the build separately. It would be hard and meaningless to follow this ideology for all the projects. It is done only for several biggest and most important projects that should be studied in detail in case of problems.

4. You can configure conditional and parallel launch of the Jobs in Multijob. Multijobs can run Multijobs using the same ways. Thus you can combine launches of different projects: to start the build of all installers or all the tests.

Viewing the build logs

It was extremely inconvenient to view the build logs of the projects, because they were mixed with the server output and had a special markup. There is no such a problem in Jenkins and there is a possibility to separate the logs for subtasks in some projects.

Getting the revision of the source code

In Jenkins a separate revision of the version is defined for every added link to the SVN repository. I.e. if we add several directories, the numbers can vary greatly, but we need only one maximum number.

According to the documentation, we should work with this as follows:

If you have multiple modules checked out, use the svnversion command. If you have multiple modules checked out, you can use the svnversion command to get the revision information, or you can use the SVN_REVISION_<n> environment variables, where <n> is a 1-based index matching the locations configured.

It was done in this very way: of all the obtained values SVN_REVISION_<n> , the maximum value was chosen and added to the built programs.

Useful plugins for Jenkins

The enhancement of the tool with the help of the plugins allows flexible configuration of the server. The plugins to run the tests in Visual Studio are perhaps the only ones that we refused to use. They had additional mandatory launch parameters, that we didn't use that's why it was easier to create a universal Job that would just start the tests from a command line. Unfortunately, Jenkins, being "out of the box" couldn't do much of what we got so used to in CCNet. However, we manage "to get back" all the necessary functionality.

Here is a list of those plugins that we found useful, with a small description:

• Multijob plugin- allows using other Jobs as stages of the build with the possibility of sequential and parallel execution.
• Environment Injector Plugin - a plugin that is used to specify global passwords. They are used as environment variables, and the plugin hides the variable value in the log.
• Pre SCM BuildStep Plugin - adding extra steps before executing the commands of the version control system.
• MSBuild Plugin - a handy plugin to build projects using MSBuild. In the settings you specify the paths to different versions of MSBuild only once, later in the project you can easily add build steps.
• Parameterized Trigger plugin - add parameters of a project launch. For example, you can make a choice of a trunk/stable branch to build a distribution.
• Post Build Script Plug-in - performing additional steps after the build.
• Throttle Concurrent Builds Plug-in - this plugin allows adjusting the number of concurrently running project builds either globally or within a specific category. It lets you get the functionality in Jenkins similar to the queues in CCNet - an ability to execute in parallel several projects from different categories (queues), while providing consistent execution of the projects within a single queue. For example, we have Installers queues (distributions) and Tests. We want to have an ability to build some distribution in parallel with running the tests, but at the same time, tests shouldn't work in parallel - there won't be enough "cores" on the server.
• Build Name Setter Plugin - allows specifying the name in a necessary format: Major.Minor.Revision.Build.
• Dashboard View - allows adding a custom Jobs display in the browser. As we have universal Jobs, that are of no use to run manually, we created a list without them with the help of this plugin.
• ZenTimestamp Plugin - a handy plugin that adds timestamps in the build logs.

Overview of desktop clients

To receive notifications from the CCNet we used a client for Windows - CCTray.

Here are variants that are now available to work with Jenkins:

1. CCTray - this program can be used for a Jenkins server as well. The projects will look approximately the same as they were before (figure 5).

Figure 5 - CCTray screenshot

The description of CCTray as a Jenkins client:

• It is not developing, the same as CCNet;
• Can't show subtasks (works only for CCNet);
• Cannot run the projects;
• You can go to the project page by clicking on the title;
• Configurable display of projects (Icons, List, Details);
• Open source.

2. CatLight (figure 6)

Figure 6 - a screenshot of CatLight

Client description:

• At this moment it is a beta version, the final version will be paid;
• There are crashes during the setup and work, glitches in the interface;
• When the PC "wakes up" after the hibernation, the status on the dashboard doesn't get automatically updated.
• Can't show subtasks for Multijobs;
• Cannot run the projects;
• The project display is not customizable (the only possible display - figure 6);
• You can go to the project page by clicking on the title;
• Ability to view the status of last 5 launches and jump-click to them;
• Ability to view the progress of the running project;
• If several servers are added, they are conveniently separated by a line;
• Versions for Windows, Linux and Mac.

3. Kato (figure 7)

Figure 7 - screenshot of Kato

Client description:

• Can't show subtasks for Multijobs;
• Can run the projects; Unfortunately, does not support projects with a parameterized launch - the utility crashes when trying to run such a project;
• Projects from different servers are displayed in one list and are not separated in any way (not always convenient);
• Configurable project display (List, Grid);
• You can go to the project page by clicking on the title;
• You can view the latest log directly in the client, but due to lack of monospace text it is not very convenient;
• Open source.
• Only for Windows.

4. CCMenu - a client only for Mac, open source. It is not relevant for us, but perhaps, some may need it.

Conclusion

Using CI is useful in any project. There is a great free tool Jenkins that was reviewed in the article, and also there are a lot of other free and paid Cl. It is rather pleasant to use a developing the project: a large number of upgrades are released for Jenkins and plugins. New solutions get created, as for example, Blue Ocean project that is still in Beta. You may find it on the main page of the Jenkins site.

Although, the clients for monitoring Jenkins projects weren't much impressive to me. A lot of obvious features are missing. Perhaps desktop clients are particularly in demand and it is more correct to use only some web interface.

When moving to a new server, we couldn't use Jenkins as a Windows service, because in this mode the UI tests do not run. We found a way out by setting up a server as a console application with a hidden window.

If you have any comments to this material of some interesting solutions of the problems we've mentioned here, we'll be glad to get them in the comment section of via a feedback form.