Scaling CI–switching poll to push

October 21, 2014 on 10:13 pm | In Continuous Integration, DotNet, SVN, TFS, Windows | No Comments

Scaling CI has many flavors. For example:

When:

  • Code base / test no. increases -> build time increases,
  • Teams grow,
  • No. of projects grows.

Then:

  • Create targeted builds (dev build, qa build),
  • Write fast unit tests,
  • Smaller teams with local integration servers,
  • Modularize the code base:
    • Scale hardware,
    • Add more build agents,
    • Parallelize.

and last but not least:

  • Ease the source control system.

Let me show you how to make Subversion and (TFS) Git pro actively inform Jenkins CI about changes in source control.

The most straight forward way to let the CI server know that something changed in the repository is to configure polling. What it means is that the CI server periodically asks the source control system “do you have changes for me”. In Jenkins CI you are configuring it under “Build Triggers” and “Poll SCM”. Jenkins uses Cron style notation like this:

image

Five stars “* * * * *” means: poll every minute. Ovary minute is as close to continuous as you can get. More often is not possible. Most of the times it is not a problem. Once a minute is quite enough. But what if you have many repositories under CI. The single Jenkins CI requests cost not so much, but if there are many repositories to check it can mean a significant delay.

There is a way to change it. Switching from poll to push. How about letting source control system inform the CI server “I have something new for you”. The mechanism that makes it possible is called hooks (at least its hooks in Subversion and Git). Hooks are scripts that are executed in different situations. On the client before or after commit in (pre-commit, post-commit). Before or after update (pre-update, post-update) and so on. Or on the server before or after receive (pre-commit, post-commit). What is interesting for us are post-commit hook in Subversion (look for hooks subdirectory on the server) or post-receive in Git (look in .git\hooks). Because Git is distributed you have it in every repo but, the one that is interesting for us is of course the repo destined for the CI server, and from its point of view it is the post-receive hooks that needs to be executed. In those hooks you can do basically everything you want. We will get back to it soon.

On the the Jenkins CI side you change to change the trigger to “Trigger build remotely”. This option is only visible if your installation of Jenkins is not secured with long and password.

image

In this case you can always trigger the build by simply calling the URL:

http://[jenkins_server]/jobs/[job_name]/build

If your installation is secured you have to flag the “Trigger build remotely” and you can set the security token for the build. Only with this token the build will be triggered.

image

The URL that needs to be called in this case is

http://[jenkins_server]/jobs/[job_name]/build?token=[token]

If you have the repository viewable without authentication it will be possible to trigger the build. But sometimes the Jenkins CI will be secured that way that nothing is viewable without log in. How to trigger a build in this case? Well there is a plug-in for that. It is called “Build Authorization Token Root Plugin” and it is available under https://wiki.jenkins-ci.org/display/JENKINS/Build+Token+Root+Plugin. In this case the URL will be

http://[jenkins_server]/buildByToken/build?job=[job_name]]&token=[token]

We are ready on the Jenkins CI side. Lets make it ready on the source control system side. Since we are Microsoft minded at CODEFUSION (my company). We have Subversion on our own Windows Server and Git on Microsoft Visual Studio Cloud.

In Subversion go to the server and look for the repositories. Go to repository you want to trigger and to hooks subdirectory. Create a file called post-commit.cmd. Subversion will run this script every time something comes in. We want to simply call an URL. Under Linux you would use the curl command. Here you can do it also but you will have to download the curl for Windows and place it somewhere on the server. But there is a better way. You can use PowerShell do call the URL. So create a post-commit.ps1 file (the name does not matter actually but lets keep it “in ordnung”). Inside write the script:

[System.Net.ServicePointManager]::ServerCertificateValidationCallback = {$true}
$url="https://[jenkins_server]/buildByToken/build?job=[job_name]]&token=[token]"
(New-Object System.Net.WebClient).DownloadString("$url");

The first line is only if you have Jenkins running over SSL with self issued certificate (like we have). In the second line please fill the gaps with to form correct URL. The third line calls this URL. Nice thing about it you have most likely PowerShell installed if you are on modern Windows Server.

Now call the PowerShell script from the post-commit.cmd like this:

PowerShell -NoProfile -ExecutionPolicy Bypass -Command "& '%~dp0post-commit.ps1'"

The NoProfile and ExecutionPolicy switches are to make it possible to call a script from command line. In Command switch pay attention to the syntax. The %~dp0 switch means current directory (of course).

Now check something in and watch the build being triggered (if it’s not – check it once again – it worked on my machine).

Now Git. We were using TFS Git from visualstudio.com. There is no access to hooks under TFS. But Microsoft was kind enough to make it possible in other way. Log into visualstudio.com. Go to your project and look for “Service Hooks”.

image

It lets you integrate with various 3rd party services. One of them is Jenkins CI.

image

I would like Microsoft to let me make simple URL call among those “Services”. Please. But since it is not possible let’s choose Jenkins.

image

Decided to trigger the build after every code push. You can set the filers to get it triggered only for certain repos or branches. Then choose to trigger generic build and provide all the necessary information like Jenkins URL, user name, API token (more to it later), build (it is job name provided automatically) and build token (as in case of SVN – provided by Jenkins when you configure “Trigger build remotely”). To get the API token on Jenkins CI go to “People”, search for the configured user and choose “Configure”

image

Look for API token and use it on visualstudio.com.

Test it and check it the build was triggered. It should. It worked on my machines.

I hope it was useful!

Keep your lib folder up to date

February 9, 2009 on 10:22 pm | In NUnit, SVN | No Comments

I’m keeping all the 3rd party assemblies that I use in my .NET projects in a separate library folder called lib. It is a part of my Subversion repository. In most of my project it is placed in linked repository (with svn:externals property). It lets my check out the whole project on any machine I like and it compiles right away.

But be aware! In such scenario (and, well… in any other scenario as well) you have to keep your lib up to date. Take a look at this nasty bug. I was using the NUnit version 2.4.6.0

Take a look at this test method:

[NUnit.Framework.Test]
public void Test()
{
    System.Text.ASCIIEncoding ASCIIEncoding = new System.Text.ASCIIEncoding();

    string str1 = "abc";

    byte[] array = ASCIIEncoding.GetBytes(str1.ToCharArray());

    System.Array.Resize<byte>(ref array, 10);

    string str2 = ASCIIEncoding.GetString(array);

    NUnit.Framework.Assert.AreEqual(str1, str2);
}

What would you expect?

image

Fail! Isn’t it?

No! 1 passed, 0 failed, 0 skipped, took 0,54 seconds.

Even NUnit hast bugs to! I’ve updated to the newest version and it works as a charm! So keep you lib folder up to date!

Multiple projects – multiple repositories

April 26, 2008 on 9:00 pm | In SVN | No Comments

Do you have one project you want to share between multiple solutions? Something like a big set of helpers or a framework you need here and there? There are multiple ways ones have achieved this goal. Let me describe how I’ve done this.

We work with Visual Studio as our main development environment and Subversion as our source control system. What we needed was a systematic approach where we share our framework across all of our projects. We are actively developing our framework so simple dll reference is not an option. Our rules according to framework development make it fairly save to share the same code across multiple projects and to develop at the same time.

We decided that the framework should by placed in separate SVN repository and a separate Visual Studio project. But we wanted the framework and the project being pullable from source control as a whole. Firstly because we are (I’m!) lazy and I don’t want to remember that I have to check out the framework and than the project separately. Secondly we have a living Continuous Integration system running on CruiseControl.NET and it should by able to detect the changes and build to project in one shot.

Our way is to add the framework as an existing project from separate solutions…

Project structure

and to connect the main SVN project repository with framework repository as a svn:external.

Folder structure

Step by step solution:

1. Create “Framework” Project in Visual Studio

2. Create a SVN repository for this project (check everything you need in)

3. Create new solution in Visual Studio that will hold your “Project”

4. Create a SVN repository for this project

5. Create new folder on hard drive inside the “Project” folder

6. Check out the framework from repository into newly created folder

7. In Visual Studio add the framework to you solution as a existing project.

Adding exisitng project

8. Link the Framework repository with the Project repository (using svn propset or with TortoiseSVN like this)

svn:externals

You are done!

Now if you check out the “Project” you will get the “Framework” automatically. Unfortunately if you will make changes into “Framework” you will not by able to checkc everything in. You will have to separately check the “Framework” changes and “Project” changes. But if you are using Tortoise SVN you will by warned.

TortoiseSvn externals warning

Is is cool or not!?

Powered by WordPress with Pool theme design by Borja Fernandez.
Text © Marcin Kawalerowicz. Hosting CODEFUSION.