Scaling CI–switching poll to push

October 21, 2014 on 10:13 pm | In Continuous Integration, DotNet, SVN, TFS, Windows | No Comments

Scaling CI has many flavors. For example:

When:

  • Code base / test no. increases -> build time increases,
  • Teams grow,
  • No. of projects grows.

Then:

  • Create targeted builds (dev build, qa build),
  • Write fast unit tests,
  • Smaller teams with local integration servers,
  • Modularize the code base:
    • Scale hardware,
    • Add more build agents,
    • Parallelize.

and last but not least:

  • Ease the source control system.

Let me show you how to make Subversion and (TFS) Git pro actively inform Jenkins CI about changes in source control.

The most straight forward way to let the CI server know that something changed in the repository is to configure polling. What it means is that the CI server periodically asks the source control system “do you have changes for me”. In Jenkins CI you are configuring it under “Build Triggers” and “Poll SCM”. Jenkins uses Cron style notation like this:

image

Five stars “* * * * *” means: poll every minute. Ovary minute is as close to continuous as you can get. More often is not possible. Most of the times it is not a problem. Once a minute is quite enough. But what if you have many repositories under CI. The single Jenkins CI requests cost not so much, but if there are many repositories to check it can mean a significant delay.

There is a way to change it. Switching from poll to push. How about letting source control system inform the CI server “I have something new for you”. The mechanism that makes it possible is called hooks (at least its hooks in Subversion and Git). Hooks are scripts that are executed in different situations. On the client before or after commit in (pre-commit, post-commit). Before or after update (pre-update, post-update) and so on. Or on the server before or after receive (pre-commit, post-commit). What is interesting for us are post-commit hook in Subversion (look for hooks subdirectory on the server) or post-receive in Git (look in .git\hooks). Because Git is distributed you have it in every repo but, the one that is interesting for us is of course the repo destined for the CI server, and from its point of view it is the post-receive hooks that needs to be executed. In those hooks you can do basically everything you want. We will get back to it soon.

On the the Jenkins CI side you change to change the trigger to “Trigger build remotely”. This option is only visible if your installation of Jenkins is not secured with long and password.

image

In this case you can always trigger the build by simply calling the URL:

http://[jenkins_server]/jobs/[job_name]/build

If your installation is secured you have to flag the “Trigger build remotely” and you can set the security token for the build. Only with this token the build will be triggered.

image

The URL that needs to be called in this case is

http://[jenkins_server]/jobs/[job_name]/build?token=[token]

If you have the repository viewable without authentication it will be possible to trigger the build. But sometimes the Jenkins CI will be secured that way that nothing is viewable without log in. How to trigger a build in this case? Well there is a plug-in for that. It is called “Build Authorization Token Root Plugin” and it is available under https://wiki.jenkins-ci.org/display/JENKINS/Build+Token+Root+Plugin. In this case the URL will be

http://[jenkins_server]/buildByToken/build?job=[job_name]]&token=[token]

We are ready on the Jenkins CI side. Lets make it ready on the source control system side. Since we are Microsoft minded at CODEFUSION (my company). We have Subversion on our own Windows Server and Git on Microsoft Visual Studio Cloud.

In Subversion go to the server and look for the repositories. Go to repository you want to trigger and to hooks subdirectory. Create a file called post-commit.cmd. Subversion will run this script every time something comes in. We want to simply call an URL. Under Linux you would use the curl command. Here you can do it also but you will have to download the curl for Windows and place it somewhere on the server. But there is a better way. You can use PowerShell do call the URL. So create a post-commit.ps1 file (the name does not matter actually but lets keep it “in ordnung”). Inside write the script:

[System.Net.ServicePointManager]::ServerCertificateValidationCallback = {$true}
$url="https://[jenkins_server]/buildByToken/build?job=[job_name]]&token=[token]"
(New-Object System.Net.WebClient).DownloadString("$url");

The first line is only if you have Jenkins running over SSL with self issued certificate (like we have). In the second line please fill the gaps with to form correct URL. The third line calls this URL. Nice thing about it you have most likely PowerShell installed if you are on modern Windows Server.

Now call the PowerShell script from the post-commit.cmd like this:

PowerShell -NoProfile -ExecutionPolicy Bypass -Command "& '%~dp0post-commit.ps1'"

The NoProfile and ExecutionPolicy switches are to make it possible to call a script from command line. In Command switch pay attention to the syntax. The %~dp0 switch means current directory (of course).

Now check something in and watch the build being triggered (if it’s not – check it once again – it worked on my machine).

Now Git. We were using TFS Git from visualstudio.com. There is no access to hooks under TFS. But Microsoft was kind enough to make it possible in other way. Log into visualstudio.com. Go to your project and look for “Service Hooks”.

image

It lets you integrate with various 3rd party services. One of them is Jenkins CI.

image

I would like Microsoft to let me make simple URL call among those “Services”. Please. But since it is not possible let’s choose Jenkins.

image

Decided to trigger the build after every code push. You can set the filers to get it triggered only for certain repos or branches. Then choose to trigger generic build and provide all the necessary information like Jenkins URL, user name, API token (more to it later), build (it is job name provided automatically) and build token (as in case of SVN – provided by Jenkins when you configure “Trigger build remotely”). To get the API token on Jenkins CI go to “People”, search for the configured user and choose “Configure”

image

Look for API token and use it on visualstudio.com.

Test it and check it the build was triggered. It should. It worked on my machines.

I hope it was useful!

Vanilla build server and a little NuGet gem

October 6, 2014 on 7:37 pm | In ASP.NET MVC, Continuous Integration, DotNet, MSBuild | No Comments

Vanilla build server is a concept that says that the build server should have as few dependencies as possible. It should be like vanilla ice cream without any raisins (I have raisins in ice cream). Let me cite the classic (from: Continuous Integration in .NET):

“It’s strongly suggested that you dedicate a separate machine to act as the CI server. Why? Because a correctly created CI process should have as few dependencies as possible. This means your machine should be as vanilla as possible. For a .NET setup, it’s best to have only the operating system, the .NET framework, and probably the source control client. Some CI servers also need IIS or SharePoint Services to extend their functionality. We recommend that you not install any additional applications on the build server unless they’re taking part in the build process.”

I was recently preparing a talk for a conference and setting up a brand new CI server on Windows Server 2012. My ASP.NET MVC project build ended up of course with following error:

error MSB4019: The imported project "C:\Program Files 
(x86)\MSBuild\Microsoft\VisualStudio\v11.0\
WebApplications\Microsoft.WebApplication.targets" 
was not found. Confirm that the path in the <Import> 
declaration is correct, and that the file exists on disk.

Well of course. I have a vanilla machine without any MSBuild targets for ASP.NET MVC. I was going to solve it like usual. Create a tools directory, copy the needed targets into the repository and configure the MSBuild paths to take the targets provided with the repository. It worked like a charm in the past and it would work now. But something (call it intuition) made check over at NuGet and to my joy I found this little gem:

https://www.nuget.org/packages/MSBuild.Microsoft.VisualStudio.Web.targets/12.0.1

“MSBuild targets for Web and WebApplications that come with Visual Studio. Useful for build servers that do not have Visual Studio installed.” Exactly!

I quickly installed it. Configured the MSBuild on the build server to use it like this:

/p:VSToolsPath=’..\packages\MSBuild.Microsoft.VisualStudio.Web.targets.12.0.1\tools\VSToolsPath’

It is a command line parameter I’ve added to the build arguments.

An voila!

Powered by WordPress with Pool theme design by Borja Fernandez.
Text © Marcin Kawalerowicz. Hosting CODEFUSION.