Channels ▼
RSS

Global Developer

Driving Continuous Integration from Git


If you're among the rising number of Git users out there, you're in luck: You can automate pieces of your development workflow with Git hooks. Hooks are a native Git mechanism for firing off custom scripts before or after certain operations such as commit, merge, applypatch, and others. Think of them as henchmen for your Git repo. Pre-operation hooks act as bouncers, guarding your repo with a velvet rope. And post-operation hooks are your Man Friday, faithfully carrying out follow-up tasks on your behalf.

Installing hooks for a Git repository is fairly straightforward, and well-documented. In this article, we focus on using Git hooks to augment continuous integration practices, starting with an example that makes combining Git and continuous integration (CI) less painful. The code is written in Ruby. Fortunately, Ruby is a language that highly prizes readability, so even if you don't know Ruby, you can easily follow along.

Automate CI Configuration for Git Branches

One of the blessings of Git is how easy it is to branch off and develop in isolation. This means the master stays releasable, you get the freedom to experiment, and your teammates aren't derailed if code from the experimentation proves to be half-baked. One challenge of Git, however, is how many branches a team ends up with — scores of active branches, most of which live for only a few days. Who is going to take the time to set up continuous integration for all those piddly little branches? Your henchmen, that's who.

To automatically apply CI to new development branches, you'll use the "post-receive" hook type. These are server-side hooks, triggered after pushes to the repository are completed. In such cases, you can use the post-receive hook to fire off a script that programmatically clones a master's CI configs and applies them to new branches using the CI server's exposed API. It might look something like this, when using the open-source and hugely popular Jenkins CI server:

#!/usr/bin/env ruby
 
 # Ref update hook for creating new Jenkins job 
 # configurations for newly pushed branches.
 #
 # requires Ruby 1.9.3+ 
 
 require 'yaml'
 require 'net/https'
 require 'uri'
 require 'rexml/document'
 include REXML
 
 # load ci-config.yml from hook directory
 def load_config
     hookDir = File.expand_path File.dirname(__FILE__)
     configPath = hookDir + "/ci-config.yml"
     puts configPath
     raise "No ci-config.yml found." unless File.exists? configPath
     YAML.load_file(configPath)
 end
 
 # Grab the configured Jenkins server
 config = load_config
 raise "ci-config.yml file is incomplete: missing jenkins_server" unless
     config["jenkins_server"]
 server = config["jenkins_server"]
 raise "ci-config.yml file is incomplete: username, password, url and
     default_job are required for jenkins_server" unless
         server['url'] and server['username'] and 
         server['password'] and server['default_job']
 
 # iterate through updated refs looking for new branches
 ARGF.readlines.each { |line|
     args = line.split
     oldVal = args[0]
     newVal = args[1]
     ref = args[2]
 
     if /^0{40}$/.match(oldVal) and ref.start_with?("refs/heads/") 
         # new branch!
         # retrieve the jenkins job config
         # TODO only need to do this once!
         uri = URI.parse(
            "#{server['url']}/job/#{server['default_job']}/config.xml")
         req = Net::HTTP::Get.new(uri.to_s)
         req.basic_auth server['username'], server['password']
         http = Net::HTTP.new(uri.host, uri.port)
         http.verify_mode = OpenSSL::SSL::VERIFY_NONE
         http.use_ssl = uri.scheme.eql?("https")
 
         # execute the request
         response = http.start {|http| http.request(req)}
 
         raise "Bad response from jenkins, is your ci-config.yml correct?"
             unless response.is_a? Net::HTTPOK
 
         # parse the config.xml from the response
         doc = Document.new response.body
         doc.root.get_elements(
             "//branches/hudson.plugins.git.BranchSpec/name").each { 
                 # overwrite branch to be our new ref
                 |elem| elem.text = ref 
         }
 
         # create a new request to upload the modified config.xml
         newJob = ""
         doc.write newJob
 
         newJobName = ref["refs/heads/".length..-1].gsub("/", "-")
         uri = URI.parse("#{server['url']}/createItem?name=#{newJobName}")        
         req = Net::HTTP::Post.new(uri.to_s, 
         initheader = {'Content-Type' => 'application/xml'})
 
         req.basic_auth server['username'], server['password']
         req.body = newJob
 
         # upload the new job
         response = http.start {|http| http.request(req)}
         raise "Failed to post new job to jenkins" unless 
             response.is_a? Net::HTTPOK
     end
 }

With this hook in place, you need only push a dev branch to the repo, and it will automatically be put under test. (It's possible to run CI builds against branches using a build parameter to represent the target branch, but that muddles the build history. The cloning approach provides a clean, clear history.) Applying every last facet of the CI scheme to branches isn't necessary — for example, running each and every branch through the load test gamut might be overkill. But even if you skip the load and UI tests, and run just unit and API- or integration-level tests, these are huge wins.

The risk of introducing defects into master is greatly reduced by testing on the branch before merging. Developers can also work more efficiently and confidently because of the frequent feedback on changes (instead of the old merge-then-pray technique). And for teams who include testing as part of their definition of "done," managers and scrum master types catch a break. With the Git hook automatically putting branch code under test, the team's practices and values are being enforced without the need for nag-mails or raised eyebrows during stand-up.

Vet Merges to Master

Two hallmarks of coding craftsmanship are an affinity for automated tests, and adherence to stylistic rules (such as avoiding empty try/catch blocks or duplicated code). Despite best intentions, everyone neglects best practices from time to time. That's where Git hooks come in. Pre-receive hooks living in the central repository qualify incoming pushes, making sure they're good enough to get past the velvet rope. Let's look at three hooks designed to protect master from slip-ups made on development branches.

Require Passing Branch Builds

The whole point of working on a development branch is to isolate yourself and create a space to experiment (read: "break stuff"). So it's natural to see failing tests on the branch while development is in progress. When it's time to merge to master, however, things had better be tidied up. This can be enforced programmatically with a hook that checks to see whether the incoming push is a merge to master, and if so, verify that all tests are passing on the branch before processing the merge.

If you happen to be using Bamboo, you can cleanly fetch test results for a given commit. If you use Jenkins or its predecessor, Hudson, you can fetch a set of recent build results then parse through them to see which builds ran against the commit in question. (This hook, and those that follow are implemented for the Bamboo CI server, but they can be implemented in more or less the same way on all CI servers.)

#!/usr/bin/env ruby
 
 # Ref update hook for verifying the build status of 
 # a topic branch being merged into 
 # a protected branch (e.g. master) from a Bamboo server.
 #
 # requires Ruby 1.9.3+ 
 
 require_relative 'ci-util'
 require 'json'
 
 # parse args supplied by git: <ref_name> <old_sha> <new_sha>
 ref = simple_branch_name ARGV[0]
 prevCommit = ARGV[1]
 newCommit = ARGV[2]
 
 # test if the updated ref is one we want to enforce green 
 # builds for exit_if_not_protected_ref(ref)
 
 # get the tip of the most recently merged branch
 tip_of_merged_branch = 
    find_newest_non_merge_commit(prevCommit, newCommit)
 
 # parse our Bamboo server config
 bamboo = read_config("bamboo", ["url", "username", "password"])

 # query Bamboo for build results
 response = httpGet(
    bamboo, 
    "/rest/api/latest/result/byChangeset/#{tip_of_merged_branch}.json")
 body = JSON.parse(response.body)        
 # tally the results
 failed = successful = in_progress = 0
 body['results']['result'].collect { |result|
     case result['state']
     when "Failed" 
       failed += 1
     when "Successful"
       successful += 1
     when "Unknown"
       if result['lifeCycleState'] == "InProgress"
         in_progress += 1
       end
     end
 }
 
 # display a short message describing the build status for 
 #the merged branch and abort if necessary
 if failed > 0
     # at least one red build - block the branch update
     abort "#{shortSha(tip_of_merged_branch)} has #{failed} 
     red #{pluralize(failed, 'build', 'builds')}."
 elsif in_progress > 0
     # at least one incomplete build - block the branch update
     abort "#{shortSha(tip_of_merged_branch)} has #{in_progress}
     #{pluralize(in_progress, 'build', 'builds')} that have not
     completed yet."
 else    
     # all green builds - allow the branch update
     puts "#{shortSha(tip_of_merged_branch)} has #{successful} 
         green #{pluralize(successful, 'build', 'builds')}."
 end

Enforce Code Coverage Requirements

Along with successful test runs, you want to make sure that new code added on development branches is tested as thoroughly as code already on master. This ensures that the overall test coverage level of the project doesn't drop when a development branch is merged back in. This, too, can be checked with Git hooks.

A simple Git hook can verify that coverage on the branch meets the minimum threshold. To enforce this, a hook can be created to compare the coverage rate on master with that of the branch, and reject the merge if the branch's coverage is inferior.


Related Reading


More Insights






Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

 
Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.
 

Video