Software Gardening: Harvesting (Continuous Delivery Techniques)

Posted by: Craig Berntson , on 1/1/2015, in Category Software Gardening
Views: 31700
Abstract: This article talks about applying Continuous Delivery techniques to deliver working software.

Anyone that has a vegetable garden knows the importance of harvesting the veggies at exactly the right time. If you pick a vegetable too soon, it won’t have good flavor. Pick it too late and it will be soft, mushy, and rotten. Just like the gardener, we need to harvest our software applications when they are at the height of ripeness. Too soon and the app will be full of bugs. Too late and we’ll have lots of feature creep or the business needs will have changed and the application will be full of rotting code.

How do you determine the right time to harvest or release your application? Applying Agile, you release often. In fact, every two weeks is very common. This schedule helps you know that the application is not overly ripe. But that doesn’t help you know that application is actually properly ripened and ready for release. What does help you know if the code is ready for release, is by applying Continuous Delivery techniques.

This article is published from the DotNetCurry .NET Magazine – A Free High Quality Digital Magazine for .NET professionals published once every two months. Subscribe to this eMagazine for Free and get access to hundreds of free .NET tutorials from experts

Continuous Delivery uses what is called the “deployment pipeline”. Code goes in one side and fully tested applications come out the other. The idea of the deployment pipeline is that the further to the left you find a bug, the less costly it is to fix. This is commonly called “fail fast”.


Figure 1. The Deployment Pipeline

The other idea of the Deployment Pipeline is that code progresses in an automated fashion from left to right. This means that you automate deployments and automate testing, as much as possible. In fact, you can automate almost all the testing including unit, regression, integration, capacity, acceptance, functional, system, stress, performance, and deployment. Manual testing includes things such as exploratory, usability, showcases, look and feel, and worst-case testing.

You can see that Harvesting is a large topic. I only have space to discuss one aspect, so let’s talk about the first step, Continuous Integration. It’s here that your code is integrated with code from your co-workers and then unit tested to ensure the minimal functionality is still, well, functional. Ideally, the integration happens with each check-in to version control and takes five minutes or less to complete.


Figure 2. The Continuous Integration process

Using Continuous Integration (CI), each developer builds and tests their code locally. When the functionality is complete and their tests pass, updated code that has been written by other team members is brought locally and merged. This is the integration step. Tests are then run again locally to ensure code still passes. This is important. Once you get the unit tests to pass, check in the code. Ideally, this happens several times a day. Hence, “continuous integration”. Once the code passes this step, you check in your code.

On the other side of the version control system is a server that watches for code check-ins. When one occurs, it grabs the code and compiles it. If the compilation is successful, unit tests are run against the code. This unit testing step is critical as it identifies areas where code from one developer has side effects on code from another developer. The code doesn’t integrate. If you don’t have unit tests, you don’t have Continuous Integration. You only have an automated build.

Some systems, such as TFS, use a “gated check-in”. When you check in the code, it’s put in isolation and unit tests are again run. Only if those tests pass, is the code allowed into the version control repository. In this column, I will show you how to setup Team City for your CI server. Because Team City does not have its own source repository, it cannot force code to pass tests before check-in. However, Team City does support a “personal build” that you can run before check in. Note I will not show the personal build, but it is fully documented on the Team City web site.

The outputs of the CI build and test process are typically assemblies and other files. These are called “artifacts” in CI parlance. Hopefully your tests passed so you can get these artifacts.

The final step is reporting back to the team on the outcome of the integration. This is typically done through a web site.

The CI server software is a program designed to make this process easier. There are many CI servers that you can choose from including TFS, Go, Jenkins, TeamCity, and others. I will use Team City for my examples because I believe Team City is best of breed, even surpassing TFS. Why? TFS is very process heavy and you’re pretty much tied to how Microsoft thinks things should be done. Team City allows you to pick best of breed for version control, bug tracking, project management, deployment, etc. Depending on which other tools you choose, Team City or the other tool may have built-in integration points with them. Also, if your team uses languages and tools other than .Net, Team City may work out to be a much less expensive option. The cost is free up to 20 configurations (I’ll explain that in a bit), then its $2000 US. Really a small price to pay for a great tool. You may also be thinking that Jenkins, which is always free, is cheaper. Yes, initially it is, but you have to deal with lots more XML files and do more manual configuration, making it more costly in the end.


Before getting into Team City, we need to talk about MSBuild. Every time you build in Visual Studio, an MSBuild file is used. In fact, .csproj and .vbproj files are really MSBuild files. We’ll use a special MSBuild file for Team City.


A bit of explanation is needed. Targets are the sections that do the work. Think of them as procedures. There is one target, ReleaseBuild, in this project. A PropertyGroup is basically a property bag. You can see our solution name is TeamCity. In my example, you will save this file as Build.proj in the same folder as the solution.  

You can open a Visual Studio command prompt, navigate to the solution folder and type MSBuild Build.proj and see the output in the console window. Yes, MSBuild is a command line tool. It originally shipped as part of the .Net Framework but as of Visual Studio 2013 it has been broken out into its own package called Microsoft Build Tools. It gets installed with Visual Studio, but will need to be installed separately on the build server. Never install Visual Studio on your build server.  

Setting up Team City

I will use the term “build server” when referring to the actual server that runs the CI software, and “CI Server” as the generic term for the CI server software. You can download Team City from the Jet Brains web site. You can connect Team City to use Active Directory or use its own internal security. You also need a database for Team City to store configuration information. It ships with a small database server, but it is recommended you use it only for evaluation purposes. I typically use SQL Server Express for production build servers.  

You will also need to install the version of the .Net Framework you are using, the Microsoft Build Tools, and NuGet. These are all free so you don’t need to worry about licensing costs.  

Team City uses a hierarchy of objects to manage your build. At the top is a project. You can think of this as a Visual Studio solution. You can nest one Team City project inside another if you wish, but I haven’t had a need to do this.  


Figure 3. The Team City object hierarchy  

Next is the Configuration. This is where the work is done. For example, everything needed to do an Integration Build is a Configuration. If you do a nightly build to run additional tests, you have another Configuration. If you have Team City deploy to your QA server, that’s another Configuration. Remember, the free version of Team City supports up to 20 Configurations.  

Inside a Configuration you have several types of objects such as Build Steps (we’ll see those in a moment), Triggers that determine when the build runs, Dependencies (one Configuration can depend on another), etc. We’ll look at some of these objects as we setup an Integration build.  

Creating the Team City project

Once you have Team City and the other tools installed, navigate to the Team City website on your server. In the upper right-hand corner, click “Administration” then “Create Project”. I usually give the Team City project the name of the application I’m working on and then accept the default value for the Project ID. In my example, I named the Team City project DNC even though my Visual Studio solution is named TeamCity.sln.  


Figure 4. Team City project settings  

Looking at the left-hand side of the TeamCity configuration, you’ll now see the Project Settings menu. This menu is context sensitive and will change based on where we are in our build configuration.  

Connecting to version control

The first step is setup the connection to version control. Click on VCS Roots then Create VCS root. I’ve found it’s generally easier to select the type of VCS. You’ll need to supply the URL to the VCS source, a username, and password. Depending on your version control system, you may need to provide additional configuration data.  


Figure 5.: Configuring the VCS Root  

Once you have finished configuring the VCS Root, click Create. You will see a screen listing all VCS Roots for this project. The next thing is to configure the build.  

Configuring the build

Click General Settings on the Project Settings menu then click Create build configuration. You will need to enter the name of the build. As this is the integration build, I selected “100 – Integration Build”. By numbering the different builds, they will be displayed in the same order that they run. So, if the next build is to deploy to QA, you would name that build “200 – Deploy to QA”. Once you’re named this build, click Create.  

You need to tell this build to use the VCS Root you created earlier. Select it from the dropdown and click Attach. Team City now examines the files in version control and suggests different methods to do the build. I prefer to do this manually. Click the link that says “configure build steps manually” then on the next page, select NuGet Installer as the build runner type.  


Figure 6: Restoring NuGet files  

It’s necessary to restore NuGet packages because they shouldn’t be checked into version control. It will be impossible to build without these files. In Figure 6, you can see that I named this step NuGet Restore, I selected the version of NuGet to use and entered the path to TeamCity.sln, the Visual Studio solution. Note also that the left context menu has changed to Build Configuration Settings. Save this step then Add a build step. This next step will be the actual build so select MSBuild as the build runner type  


Figure 7: Settings for the Compile build step  

Name the step, point to Build.proj for doing the build, and select the proper MSBuild version. Then click Save. We have one more build step to add. This one will run unit tests. Select NUnit for the build runner type.  


Figure 8: Configuring unit tests  

As part of the unit test configuration, you need to enter the path to the unit test .dll file. This path is relative to the root directory of the Visual Studio solution file. Click Save. Team City has pre-build runners for NUnit and MSTest, but you can hook up any unit testing framework that has its own console runner.  

Setting the version number

Before we go too much further, let’s step back just a bit and add some information about the build itself. Click General Settings. One of the features of Team City is the ability to number the build. Think about this for a moment. If you have five developers, each will have a different build number on their local machine. How will Team City know which version number is correct? Well, because the most recent check-in wins, it will be the more recent version of AssemblyInfo.cs. Let’s setup TeamCity to handle the build number. This is done in two steps.  


Figure 9: Setting the build number and artifacts  

The first step is the build number format. I’ve used 1.0.0.%build.counter%. Team City will increment the build counter with each build. So, the first build will be, the second and so on. When you’re ready to work on version 2 of your application, just set the build number to 2.0.0.%build.number% and Reset the build counter.  

Before moving to step two, we need to tell Team City to keep the generated artifacts. In the Artifacts paths box, enter ** => This tells Team City to take all the files that it checked out and those it created in the build and put them in a zip file with a file name, the same as the build number.  

Now Click Save.  

The second step is to tell TeamCity to update AssemblyInfo.cs with the new build number. On the Build Configuration Settings menu, click Build Features then Add build feature. And select Assembly Info Patcher. Team City will automatically select the proper build number format. Click Save.  

Tagging version control

There’s one more thing we need to configure on this build. It would be nice if we had Team City tag version control with the build number. This way, we always know what version of source control files went into a specific build number. Again, click Build Features and Add build feature. This time, pick VCS Labeling and then select the VCS Root for this project.  


Figure 10: Tagging version control with the build number  

Once you configured this feature, click Save.  

Scheduling the build

There’s one last thing we need to setup. We need to schedule this build to run whenever Team City detects a change in version control. To do this, click Triggers then Add new trigger and select VCS Trigger from the drop down. Note you can also manually run the build or schedule the build to run on a specific day or time.  

If you have a busy build server, Team City will queue up the builds and run them in order or you can purchase additional build agents and install them on another server (The Team City Enterprise license gives you three build runner licenses.) Using this scenario, one server hosts Team City and handles farming out the builds to other servers.  

Finally, the only thing left is to run the build. There are a couple of ways to do this. You can modify a file in the Visual Studio project and check it in or you can manually trigger the build. Return to the Team City home page, expand the DNC project and click the Run button for the build configuration we just created.  

Team City will go out to version control, get the source files, copy them locally, then run each build step in order. If all went well, you’ll see the build result will be green. If not, it will be red, and you’ll have to drill into the build results to determine what went wrong.  

Developers can get results in three different ways. First is a system tray application that simply shows a red/green condition for all builds you are monitoring. The second is a Visual Studio plug-in. Third is by going to the Team City site of your build server and looking at the build status. If a build fails for any reason, this is where you go to get the results. You can drill-down to the details.  


Figure 11: Results of the build on the Team City home page  

One final note. You can also hook up additional tools like StyleCop or FxCop, but I don’t recommend these for an Integration build. Why? FxCop is a bit outdated and doesn’t support .Net 4.x very well. Additionally, Team City includes its own inspections module that does the same basic job (this is the same inspections that you can run from Resharper). The second reason for not running FxCop is the same reason I wouldn’t run StyleCop, SQLCop, etc as part of the integration build…they take time. Ideally, an integration build should be run in five minutes or less and running these additional pieces adds time to the build.  

Instead, create an Inspections Build and run it at night. This is when you run inspections such as StyleCopy, TeamCity Inspections, SQLCop, or any other review tool you wish. In other words, inspections are not part of Continuous Integration and out of scope for this instance of Software Gardening. I’ll come back and address them in the future.  

So, how important is Continuous Integration? Well, the Agile Manifesto says you are to deliver working software. If you are trying to deliver every two weeks, there is no way you can deliver working software without using CI. It would be near impossible to determine everything is working correctly without it.  

Hopefully you understand what CI can do for you and how easy it is to setup in Team City. After all, it is only by using CI as the first step in your deployment pipeline that you can ensure your software stays lush, green, and vibrant.  

About Software Gardening - Comparing software development to constructing a building says that software is solid and difficult to change. Instead, we should compare software development to gardening as a garden changes all the time. Software Gardening embraces practices and tools that help you create the best possible garden for your software, allowing it to grow and change with less effort. Learn more in the Software Gardening article series.

This article has been editorially reviewed by Suprotim Agarwal.

Absolutely Awesome Book on C# and .NET

C# and .NET have been around for a very long time, but their constant growth means there’s always more to learn.

We at DotNetCurry are very excited to announce The Absolutely Awesome Book on C# and .NET. This is a 500 pages concise technical eBook available in PDF, ePub (iPad), and Mobi (Kindle).

Organized around concepts, this Book aims to provide a concise, yet solid foundation in C# and .NET, covering C# 6.0, C# 7.0 and .NET Core, with chapters on the latest .NET Core 3.0, .NET Standard and C# 8.0 (final release) too. Use these concepts to deepen your existing knowledge of C# and .NET, to have a solid grasp of the latest in C# and .NET OR to crack your next .NET Interview.

Click here to Explore the Table of Contents or Download Sample Chapters!

What Others Are Reading!
Was this article worth reading? Share it with fellow developers too. Thanks!
Share on LinkedIn
Share on Google+

Craig Berntson is a software architect specializing in breaking up the monolith and improving developer teams and processes. He is the author of two books on software development and has been a speaker at conferences across North America and Europe. He received the Microsoft MVP award twenty-two straight years. He lives in Salt Lake City, Utah.

Page copy protected against web site content infringement 	by Copyscape

Feedback - Leave us some adulation, criticism and everything in between!
Comment posted by Jesse on Monday, January 5, 2015 11:50 PM
Excellent CI concept overview, but I have three critical notes:

#1 Telling .NET CI novices “Never install Visual Studio on your build server” with zero explanation of your *preference* is unreasonable in the extreme. Not having the VS build chain on the server is asking for trouble in many, many scenarios. The blogosphere is alight with very smart people complaining about how MS needs to make this process both easier and possible because these aren’t just edge cases where having VS is needed. The easiest way to get the .NET CI novice moving in the right direction is to tell them to install VS on the build server, analyze their dependencies, and maybe later suggest doing it over again without VS as a varsity-level project.

Assembling your own build chain is not trivial unless you’re building the barest of vanilla code, and ends up scaring most people away from CI altogether after their first few problems where the code builds fine on their developer machine but won’t build at all on their homemade CI toolchain. Did MS sucker you into using MSTest? Good luck getting that working without VS installed. If you need anything later than 2010 support it’s all but impossible. Working on a project type that’s using a non-standard runtime like VSTO? Need to build a VS-only project type like an installer? Microsoft Build Tools laughs in your face when you try that. Just install VS locally and don’t ask rookies to waste days or weeks building or rebuilding everything to accommodate your *preference*.

#2 FXCop works great on all .NET versions unless you stubbornly insist on running the 6 year old free version. If, however, you install VS on the build server (and fix the codeanalysispath and fxcopdir env vars if you’re upgrading), you can run e.g. \Microsoft Visual Studio 12.0\Team Tools\Static Analysis Tools\FxCop\fxcopcmd.exe against any .NET code and it works great.

Also, it doesn’t take much time to run FXCop--maybe the ancient free version is really slow, but fxcopcmd from the VS2013 folder is nice and speedy. On our longest full site builds FXCop accounts for <1% of build time and pays huge dividends with invaluable code quality feedback. Just let a few CA2000s slip into production in the wrong loop and see how everything comes to a mysterious grinding halt until you hand-debug it. Good luck!

#3 Your Team City bias is completely unwarranted at this advanced stage of CI tool awesomeness. At least you had the presence of mind to compare it seriously to Jenkins instead of only mentioning CruiseControl and TFS like in your book, but literally everything you described Team City doing can be done on Jenkins with similar UI effort and *zero* use of XML. I nearly stopped reading after you complained that "always free" Jenkins requires you to mess with a lot of XML, because in the *next paragraph* you displayed a giant MSBuild XML file and explained its innards in great detail, proving handily that your target audience is more than capable of dealing with it--should it even come up (spoiler: it won't).

Jenkins has >1000 plugins for talking to a huge array of tools and services and scales out really well with no use for a database. I noticed that TeamCity has serious plugin support since the last time I looked and appears to have about 50 plugins--14 of which they'll actually let you see the source code to! Based on how closed off it is, the only thing Team City seems specifically optimized for is generating purchase orders to JetBrains for support and features.

After using Hudson for years to do .NET CI at several companies, I read your book when it came out because the Jenkins/Hudson fork had just occurred. Because of you I ended up seriously reviewing Team City, Cruise Control, and TFS. I loved your book, but I ended up hating TC and CC since both had hellish never-ending wizard-driven UIs and extremely limited integration with other tools and systems, comparatively speaking. I came to the same conclusion that you did about TFS (their horrible UI in 2010 didn’t help their case, either). I ended up following the Hudson community to Jenkins and I'm proud to say I've left a trail of happy and capable CI users in my wake at various companies where I've setup operations.

As a Jenkins user, I’ve been able to look at the source code for both the core product and the numerous plugins I rely on for fetching, building, testing, and analyzing .NET and JS code (we use a lot of stuff--FxCop, Checkmarx, dotCover, NCover, MSTest, NUnit, JSHint, Jasmine, blanket, etc.). Even though I'm not a professional Java developer I've been able to troubleshoot and help fix my own product/plugin bugs over the years due the way core code, issues, and plugins are managed by the community, whereas I have issues and feature requests sitting on the JetBrains tracker for TC, R#, and dotCover that are years old with zero responses.

I really think it’s time you considered more acceptance criteria than whatever XML problem made you hate Jenkins/Hudson however long ago and give Jenkins a more serious look for .NET CI. It takes about 5 minutes and 20 clicks to install and integrate with Active Directory. Building solutions from the SCM of your choice and testing, covering, and analyzing your .NET and JS code isn’t much farther in the distance. There are also several great plugins that help you generating workflows and automate deployments for continuous delivery, too.
Comment posted by Craig Berntson on Wednesday, January 7, 2015 5:45 PM
1. - Can you provide a reason you need VS on the build server because I can't think of one. Putting VS on the build server is bad because a)You have to pay for the VS license b) You should not have a dependency on VS. You should have a dependency on MSBuild and the .Net Framework. Further, go talk to CI "experts" and they'll give the same advice, ie, you should NOT install your IDE on the build server.
- No, MS had no input on the article. And it's dead easy to get MSTest working on Team City as it has a test runner for MSTest. In my production work, I actually use NUnit.
2. FXCop was great in its day but it hasn't been updated for sometime. That means it doesn't support new language features. Additionally, the code analysis built into Team City does most of what you need.
3. It's good to hear that you've been successful with Jenkins, but truthfully, you're the only person I've heard from that has had enough success with .Net and Jenkins to recommend it and I've talked to a lot of people that have used it.

Finally, any article written by anyone can have people disagree with the viewpoints provided. I appreciate your differing views.