Testing and deployment

After doing a few spikes it’s time for some actual work on my task based Silverlight application. In my previous post I indicated that I wanted to work in a ATDD and TDD fashion. The reason that I wanted to do this has to do with the automated feedback you constantly g et when implementing your code and you eventually have a big set of automated tests which help prevent regression problems.

ATTD & TDD

If you know TDD then the phrase ‘Red, Green, Refactor’ will be very familiar to you. If you work with ATDD then you will have a very similar flow. I tried to indicate in the following picture how I see ATDD and TDD work together:

So you start by writing a failing acceptance test (or multiple if this is handled by a product owner together with a tester). Since I will be doing this on my own, I will write those test myself. I will use SpecFlow to write the acceptance tests since it integrates nicely in Visual Studio and in MSTest. Next I’ll create the code to make the acceptance tests run the actual tests. The failing acceptance test will describe the expected behavior of the piece of functionality I will be adding. Next I’ll implement the functionality in the familiar TDD fashion, so using the ‘Red, Green, Refactor’ principle. When eventually the acceptance test(s) succeed, I’ve got some new functionality in my application for which I know that it works, because I started out by describing the functionality using SpecFlow and those tests now pass. The beauty of this is that the automated tests also become a great way of preventing regression problems because every feature you added is also covered by automated tests.

Continuous integration

I’ve got a test TFS server running, so I will be using this to run a CI build on every check-in. This build will compile all code, run the unit tests and I also want it to run all the acceptance tests. To be able to do this, I will have to make sure that all tests run very quickly. Whenever the tests take several minutes or longer to execute, it will be a show stopper, since history have proved that when tests take too long, they will get skipped and you loose the advantage that the tests give you. To keep the tests fast, they will have to run completely in-memory. I’ll get into the decisions that I make related to that in a future blog post.

Continuous deployment

With all the testing in place, there is the possibility to try out continuous deployment. Lately there is a lot talk about this and I really see advantages of it. To get this working, one of the mayor prerequisites is to have a good automated testing strategy. I described some of the testing behavior already, which is the unit tests and the acceptance tests. Next to that I also want to add automated UI tests to test the behavior of the actual user interface. These tests will not test any business rules or logic, since those are tested using the acceptance tests. Last category of tests which are relevant would be integration and performance tests. These kind of test could be the integration with a database, but also with other linked systems. I’ll probably will not use these kind of tests a lot (only for the integration between all the layers of the application, including the actual database), but whenever you’re facing a more distributed environment, including links to other systems, those links should also be tested using the automated integration and performance tests.

The following picture gives an idea of how I would like to have the automated deployment working.

So the automated build consists of several steps. Whenever the solutions compiles and all unit tests pass, it will be deployment to the development server. When also all the acceptance tests succeed, I want it to be deployed to the test environment. Then the automated UI tests will be run against the test environment. If they also pass, then the solution will be deployed to the acceptance environment, where the performance and integration tests will be run. Then the application should be ready for deployment to a production environment.

For my application, the deployment to the different environment will not be that different, since I don’t have several different servers to deploy the application to. However, in a enterprise scenario, there will be a lot more challenges to get your deployment completely automated.

Thread safe DateTime for unit testing

We’ve been using the construction of a wrapper class around the DateTime for unit testing which Ayende described here.

Today one of my colleagues discovered that there is a possibility to run multiple unittests in parallel. How to do that is described here. However, the approach we used with the SystemDateTime class no longer worked, since it uses a static field which caused problem with two tests running at the same time. I tweaked it a bit to fix this, so I wanted to share the code we use for it.

/// <summary>
/// Facade for the System.DateTime class that makes it possible to mock the current time.
/// </summary>
public static class SystemDateTime
{
    [ThreadStatic]
    private static Func<DateTime> now;

    [ThreadStatic]
    private static Func<DateTime> today;

    /// <summary>
    /// Returns the current time as a DateTime object.
    /// </summary>
    public static Func<DateTime> Now
    {
        get { return now ?? (now = () => DateTime.Now.Truncate()); }
        set { now = value; }
    }

    /// <summary>
    /// Gets the current date.
    /// </summary>
    public static Func<DateTime> Today
    {
        get { return today ?? (today = () => Now().Date); }
        set { today = value; }
    }
}

Hope it helps some of you.

Custom C# formatter for Selenium

In my previous post I mentioned that I had been working on a C# formatter for selenium which uses MSTest as test framework and also FluentAssertions to make the tests more readable. After a few hours last Saturday night I had something working which looks pretty ok for now. I guess I’m not the only one wanting to generate this code, so I wanted to share what I’ve got.

I started off by taking the source of the C# formatter which is installed together with the selenium IDE. Note that there is a bug (maybe more, but I ran into this one ) in the source, which results in a lack of options to set in the IDE. The problem is regarding the configForm setting, which is done in the options. This is pretty strange if you look at it, because that configForm is used to generate the UI for the options (which options are available can be found here). The configForm should be set outside the options. I tried to modify the source, but that’s not possible, it’s read-only. If you still want to use the NUnit formatting as done in the formatter, you can add another formatter and add the modified source. You can download the modified source here.

Now to the formatter I wrote. I wanted a few things:

• MSTest as test framework
• use FluentAssertions for readability
• Start the selenium server when starting test run (and close it when tests finished)

There is a description here on how to start creating a custom formatter. However, I was lazy, so I copied the source from the existing C# formatter and modified it to accommodate my needs. First of all I fixed the problem with the options, which I described above.
Next I changed the assertion methods to use the FluentAssertions syntax.
I updated the generated header and footer, since I don’t want all the initialization of a Selenium instance etc. to be done on every TestInitialize since it takes several seconds. I changed it to be done on a central location.

The biggest change I made was add formatting for a test suite, which is not available in the default C# formatter. In the code for the test suite, I added methods for AssemblyInitialize and AssemblyCleanup. In these method, I will start and tear down the selenium server process and the selenium instance. This uses a static selenium instance, so this will cause problems if you want to use it in a multi-threaded environment, but for now, it matches my needs. The complete source of the formatter can be downloaded here.

I hope some of you find this helpful also. If you have some good suggestions for the formatter, then please let me know.

Edit 22-02-2011: The links to the sources were incorrect. I updated the links now, so you can download the files from my skydrive.

Comparison of Web UI Testing toolkits

I wanted to started using web UI tests because the current project I’m working on I lacking this and for my personal web shop application I also need it. So I had to make a choice for a testing framework. I already knew of Coded UI Tests in Visual Studio 2010 and Selenium. Another colleague at my current project also mentioned Watin, so I decided to do a quick test of the three and see if it matches my expectations.

Coded UI Tests in Visual Studio

This was my first choice, since it’s integrated in Visual Studio which makes it easier when testing my deployed applications from a build server. So I started by creating the test in de Visual Studio IDE and I clicked some pages. Then added a few assertions and done. Generated the coded test and I looked at the coded that was generated. And that what a bit of a shock . It was very unreadable code, which actually shouldn’t matter since it’s generated code, but when you would like to tweak or edit it a bit (like I want) it’s not very nice to do that. I wanted to edit the tests to be able to use some parameters and stuff to make the tests more robust.
Now there is also the option of created tests using the Test Manager in Visual Studio 2010, but I haven’t tried this. That’s something I still want to do, to see if this makes it easier or better. For now, I’m a bit disappointed in using the coded UI tests. When I’ve got some time left, I’m going to check how the creation of coded UI tests is when using the Visual Studio 2010 Test Manager

Selenium

Selenium is a tool which I’ve heard of several times, so I wanted to give it a try now myself. First you’ll have to install some stuff. I installed the following:

• Selenium IDE
• Selenium client drivers for .NET
• Selenium IDE batch converter add-on
• Selenium server (which requires the java runtime)

After installing all stuff (which is more work than I wanted ) I started creating my first test. Clicking the test and verifications is also pretty easy with the integration in Firefox. Then creating the code. There are a few formatters available, one of them being for c#, so I generated the code, but unfortunately it was based on NUnit and there was some stuff in it I didn’t like. But before I gave up, I looked a bit better in Selenium IDE and I saw that there is also the possibility to create your own formatter, so I decided to give that a try. It took some time with some mistakes that I made, but eventually I had a working formatter which created the C# code the way I wanted it. So I started the tests in visual studio (after starting the java application for the selenium server) and it worked .

Watin

Because my colleague mentioned this framework, I wanted to give it a try. Unfortunately there was no IDE or something to create the tests. Since I was already more convinced of both other frameworks, I didn’t want to spend more time on this one, since it was also more difficult to work with it in code only mode.

Comparison

So a small comparison

	Coded UI Tests	Selenium	Watin
Positive	Integrated in Visual Studio, so no installation required	Control over the generated code Code easy to modify / extend
Negative	Ugly generated code	IDE is a bit buggy Custom installation required	No IDE Difficult to work with

So eventually I chose selenium for now, since some other people on my current project are also trying it out and were positive about it. I’ll write another blog post on the custom formatter I’ve been working on. It generates code for the MSTest framework and makes use of the FluentAssertions to make the tests more readable.