Database concurrency - simplified

Just the other day I was asked why we did not use NHibernate since using an ORM is the only way to ensure concurrent writes to the database. Well, after a while I understood that we were not actually talking about concurrency per se, but rather how to make sure that we don’t update the database using stale data.
In other words making sure that we don’t overwrite somebody else’s data.
Since I strongly disapprove of the very concept of ORM’s, I thought this would be a perfect example to explain how simple this really is.

Basically we have three ways to approach this problem.

• No concurrency check - Last write wins which might be fine for quite a number of cases.
• Using a version/timestamp column - Always include a “version” column where the “new” values must match the value in the database.
• Using the old values from the modified columns.

CREATE TABLE Persons
(
    PersonId int,
    FirstName varchar(50),
    LastName varchar(50),
    Version int
)

Update without concurrency check

UPDATE Persons
SET
    FirstName = "NewName"
WHERE 
    PersonId = 1      

Without the concurrency check it will always be “last write wins”

Update using a version column

UPDATE Persons
SET
    FirstName = "NewName"
WHERE 
    PersonId = 1 AND
    Version = 100     

The update will only succeed if the version number matches the given number.

It should be fairly easy to ensure that the Version column is always presents in the update statement using an update trigger on the table. Besides validating the presence of the version column in a trigger ensures that this will work regardless of the tool used to access the database.

Update using modified columns

UPDATE Persons
SET 
    FirstName = "NewName"
WHERE 
    PersonId = 1 AND
    FirstName = "OldName"   

This approach allows concurrent updates to the same row as long as the updates does not update the same columns. The disadvantage of this model is that it requires the client to keep track of the old column values and this tends to add complexity to the data access layer. There is also an indexing problem as we have no guarantee that we have an index that covers the columns used in the where clause of the update statement.

Conclusion

In terms of simplicity and something that is easy to understand and implement, I recommend going with a version column. This does not have to be an integer as shown here, but can also be a datetime/timestamp or the like. And believe me, you don't need NHibernate or any other crap ORM tool to implement concurrent writes to you database. Keep things simple and you will live happily ever after.

SQL files using Resharper

This post is going to show how to create a file template in Resharper that allows you to quickly create new SQL files (files ending with *.sql) in Visual Studio.

By default the quick access list contains the following items.

• Class
• Interface
• Struct
• Enum

Goto Resharper -> Template Explorer

Choose New Template

Save the template and it will appear in the template explorer.

Drag the SQL template into the quicklist and you are good to go.

The quick access list will now contain the following items.

• Class
• Interface
• Struct
• Enum
• SQL

Enjoy!!

RunOnce

Sometimes I find my self in a situation where I need to make sure that a piece of code is only executed once.
This is typically some kind of initialization code that must be executed before code that relies on the initialization can execute.

I am pretty sure that we all have seen code that looks similar to this

public class Foo
{
    private bool isInitialized;

    public void DoSomething()
    {
        Initialize();

        // Continue
    }

    private void Initialize()
    {
        if (isInitialized)
        {
            return;
        }

        // Do some initialization code.

        isInitialized = true;
    }
}

Adding thread safety introduces more complexity. The following example uses Double-checked locking.

public class Foo
{
    private bool isInitialized;
    private readonly object lockObject = new object(); 

    public void DoSomething()
    {
        Initialize();

        // Continue
    }

    private void Initialize()
    {
        if (isInitialized)
        {
            return;
        }
        lock (lockObject)
        {
            if (isInitialized)
            {
                return;
            }

            // Do some initialization code.

            isInitialized = true;
        }            
    }
}

The idea here is to wrap this plumbing code into a class that sort of acts as the Lazy<T> class besides the fact that we will not be returning a value.

/// <summary>
/// A thread safe class that ensures that a given 
/// <see cref="Action"/> is only executed once.
/// </summary>
public class RunOnce
{
    private readonly Action action;
    private int hasExecuted;

    /// <summary>
    /// Initializes a new instance of the <see cref="RunOnce"/> class.
    /// </summary>
    /// <param name="action">The <see cref="Action"/> delegate to be executed once.</param>
    public RunOnce(Action action)
    {
        this.action = action;
    }

    /// <summary>
    /// Executes the <see cref="Action"/> only if not already executed.
    /// </summary>
    public void Run()
    {
        if (Interlocked.Exchange(ref hasExecuted, 1) == 0)
        {
            action();
        }
    }       
}    

Once interesting aspect of this class is that it uses no lock to provide thread safety.
The following article explains this in detail.

Now that we have our RunOnce class we can significantly simplify our code.

public class Foo
{
    private RunOnce initializeOnce;

    public Foo()
    {
        initializeOnce = new RunOnce(Initialize);
    }

    public void DoSomething()
    {
        initializeOnce.Run();

        // Continue
    }

    private void Initialize()
    {           
        // Do some initialization code.            
    }
}  

Happy initialization!

Linq to Expression Trees

If you are working with expressions trees in .Net, chances are that you have come across the need for inspecting the nodes of the expression tree.

The idea here is to flatten the expression tree into an IEnumerable<Expression>.

/// <summary>
/// Extends the <see cref="Expression"/> class.
/// </summary>
public static class ExpressionExtensions
{
    /// <summary>
    /// Flattens the <paramref name="expression"/> into an <see cref="IEnumerable{T}"/>.
    /// </summary>
    /// <param name="expression">The target <see cref="Expression"/>.</param>
    /// <returns>The <see cref="Expression"/> represented as a list of sub expressions.</returns>
    public static IEnumerable<Expression> AsEnumerable(this Expression expression)
    {
        var flattener = new ExpressionTreeFlattener();
        return flattener.Flatten(expression);
    }

    private class ExpressionTreeFlattener : ExpressionVisitor
    {
        private readonly ICollection<Expression> nodes = new Collection<Expression>();

        public IEnumerable<Expression> Flatten(Expression expression)
        {
            Visit(expression);
            return nodes;
        }

        public override Expression Visit(Expression node)
        {
            nodes.Add(node);
            return base.Visit(node);
        }
    }
}

Now we can take any expression tree and convert it into a list of expressions and just apply Linq to Objects.

Expression<Func<int>> expression = () => 42;
var result = expression.AsEnumerable().FirstOrDefault(e => e is ConstantExpression);
Console.WriteLine(result);

Output:

42  (ConstantExpression)

Happy Linqing!

Async/Await and code coverage

First of all I should say that I think good code coverage is a good thing. What "good coverage" means is probably something that can be discussed for days, but I like mine in the high 90's. I simply don't like shipping code that has never been executed. Call me paranoid, but that is how I feel about it :)

I have been playing around with the async/await features in C#5 for a while now and finally I was able to put it to good use in a new project that I am working on. Yes, that's right, we were actually given permission to use .Net 4.5 (Damn you Windows XP).

Okay, enough smalltalk (no pun intended) , lets dig into some code.

public interface IFoo
{
    Task<int> ExecuteAsync();
}

public class Foo : IFoo
{
    public async Task<int> ExecuteAsync()
    {
        await Task.Delay(1000);
        return 42;
    }
}     

Simple enough, we mimic the behavior of a long running method by awaiting a delay of 1000 milliseconds.

Let's write a test that executes the method.

[Fact]
public async Task LongRunningTest()
{
    var foo = new Foo();
    var result = await foo.ExecuteAsync();
    Assert.Equal(42, result);
}

The code coverage for the LongRunningTest reports 100% as expected.

Note: We now are essentially looking at the code coverage for the test method, but remember that the ExecuteAsync method could just as easily be used deep down in some business logic where it would make more sense to actually mock the IFoo interface.

Now we try the same thing using a mock

[Fact]
public async Task MockTest()
{
    var fooMock = new Mock<IFoo>();
    fooMock.Setup(m => m.ExecuteAsync()).ReturnsAsync(42);
    var result = await fooMock.Object.ExecuteAsync();
    Assert.Equal(42, result);
}

Running the MockTest with code coverage shows 85% and looking at the results it reports that some mysterious MoveNext method is not covered. So where does this method come from? We most certainly did not write it. As it turns out, the compiler did when it encountered the await keyword.

To get a better understanding of what is going on, we need to take a look at the actual code that gets compiled.

Looking at the LongRunningTest method in Reflector (or ILSpy) shows that in addition to the method itself, there is a corresponding state machine that takes care of the gory details with regards to asynchronous execution.

[CompilerGenerated]
private struct <LongRunningTest>d__0 : IAsyncStateMachine
{
    public int <>1__state;
    public Tests <>4__this;
    public AsyncTaskMethodBuilder <>t__builder;
    private object <>t__stack;
    private TaskAwaiter<int> <>u__$awaiter3;
    public Foo <foo>5__1;
    public int <result>5__2;

    private void MoveNext()
    {
        try
        {
            TaskAwaiter<int> awaiter;
            bool flag = true;
            switch (this.<>1__state)
            {
                case -3:
                    goto Label_00C3;

                case 0:
                    break;

                default:
                    this.<foo>5__1 = new Foo();
                    awaiter = this.<foo>5__1.ExecuteAsync().GetAwaiter();
                    if (awaiter.IsCompleted)
                    {
                        goto Label_0082;
                    }
                    this.<>1__state = 0;
                    this.<>u__$awaiter3 = awaiter;
                    this.<>t__builder.AwaitUnsafeOnCompleted<TaskAwaiter<int>, Tests.<LongRunningTest>d__0>(ref awaiter, ref this);
                    flag = false;
                    return;
            }
            awaiter = this.<>u__$awaiter3;
            this.<>u__$awaiter3 = new TaskAwaiter<int>();
            this.<>1__state = -1;
        Label_0082:
            int introduced6 = awaiter.GetResult();
            awaiter = new TaskAwaiter<int>();
            int num2 = introduced6;
            this.<result>5__2 = num2;
            Assert.Equal<int>(0x2a, this.<result>5__2);
        }
        catch (Exception exception)
        {
            this.<>1__state = -2;
            this.<>t__builder.SetException(exception);
            return;
        }
    Label_00C3:
        this.<>1__state = -2;
        this.<>t__builder.SetResult();
    }

    [DebuggerHidden]
    private void SetStateMachine(IAsyncStateMachine param0)
    {
        this.<>t__builder.SetStateMachine(param0);
    }
}

Without going through the code in detail (very few people can, Jon Skeet definitely being one of them), we can at least see that there is a MoveNext method and that is exactly the method that did not get covered in the MockTest test method. The question is why did it get invoked in the LongRunningTest and not invoked in the MockTest?

The reason that is did not execute is that the asynchronous method was already completed at the time it was invoked. In the Foo class we simulated a delay that caused what is called continuation and in turn invocation of the state machine's MoveNext method.

If we take a look at the ReturnsAsync method from the Moq library, we see the following:

public static IReturnsResult<TMock> ReturnsAsync<TMock, TResult>(this IReturns<TMock, Task<TResult>> mock, TResult value) where TMock : class
{
  TaskCompletionSource<TResult> completionSource = new TaskCompletionSource<TResult>();
  completionSource.SetResult(value);
  return mock.Returns(completionSource.Task);
}    

The ReturnsAsync method uses the TaskCompletionSource<TResult> class and the SetResult method transitions the state of the underlying task to completed. We could get around this issue by creating an extension method that delays the asynchronous method to cause continuation, but that would definitely hurt the duration of test execution.

What we need is some way to create a task that is not completed even if it actually is.

The await keyword expects something awaitable and what does it take for a class to be awaitable?

The following example shows the simplest thing that can be awaited and return a value.

public class CustomAwaitable
{
    public CustomAwaiter GetAwaiter()
    {
        Console.WriteLine("GetAwaiter");
        return new CustomAwaiter();
    }
}

public class CustomAwaiter : INotifyCompletion
{
    public int GetResult()
    {
        Console.WriteLine("GetResult");
        return 42;
    }

    public bool IsCompleted
    {
        get
        {
            Console.WriteLine("IsCompleted (Get)");
            return false;
        }
    }

    public void OnCompleted(Action continuation)
    {
        Console.WriteLine("OnCompleted");
        continuation();
    }
}

There is no reason that this could not have been implemented in the same class, but since this sort of resembles the relationship between the IEnumerable<T> and IEnumerator<T> interface, I found it easier to read if they were split into to classes.

The interesting methods here are the IsCompleted method and the OnCompleted method.

The IsCompleted method is pretty self explanatory and simply indicates if the value is available via the GetResult method. The reason we return false here is that we want to force continuation.

The OnCompleted method is not so self explanatory as one might think that it gets executed when something is completed. What this method actually does is attaching a continuation if the form of a delegate to be invoked once this await operation is completed. This method is never executed unless the IsCompleted property returns false.

Lets create a test for this and take a look at the output.

[Fact]
public async Task CustomAwaiterTest()
{
    var awaitable = new CustomAwaitable();
    int result = await awaitable;
    Assert.Equal(42, result);
}  

The output from this test shows the methods involved.

GetAwaiter
IsCompleted (Get)
OnCompleted
GetResult

Now lets change the IsCompleted property to return true

public bool IsCompleted
{
    get
    {
        Console.WriteLine("IsCompleted (Get)");
        return true;
    }
}

Test output

GetAwaiter
IsCompleted (Get)
GetResult

As we can see the OnCompleted method is never executed as it is no need to attach a continuation.

With a basic understanding of how this works, we can move on to solve the actual problem. So we need to create an awaitable that is not completed even if it actually is completed. The solution is to create a decorator that wrappes an existing awaitable and always return false in the IsCompleted property to ensure continuation.

    /// <summary>
/// Represents an awaitable who's awaiter has not yet completed.
/// </summary>
/// <typeparam name="TResult">The result type.</typeparam>
public class IncompleteAwaitable<TResult>
{
    private TaskAwaiter<TResult> taskAwaiter;

    /// <summary>
    /// Initializes a new instance of the <see cref="IncompleteAwaitable{T}"/> class.
    /// </summary>
    /// <param name="task">The <see cref="Task{TResult}"/> wrapped by this <see cref="IncompleteAwaitable{T}"/>.</param>
    public IncompleteAwaitable(Task<TResult> task)
    {
        taskAwaiter = task.GetAwaiter();
    }

    ///// <summary>
    ///// Gets an awaiter used to await this <see cref="Task{TResult}"/>
    ///// </summary>
    ///// <returns></returns>
    public IncompleteAwaiter<TResult> GetAwaiter()
    {
        return new IncompleteAwaiter<TResult>(taskAwaiter);
    }
}

/// <summary>
/// An awaiter that has never completed regardsless of the 
/// state of the underlying <see cref="TaskAwaiter{TResult}"/>/>
/// </summary>
/// <typeparam name="TResult">The result type.</typeparam>
public class IncompleteAwaiter<TResult> : INotifyCompletion
{
    private readonly TaskAwaiter<TResult> taskAwaiter;

    /// <summary>
    /// Initializes a new instance of the <see cref="IncompleteAwaiter{TResult}"/> class.
    /// </summary>
    /// <param name="taskAwaiter">The underlying <see cref="TaskAwaiter{TResult}"/></param>
    public IncompleteAwaiter(TaskAwaiter<TResult> taskAwaiter)
    {
        this.taskAwaiter = taskAwaiter;
    }

    /// <summary>
    /// Attaches a continuation to the underlying <see cref="TaskAwaiter{TResult}"/>. 
    /// </summary>
    /// <param name="continuation">The continuation delegate to be attached.</param>
    public void OnCompleted(Action continuation)
    {          
        taskAwaiter.OnCompleted(continuation);
    }

    /// <summary>
    /// Gets a value indicating whether this awaiter is completed.
    /// </summary>
    /// <remarks>
    /// This property will always return false to ensure continuation.
    /// </remarks>
    public bool IsCompleted
    {
        get
        {
            return false;
        }
    }

    /// <summary>
    /// Gets the result from the underlying <see cref="TaskAwaiter{TResult}"/>.
    /// </summary>
    /// <returns>The completed <typeparamref name="TResult"/> value.</returns>
    public TResult GetResult()
    {
        return taskAwaiter.GetResult();
    }
}

The IncompleteAwaitable<TResult> simply takes an existing Task<TResult> and by using the GetAwaiter method we get the awaiter from the task and we pass that to the IncompleteAwaiter<TResult> class (decorator).

With this we can now force continuation even from a task that has completed.

[Fact]
public async Task IncompleteAwaitableTest()
{
    //This creates a task that has completed.
    var task = Task.FromResult(42);            
    var awaiter = new IncompleteAwaitable<int>(task);
    var result = await awaiter;
    Assert.Equal(42, result);
}

Looking at the code coverage for this test shows that all the code has executed including the MoveNext method from the state machine.

To make this a little easier to use we can create an extension method that allows us to create an incomplete task from en existing task.

/// <summary>
/// Extends the <see cref="Task{TResult}"/> class.
/// </summary>
public static class TaskExtensions
{        
    /// <summary>
    /// Creates an "incomplete" <see cref="Task{TResult}"/> from 
    /// the given <paramref name="task"/>. 
    /// </summary>
    /// <typeparam name="TResult">The result type returned from the task.</typeparam>
    /// <param name="task">The target <see cref="Task{TResult}"/>.</param>
    /// <returns>A new <see cref="Task{TResult}"/> where the <see cref="Task.IsCompleted"/> is set to false.</returns>
    public static Task<TResult> GetIncompleteTask<TResult>(this Task<TResult> task)
    {
        var incompleteAwaitable = new IncompleteAwaitable<TResult>(task);
        return Task<TResult>.Run(async () => await incompleteAwaitable);
    }
}

We can now create an incomplete task from an existing task.

[Fact]
public async Task IncompleteTaskTest()
{
    var task = Task.FromResult(42);
    var result = await task.GetIncompleteTask();
    Assert.Equal(42, result);
}

And code coverage for this test is still 100% as expected.

Using the Moq library we can create a mock like this.

[Fact]
public async Task IncompleteMockedTask()
{
    var mock = new Mock<IFoo>();
    mock.Setup(m => m.ExecuteAsync()).Returns(() => Task.FromResult(42).GetIncompleteTask());
    var result = await mock.Object.ExecuteAsync();
    Assert.Equal(42, result);
}  

Please note that it is important to use the Returns overload that takes a value factory. Otherwise, Moq will evaluate the task and hence prevent the continuation in the test.

We could simplify this even further by creating an extension method specially tailored for Moq.

/// <summary>
/// Extends the <see cref="IReturns{TMock,TResult}"/> interface.
/// </summary>
public static class MoqExtensions
{
    /// <summary>
    /// Specifies the return value from the method as an "incomplete" <see cref="Task{TResult}"/>.
    /// </summary>
    /// <typeparam name="TMock">The mocked type.</typeparam>
    /// <typeparam name="TResult">The mocked method result type.</typeparam>
    /// <param name="mock">The target <see cref="IReturns{TMock,TResult}"/> instance.</param>
    /// <param name="value">The value to be returned from the method.</param>
    /// <returns>An <see cref="IReturnsResult{TMock}"/> instance.</returns>
    public static IReturnsResult<TMock> ReturnsIncompleteAsync<TMock, TResult>(this IReturns<TMock, Task<TResult>> mock, TResult value) where TMock : class
    {
        var task = Task.FromResult(value);
        return mock.Returns(task.GetIncompleteTask);                   
    }
} 

We can now mock an asynchronous method and still enforce continuation and hence 100% coverage.

[Fact]
public async Task IncompleteMockedTaskUsingExtensionMethod()
{
    var mock = new Mock<IFoo>();
    mock.Setup(m => m.ExecuteAsync()).ReturnsIncompleteAsync(42);
    var result = await mock.Object.ExecuteAsync();
    Assert.Equal(42, result);
}

Happy testing!!

I Am My Code

I Am My Code

I really dislike the saying "You are not your code" even though its meaning migth not be as bad as it sounds. It sort of means that we should criticize the code and not the person who wrote it. Well, fair enough, we want to stay civilized a long as possible, right?

It's just that "You are not your code", to me, sounds like the words from a programmer that does not really care. "I'm just going to throw some code in there.. it does not really matter cause it's not who I am". That's SLOPPINESS of first degree and you all know it.

First of all, I love programming and like the other things that I love, such as my wife and children, they are a BIG part of me. If someone speaks of your children in a negative way, you're telling me that it does not get to you?

I think of my code in very much the same way. Its my legacy, something I have put a lot of thought and consideration into. It might not be perfect, but it's the result of passion and hard work.

If you're a programmer and don't have a true passion about what you do you should seriously consider another line of work.

The "Calculator Kata" challenge

The calculator kata made by Roy Osherove is an exercise in test driven development and goes like this:

• Try not to read ahead.
• Do one task at a time. The trick is to learn to work incrementally.
• Make sure you only test for correct inputs. there is no need to test for invalid inputs for this kata

Create a simple String calculator with a method int Add(string numbers)

1. The method can take 0, 1 or 2 numbers, and will return their sum (for an empty string it will return 0) for example “” or “1” or “1,2”
   Start with the simplest test case of an empty string and move to 1 and two numbers
   Remember to solve things as simply as possible so that you force yourself to write tests you did not think about
   Remember to refactor after each passing test
2. Allow the Add method to handle an unknown amount of numbers
3. Allow the Add method to handle new lines between numbers (instead of commas).
   the following input is ok: “1\n2,3” (will equal 6)
   the following input is NOT ok: “1,\n” (not need to prove it - just clarifying)
4. Support different delimiters
   to change a delimiter, the beginning of the string will contain a separate line that looks like this: “//[delimiter]\n[numbers…]” for example “//;\n1;2” should return three where the default delimiter is ‘;’ .
   the first line is optional. all existing scenarios should still be supported
5. Calling Add with a negative number will throw an exception “negatives not allowed” - and the negative that was passed.if there are multiple negatives, show all of them in the exception message
   stop here if you are a beginner. Continue if you can finish the steps so far in less than 30 minutes.
6. Numbers bigger than 1000 should be ignored, so adding 2 + 1001 = 2
7. Delimiters can be of any length with the following format: “//[delimiter]\n” for example: “//[]\n12*3” should return 6
8. Allow multiple delimiters like this: “//[delim1][delim2]\n” for example “//[][%]\n12%3” should return 6.
   make sure you can also handle multiple delimiters with length longer than one char

With nothing better to do this afternoon I decided to to give it a try. This is what I came up with in the first version.

The tests:

   [TestMethod]
    public void Add_EmptyString_ReturnsZero()
    {
        var result = CreateCalculator().Add(string.Empty);

        Assert.AreEqual(0, result);
    }

    [TestMethod]
    public void Add_SingleNumber_ReturnsNumber()
    {
        var result = CreateCalculator().Add("1");

        Assert.AreEqual(1, result);
    }

    [TestMethod]
    public void Add_TwoNumbers_ReturnsSum()
    {
        var result = CreateCalculator().Add("12,2");

        Assert.AreEqual(14, result);
    }

    [TestMethod]
    public void Add_MultipleNumbers_ReturnsSum()
    {
        var result = CreateCalculator().Add("1,2,3");

        Assert.AreEqual(6, result);
    }

    [TestMethod]
    public void Add_UsingNewLineDeliminator_ReturnsSum()
    {
        var result = CreateCalculator().Add("1\n2,3");

        Assert.AreEqual(6, result);
    }

    [TestMethod]
    public void Add_CustomDeliminator_ReturnsSum()
    {
        var result = CreateCalculator().Add("//;\n1;2");

        Assert.AreEqual(3, result);
    }

    [TestMethod, ExpectedException(typeof(ArgumentOutOfRangeException))]
    public void Add_NegativeValue_ThrowsException()
    {
        CreateCalculator().Add("-1");            
    }

    [TestMethod]
    public void Add_NumbersLargerThan1000_ReturnsSum()
    {
        var result = CreateCalculator().Add("2,1001");

        Assert.AreEqual(2, result);
    }

    [TestMethod]
    public void Add_LongDelimiter_ReturnsSum()
    {
        var result = CreateCalculator().Add("//[***]\n1***2***3");

        Assert.AreEqual(6, result);
    }

    [TestMethod]
    public void Add_MultipleDelimiters_ReturnsSum()
    {
        var result = CreateCalculator().Add("//[*][%]\n1*2%3");

        Assert.AreEqual(6, result);
    }

    [TestMethod]
    public void Add_MultipleLongDelimiters_ReturnsSum()
    {
        var result = CreateCalculator().Add("//[***][%%%]\n1***2%%%3");

        Assert.AreEqual(6, result);
    }

And the implementation looked something like this:

public interface ICalculator
{
    int Add(string input);
}

public class Calculator : ICalculator
{
    public int Add(string input)
    {            
        if (HasCustomDelimiter(input))
        {
            string delimiters = StripOffNumbers(input);
            input = StripOffDelimiterMetadata(input);                
            return AddNumbers(input, GetDelimiters(delimiters));
        }

        return AddNumbers(input, GetDelimiters(string.Empty));
    }

    private string StripOffNumbers(string input)
    {
        return input.Substring(0, input.IndexOf('\n')).Substring(2);
    }

    private int AddNumbers(string input, string[] delimiters)
    {
        if (input.Length == 0)
        {
            return 0;
        }

        return SplitNumbers(input, delimiters).Select(ParseNumber).Where(n => n < 1000).Sum();
    }

    private static int ParseNumber(string s)
    {
        int result = int.Parse(s);
        if (result < 0)
        {
            throw new ArgumentOutOfRangeException();
        }

        return result;
    }

    private string[] SplitNumbers(string input, string[] delimiters)
    {
        return input.Split(delimiters, StringSplitOptions.None);
    }

    private string StripOffDelimiterMetadata(string input)
    {
        if (HasCustomDelimiter(input))
        {
            return input.Substring(input.IndexOf('\n') + 1);
        }

        return input;
    }

    private static bool HasCustomDelimiter(string input)
    {
        return input.StartsWith("//");
    }

    private string[] GetDelimiters(string customDelimiters)
    {
        var delimiters = new List<string> { ",", "\n" };
        if (string.IsNullOrEmpty(customDelimiters))
        {
            return delimiters.ToArray();
        }

        if (customDelimiters.StartsWith("["))
        {
            var longDelimiters = customDelimiters.Replace("[", "]").Split(new[] { ']' }, StringSplitOptions.RemoveEmptyEntries);
            delimiters.AddRange(longDelimiters);
            return delimiters.ToArray();
        }

        delimiters.Add(customDelimiters[0].ToString());
        return delimiters.ToArray();
    }

} 

Reading through the code we can see that there are a lot of string searching,splitting and even replacements.

But it works and all tests passed. Great!!

The challenge

I presented this to a colleague of mine and he said that this is all fine, but he was still waiting to see the stream version. Stream version???...oh wait, you mean that I should read the input string character by character and never read the same character twice.

The description clearly states that no effort should be done to validate for invalid input apart from not accepting negative numbers.

The first thing to do is to forget all about finding custom delimiters and splitting strings, just follow this simple rule:

Read the string from start to end and ignore everything that is not a numeric value.

This is what I ended up with.

public class Calculator : ICalculator
{
    public int Add(string input)
    {
        double sum = 0;            
        double number = 0;
        foreach (var character in input)
        {                
            double numericValue = Char.GetNumericValue(character);                
            if (numericValue != -1.0)
            {                   
                if (number > 0)
                {
                    number = number * 10;
                }                                                
                number = number + numericValue;                   
            }                                
            else
            {                 
                sum = sum + (number >= 1000 ? 0 : number);
                if (character == '-')
                {
                    throw new ArgumentOutOfRangeException();
                }

                number = 0;
            }                
        }

        return (int)sum + (int)(number >= 1000 ? 0 : number);            
    }
}  

All tests are still green.

Just goes to show that there are more than one approach to any given problem :)

OSVersion

Just recently I was involved in a discussion about logging and the need to include the name of the operation system and the service pack level in the application log file.

While you can get to this information in a number of ways, it is important that it will work as expected even for future versions of Windows.

The following code shows a console application that displays the name of the operating system along with the current service pack level.

class Program
{
    static void Main(string[] args)
    {
        Console.WriteLine("Operating System : {0} ({1} bit) ", GetOperationSystemName(), IntPtr.Size == 4 ? "32" : "64");
        Console.WriteLine("Service Pack : {0}", Environment.OSVersion.ServicePack);
    }

    private static string GetOperationSystemName()
    {
        var searcher = new ManagementObjectSearcher("SELECT Caption FROM Win32_OperatingSystem");
        var managementObject = searcher.Get().OfType<ManagementObject>().FirstOrDefault();
        return managementObject != null ? managementObject["Caption"].ToString() : "Unknown OS";
    }
}

Running the console application displays the following:

.\OSVersion.exe

Operating System : Microsoft Windows 7 Enterprise  (64 bit)
Service Pack : Service Pack 1

The fallacy of the single composition root?

This article discusses DI (dependency injection) and the pattern of having one single unique composition root.

Composition roots

When we talk about composition roots, we usually talk about a single unique location where services are registered with the IoC container. This is then executed at the application entry level.

This ensures that only the composition root needs to have a reference to the IoC container and we can make the switch to another container without going through the whole application and its assemblies. Life is good and everyone is happy.

Now let me tell you a sad story about an application using this exact approach and how we ended up rethinking the whole single composition root idea.

As with most projects, they start out small and performance problems are something that only exists in galaxies far far away.

In order to save time during development, we decided that we could just scan all the assemblies and register services using some sort of convention instead of explicitly register each service. This worked out great and developers did not have to think about anything else than to ensure that their assembly was present in the bin folder. We also executed the same composition root during unit testing which meant that we did integration tests with the same configuration as in production (This is actually a good thing).
Man, those were happy times...for a while.

First sign of trouble

One day I got an e-mail from one of my colleagues complaining about increased startup time. It also took a long time just to execute a single unit test. The application was only half way completed and we were already faced with a serious performance problem. As it turned out, the application had grown from your typical "10 assemblies greenfield project", into something that now consisted of 100 assemblies where each of these assemblies represented services that could potentially be requested from the container. It now took 10 seconds to start the application and 10 seconds to start a single unit test. Happy times were going bye-bye.

Note: Not discussing whether the container should be used in unit tests. Think of integration tests.

What's the problem

It did not took us long to realize that the problem lied within the fact that we now scanned 100 assemblies during the startup of the application. Even for unit tests that only really needed 5 of those assemblies to run, we loaded all 100 assemblies into the application domain. Of course we needed to, we did not know which 5 of those 100 assemblies that was needed for the test.
As for the application that was a desktop application, we created a really fancy splash screen to amuse the end user while all this assembly loading was going on in the background.
It did not take long for the amusement to wear off among the users.

No more assembly scanning

Having identified the assembly scanning as the root of our problems, we decided that each service should be explicitly registered with the container. This required a little more work from the developers, but that was okay. After all, we needed to fix this problem before it really got out of hand.

container.ScanAllAssemblies()  

to this

container.Register<IFoo, Foo>();
container.Register<IBar, Bar>();
--followed by hundreds more

We were now faced with another issue. The assembly that contained the composition root, now had to reference all the other 99 assemblies so that the composition root could register the services. The fact that we now needed the scrollbar just to browse the references list should have made us think twice. Sadly it did not.

The problem remains

So after a major refactoring process to explicitly register services and abandon the assembly scanning, we were still seeing a significant delay during startup although it had improved a little. The improvement was relative meaning that the startup time kept increasing as more assemblies got added to the application. The improvement came from not having to do reflection on all the types in each assembly.

So how could it be that we still saw this behavior?

To answer this, we need to look a little closer into how assembly loading work in the CLR (Common Language Runtime).

The CLR tries to minimize the number of assemblies that goes into the application domain by only loading the assemblies that are actually needed. So when does an assembly get loaded?

It gets loaded when code that uses types from that assembly is executed. To be even more precise, it gets loaded when the code is compiled by the Just-In-Time compiler.

This is actually a very simple and clever approach because the number of assemblies loaded is now dependent on the actual path of execution.

Now lets review the code from our composition root.

container.Register<IFoo, Foo>();
container.Register<IBar, Bar>();
--followed by hundreds more

What we essentially have here is a method that bypasses all this cleverness enforced by the CLR.

Because we now have a method that references types from all those 100 assemblies, we force the CLR to load all the assemblies into the application domain. As bad as this in release builds, this is even worse during debug since all the assemblies and their symbol files needs to be loaded too. Starting the application or running a unit tests still provided a nice opportunity to check the latest news on Facebook.
Happy times now seem ages ago.

Laziness is good

As opposed to life in general, in computer science, laziness is really a good thing. Doing things only when we really have to makes for highly optimized applications in terms of performance and memory footprint.

Being an DI framework author, I'm going to explain how to deal with this problem using the LightInject service container. Although the remaining examples will target this container, I'm confident that similar solutions could be applied using other DI frameworks.

In LightInject there is an interface called ICompositionRoot that serves the purpose of registering services with the service container.

The interface is rather simple and looks like this:

/// <summary>
/// Represents a class that acts as a composition root for an <see cref="IServiceRegistry"/> instance.
/// </summary>
internal interface ICompositionRoot
{
    /// <summary>
    /// Composes services by adding services to the <paramref name="serviceRegistry"/>.
    /// </summary>
    /// <param name="serviceRegistry">The target <see cref="IServiceRegistry"/>.</param>
    void Compose(IServiceRegistry serviceRegistry);
}

When an assembly is scanned, LightInject will check to see if the assembly contains implementations of this interface and execute the Compose method. If no composition root is found, services will be registered using a set of conventions (not covered here).

This implies that we are back to scanning assemblies and we already stated that did not work out to well, didn't we?

Well, it's all a matter of how and when.

Lazy Registration

HOW

How we scan an assembly is important with regards to performance when registering services. We have already said that LightInject will use the composition root if found within the target assembly. This is essentially done by inspecting all types and check to see if one or more types implements the ICompositionRoot interface. While that certainly works, it is far from optimal as this could also lead to unnecessary assemblies to be loaded into the application domain.

The solution is to provide a little metadata at the assembly level to help the assembly scanner in the right direction.

/// <summary>
/// Used at the assembly level to describe the composition root(s) for the target assembly.
/// </summary>
[AttributeUsage(AttributeTargets.Assembly, AllowMultiple = true)]
internal class CompositionRootTypeAttribute : Attribute
{
    /// <summary>
    /// Initializes a new instance of the <see cref="CompositionRootTypeAttribute"/> class.
    /// </summary>
    /// <param name="compositionRootType">A <see cref="Type"/> that implements the <see cref="ICompositionRoot"/> interface.</param>
    public CompositionRootTypeAttribute(Type compositionRootType)
    {
        CompositionRootType = compositionRootType;
    }

    /// <summary>
    /// Gets the <see cref="Type"/> that implements the <see cref="ICompositionRoot"/> interface.
    /// </summary>
    public Type CompositionRootType { get; private set; }
}

The assembly scanner will first check the assembly for this attribute and execute the composition roots according to the type information from the attribute.

This saves us from inspecting each and every type in the assembly and provides the most efficient way of locating the composition root.

To summarize how this works:

1. Check for CompositionRootTypeAttribute. If found, execute composition root(s).
2. Check all types for implementations of the ICompositionRoot interface. If found execute composition root(s).
3. Register types according to a set of conventions.

WHEN

When the container encounters a service that cannot be resolved, it will scan the assembly that contains the requested service. After the assembly has been scanned, the container will continue to try resolving the requested service. If the service is still not found, the container will throw an exception as usual.
This allows us to implement multiple lazy compositions roots and hence minimize the number of assemblies loaded into the application domain at any given time during the lifespan of the application.

Example

This example shows a simple console application that consists of the application itself and two additional assemblies.

FooAssembly

public interface IFoo {}

public class Foo : IFoo {}

public CompositionRoot : ICompositionRoot
{
    public void Compose(IServiceRegistry registry)
    {
        registery.Register<IFoo, Foo>();
    }
}  

BarAssembly

public interface IBar {}

public class Bar : IBar {}

public CompositionRoot : ICompositionRoot
{
    public void Compose(IServiceRegistry registry)
    {
        registery.Register<IBar, Bar>();
    }
}  

As we can see each assembly has its own composition root and takes care of registering services from the containing assembly.

Further both the FooAssembly and the BarAssembly will have the CompositionRootTypeAttribute defined at the assembly level.

[assembly: CompositionRootType(typeof(CompositionRoot)]

The console application now only references the FooAssembly and the composition root within the FooAssembly will be lazily executed once the IFoo instance is requested.

The Main method of our console application would look something like this.

static void Main(string[] args)
{
    var container = new ServiceContainer();                
    IFoo foo = container.GetInstance<IFoo>();                
} 

The FooAssembly is loaded into the application domain when the Main method is executed (compiled by the J.I.T), while the BarAssembly is left unloaded until we execute code that uses types from this assembly.

What's the downside?

There is always at least two sides to every story and even this has some obvious downsides.

First and foremost it means that every assembly that contains a composition root, would also need to have a reference to the assembly that contains the ICompositionRoot interface. This is a container specific interface and should ideally only be referenced at the "true" composition root (application entry).

It's all a matter of give and take while this certainly creates more references to the service container library, it is still far from the downsides of implementing the service locator pattern which basically means references to the container everywhere in our code.

Secondly, we encourage developers to explicitly registers their services in the appropriate composition root where we in the past got away with plain old assembly scanning.

While this might push more plumbing work over to the developers, it also becomes less "magic" and might become easier to debug. If the service is unable to resolved, it is probable just missing that one line of code that registers the service with the container. It's as simple as that really.

Update 8/19/2013 2:12:56 PM

The article has been updated to emphasize the meaning of having "distributed" composition roots.

Feedback

I would appreciate any feedback whether you think this looks cool or the worst idea ever :)

mail: bernhard.richter@gmail.com

twitter: @bernhardrichter

Implementing Find and Replace for Expression trees

UPDATE September 2011

The code described below in now available from NuGet that installs a single file (ExpressionExtensions.cs) into your project.

In addition to search and replace, methods for expression composition (AND/OR) has been added.

Get it from NuGet…….PM> Install-Package ExpressionTools

The source is now hosted at GitHub

Introduction

Imagine that you are working with an Expression tree and you need to check
if the tree contains a certain Expression with certain property values.

What we basically would need to do is to visit every node(Expression) in the tree
and check to see if the Expression matches what we are searching for.

First of all we need a class that visits all the nodes in the tree.

The ExpressionVisitor

The ExpressionVisitor class that can be found in the MSDN documentation is sort of an
all purpose class when dealing with expressions.

It was originally written by Matt Warren (the father of Linq to Sql) and was first published
as part of his "Building an IQueryable Provider" series

The expression visitor was considered so essential for everyone working with expression trees,
so they finally included the source in the MSDN documentation.
The implementation is really quite simple. It takes care of visiting all the nodes in the tree as well as it takes care of rewriting the tree if we make any changes to it. Great stuff!!

So what we are aiming for here is some kind of method that returns a list of expressions
that matches any predicate defined for the expression (sub)tree.

Implementing Find

Extension methods are a great way of adding new features to existing types and in this case
we want to extend the Expression class.

/// <summary>

/// Returns a list of <typeparamref name="TExpression"/> instances

/// that matches the <paramref name="predicate"/>.

/// </summary>

/// <typeparam name="TExpression">The type of <see cref="Expression"/>

/// to search for.</typeparam>

/// <param name="expression">The <see cref="Expression"/> that represents the sub tree for which to start searching.</param>

/// <param name="predicate">The <see cref="Func{T,TResult}"/> used to filter the result</param>

/// <returns>A list of <see cref="Expression"/> instances that matches the given predicate.</returns>

public static IEnumerable<TExpression> Find<TExpression>(this Expression expression, Func<TExpression, bool> predicate) where TExpression : Expression

{

    var finder = new ExpressionFinder<TExpression>();

    return finder.Find(expression, predicate);

}

Given the following expression:

Expression<Func<string, bool>> expression = (s) => s.Length == 5;

we can do a search in the expression tree by doing:

var binaryExpressions = expression.Find<BinaryExpression>(b => true);

or vi could search for ConstantExpression like this:

var constantExpressions = expression.Find<ConstantExpression>(c => (int)c.Value == 5);

A top-down search is performed in the sub tree that the target expression represents,
meaning that we can choose to investigate the whole tree or just individual sub trees.

The ExpressionFinder

This is a very simple class that override the Visit method that is called for every node in the expression tree.

If the expression being visited matches the given predicate, we add it to the list of matching expressions.

/// <summary>
/// A class used to search for <see cref="Expression"/> instances. 
/// </summary>
/// <typeparam name="TExpression">The type of <see cref="Expression"/> to search for.</typeparam>
public class ExpressionFinder<TExpression> : ExpressionVisitor where TExpression : Expression
{
    private readonly IList<TExpression> _result = new List<TExpression>();
    private Func<TExpression, bool> _predicate;

    /// <summary>
    /// Returns a list of <typeparamref name="TExpression"/> instances that matches the <paramref name="predicate"/>.
    /// </summary>
    /// <param name="expression">The <see cref="Expression"/> that represents the sub tree for which to start searching.</param>
    /// <param name="predicate">The <see cref="Func{T,TResult}"/> used to filter the result</param>
    /// <returns>A list of <see cref="Expression"/> instances that matches the given predicate.</returns>
    public IEnumerable<TExpression> Find(Expression expression, Func<TExpression, bool> predicate)
    {
        _result.Clear();
        _predicate = predicate;
        Visit(expression);
        return _result;
    }

    /// <summary>
    /// Visits each node of the <see cref="Expression"/> tree checks 
    /// if the current expression matches the predicate.         
    /// </summary>
    /// <param name="expression">The <see cref="Expression"/> currently being visited.</param>
    /// <returns><see cref="Expression"/></returns>
    protected override Expression Visit(Expression expression)
    {                        
        if (expression != null && expression is TExpression)
        {
            if (_predicate((TExpression)expression))
                _result.Add((TExpression)expression);
        }

        return base.Visit(expression);
    }
}

Implementing Replace

In addition to being able to search for expressions it would also be nice if we could do a simple replace using a similar syntax.

Another extension method is called for:

/// <summary>

/// Searches for expressions using the given <paramref name="predicate"/> and replaces matches with

/// the result from the <paramref name="replaceWith"/> delegate.

/// </summary>

/// <typeparam name="TExpression">The type of <see cref="Expression"/> to search for.</typeparam>

/// <param name="expression">The <see cref="Expression"/> that represents the sub tree

/// for which to start searching.</param>

/// <param name="predicate">The <see cref="Func{T,TResult}"/> used to filter the result</param>

/// <param name="replaceWith">The <see cref="Func{T,TResult}"/> used to specify the replacement expression.</param>

/// <returns>The modified <see cref="Expression"/> tree.</returns>

public static Expression Replace<TExpression>(this Expression expression, Func<TExpression, bool> predicate, Func<TExpression,Expression> replaceWith) where TExpression : Expression

{

    var replacer = new ExpressionReplacer<TExpression>();

    return replacer.Replace(expression, predicate, replaceWith);

}

We might think of the Replace method as an ad-hoc alternative to modifying the expression tree using an ExpressionVisitor derivate.

Given the following expression:

Expression<Func<string, bool>> expression = (s) => s.Length == 5;

we want to modify the expression tree by replacing the BinaryExpression(Equal) with a BinaryExpression(NotEqual).

This can now be achieved very easily by doing:

var modifiedExpression = expression.Replace<BinaryExpression>(b => true, 
    (b) => Expression.NotEqual(b.Left, b.Right));

The ExpressionReplacer

Similar to the ExpressionFinder it filter outs the target expressions, but instead of returning the matching expressions, it replaces the
matching expression with whatever the Func<TExpression,Expression> delegate returns.

One thing to notice here is that matching expression is passed to the replaceWith delegate.

/// <summary>
/// A class that is capable of doing a find and replace in an <see cref="Expression"/> tree.
/// </summary>
/// <typeparam name="TExpression">The type of <see cref="Expression"/> to find and replace.</typeparam>
public class ExpressionReplacer<TExpression> : ExpressionVisitor where TExpression : Expression
{
    
    private Func<TExpression, Expression> _replaceWith;
    private Func<TExpression, bool> _predicate;
    
    
    /// <summary>
    /// Searches for expressions using the given <paramref name="predicate"/> and 
    /// replaces matches with the result from the <paramref name="replaceWith"/> delegate.          
    /// </summary>
    /// <param name="expression">The <see cref="Expression"/> that 
    /// represents the sub tree for which to start searching.</param>
    /// <param name="predicate">The <see cref="Func{T,TResult}"/> used to filter the result</param>
    /// <param name="replaceWith">The <see cref="Func{T,TResult}"/> 
    /// used to specify the replacement expression.</param>
    /// <returns>The modified <see cref="Expression"/> tree.</returns>
    public Expression Replace(Expression expression, Func<TExpression, bool> predicate, 
        Func<TExpression, Expression> replaceWith)
    {
        _replaceWith = replaceWith;
        _predicate = predicate;
        return Visit(expression);
    }

    /// <summary>
    /// Visits each node of the <see cref="Expression"/> tree checks 
    /// if the current expression matches the predicate. If a match is found 
    /// the expression will be replaced.        
    /// </summary>
    /// <param name="expression">The <see cref="Expression"/> currently being visited.</param>
    /// <returns><see cref="Expression"/></returns>        
    protected override Expression Visit(Expression expression)
    {
        if (expression != null && expression is TExpression)
        {
            if (_predicate((TExpression)expression))
                return _replaceWith((TExpression)expression);
        }
        return base.Visit(expression);
    }
}

Enjoy!!!