Resource management in complex C# applications

Resource Management - Problems and Solutions

When writing software applications, many a time we need to deal with resources such as files, database connections, Windows Communication Foundation (WCF) proxies, etc.

What is essential about such resources, is that we need to manage them correctly.

For example, when a file is opened, read or written to, it should be closed. We need to make sure that we always close the file even if an error occurred during processing of data.

Consider the following example:

private static void ProcessFile()
{
    var fileReader = new StreamReader("c:\\file.txt");

    while (true)
    {
        var line = fileReader.ReadLine();

        if (line == null)
            break;

        ProcessLine(line);
    }

    fileReader.Close();
}

This code opens file.txt by invoking the constructor of the StreamReader class. It then loops to read and process the lines of the file, one by one. After the loop ends, it closes the file by invoking the Close method on the StreamReader object.

This code does not manage the file resource correctly.

Are you keeping up with new developer technologies? Advance your IT career with our Free Developer magazines covering C#, Patterns, .NET Core, MVC, Azure, Angular, React, and more. Subscribe to the DotNetCurry (DNC) Magazine for FREE and download all previous, current and upcoming editions.

If an exception is thrown in the ProcessLine method, the “fileReader.Close()” statement will not execute and the file will remain open. If we then try to write to the same file, it will fail with an IOException saying that the file cannot be accessed because it is currently being used.

In .NET, we can easily fix this problem by using the file inside a try block and then closing it in a corresponding finally block. This pattern is so common that in C# we have a special statement, the using statement, that we can use to create the effect of closing (or Disposing in more formal terminology), the resource in a finally block.

Here is an example:

using (var fileReader = new StreamReader("c:\\file.txt"))
{
    while (true)
    {
        var line = fileReader.ReadLine();

        if (line == null)
            break;

        ProcessLine(line);
    }
}

This works because the StreamReader class implements the IDisposable interface. Before execution leaves the using block, it is guaranteed that the Dispose method of the StreamReader object will be called and the file will be closed even if an exception is thrown. For more information about the IDisposable interface and the using statement, see this MSDN reference (bit.ly/dnc-dispose).

Another issue related to resource management is the efficient use of resources.

One way we can use a resource, say a HTTP connection (the underlying TCP connection), is to open the connection, invoke some HTTP request, receive the response, and then close the connection. If we need to send 10 HTTP requests, we would open a new connection ten times and close it an equal number of times.

It makes more sense however to open the connection, use it to send multiple HTTP requests for some period of time and then close it.

This is true because opening a connection is expensive.

Consider HTTPS (HTTP with SSL/TLS) connections for example. Every time we open a new HTTPS connection, the client needs to authenticate the server and sometimes even the server requires client authentication. Also, the two parties need to agree on a symmetric key to use to protect the session.

When working with resources in the scope of a single method, handling the two mentioned issues, is easy. For the first issue, we can use the try…finally syntax (or the using statement) to make sure we always close resources. And for the second issue, we can open the resource, use it many times, and then close it.

However, when a resource needs to be used from multiple objects in a complex application, these issues become harder to handle.

In order to efficiently use the resource, the application objects need to share the same resource instance. Also, some resources need to be shared by multiple objects because they represent something unique, e.g. a particular file on the file system.

If resources are shared between multiple objects, how can we make sure that they are closed when they are no longer needed? Who is responsible for closing the resource and when? Who is responsible for opening the resource in the first place?

Before going forward, let’s first describe a resource in a somewhat formal way. A resource (in this article) refers to something that:

1. Needs to be opened before used

2. Opening it is expensive

3. Needs to be closed after we are done using it

4. Might represent something unique that needs to be shared

Resources versus Single-method operations

Before going into how to deal with resources in the scope of large applications, let’s first consider the difference between how we deal with resources and how we deal with single-method operations.

In our applications, we would like to have operations that are represented by single methods.

For instance, we would like to have a method called CalculateDiscountForCart that takes a cart data object and calculates how much discount we should apply. This method by itself represents a useful operation that we can execute.

When dealing with resources on the other hand, we need to invoke multiple methods.

For example, we need to invoke a method to open the resource, another one to use it, and another one to close it.

Consider the following example:

public interface ICartDiscountCalculator
{
    decimal CalculateDiscountForCart(Cart cart);
}

This interface has a single method, CalculateDiscountForCart. Consumers need only to worry about this single method.

Now consider the following example of some resource:

public interface IResource
{
    void OpenConnection();

    string QuerySomeData();

    void CloseConnection();
}

Now the consumer of this interface needs to worry about three methods. It needs to invoke them in a particular order, and it also needs to make sure that the CloseConnection method is called if the OpenConnection method was called.

It is a lot easier to deal with contracts that have a single method (or contracts that have multiple methods but that are completely independent). Therefore, in the rest of the article, I will discuss solutions to resource management problems that will allow the direct consumers of the resource to only have to deal with a single method, e.g. QuerySomeData.

Now let’s consider a simple approach to managing resources in a large application.

Managing Resources - A simple approach

Let’s first consider an approach to resource management that handles resources correctly, provides a simple interface for consumers, but might be inefficient.

A simple way with which we can handle a resource is open and close it every time we need to use it.

As discussed before, in a complex application, we should hide this behind a simple interface so that other code blocks can use it without the need to care about any resource management.

Consider this example:

public interface IResourceUser
{
    string QuerySomeData();
}

public class SimpleResource : IResourceUser
{
    public string QuerySomeData()
    {
        //Open some resource
        try
        {
            //Query the resource
        }
        finally
        {
            //Close the resource
        }
    }
}

Now, the consuming code only needs to know about the QuerySomeData method.

Consider the following object graph:

In this figure, Unit 3 and Unit 5 are the direct consumers of a SimpleResource object. They are connected to this object via the IResourceUser interface. They use it to call the QuerySomeData method.

Unit 1 (which in this figure is the starting point, e.g. a controller in a Web API application) has a dependency on IContract1. In this figure, it is connected to an instance of the Unit2 class. Unit 1 invokes Unit 2, which in turn invokes Unit 3, Unit 4 and eventually Unit 5.

Here is how we can compose such a graph in the Composition Root:

var resource = new SimpleResource(resourceAddress);

var instance =
    new Unit1(
        new Unit2(
            new Unit3(resource),
            new Unit4(
                new Unit5(resource))));

Each time Unit 3 or Unit 5 uses the resource, the resource is opened, used, and then closed.

If Units 3 and 5 call QuerySomeData 10 times, this means that we will open and close the resource 10 times.

For some scenarios, this is perfectly acceptable.

But for others, this solution is inefficient.

How can we have the consumers know only about QuerySomeData, but still be able to use the resource multiple times before we close it?

Explicit management

To solve the issue just mentioned, we can change the resource class to have explicit Open and Close methods like this:

public interface IResourceUser
{
    string QuerySomeData();
}

public interface IOpenClosable
{
    void Open();

    void Close();
}

public interface IResource : IResourceUser, IOpenClosable
{
}

public class ExplicitResource : IResource
{
    //Handle to resource is stored in a private field

    public string QuerySomeData()
    {
        //Query the resource
    }

    public void Open()
    {
        //Open the resource
    }

    public void Close()
    {
        //Close the resource
    }
}

The direct consumer of the resource needs to have a dependency only on IResourceUser because it only needs to invoke QuerySomeData. However, someone else needs to call Open and Close. That someone needs to call Open before the direct consumers of the resource have the chance to invoke QuerySomeData, and it needs to call Close after the direct consumers are done using the resource.

Consider the following updated object graph:

Unit 1 is now connected to an instance of an ExplicitResourceManager class. This class is simply a decorator for IContract1. It intercepts the call between Unit 1 and Unit 2 and opens the resource by invoking the Open method on a dependency of type IOpenClosable (which is connected to the same resource instance as Unit 3 and 5).

When execution eventually goes back to the ExplicitResourceManager unit, it closes the resource via IOpenClosable.Close.

Here is how we compose this graph in the Composition Root:

var resource = new ExplicitResource(resourceAddress);

var instance =
    new Unit1(
        new ExplicitResourceManager(
            new Unit2(
                new Unit3(resource),
                new Unit4(
                    new Unit5(resource))),
            openClosable: resource));

So far, we have achieved two goals:

1. Direct consumers of the resource need only to deal with a single method.

2. The resource can be used multiple times before it is closed.

We can still modify our approach to solve the following issues:

1. The resource might be opened before it is really needed. For example, in the previous graph, the resource is opened immediately before Unit 2 is invoked. What if there are some 10 seconds between Unit 2 being invoked and Unit 3 and 5 deciding to use the resource?

2. What if Unit1 is invoked 100 times per second? This means that we open the resource and close it 100 times per second (although each time we might use it a few times). Wouldn’t it be better if we open the resource once, use it for 100 invocations of Unit 1, and then close it?

To solve the first issue, we can make the ExplicitResourceManager unit responsible only for closing the resource and then make the ExplicitResource object auto-open when QuerySomeData is invoked. To do so, some state inside the ExplicitResource object needs to keep track of whether the resource is currently opened.

In the next section, we will build on this to solve the second issue.

Automatic resource management

Another approach to manage resources is to automatically open and close them.

We can make the resource open itself before it is used (like we did in the simple approach), and then have it close automatically after some predefined period of idle time.

Take a look at the AutomaticManager class here. This class implements the IResourceUser interface and therefore its consumers need to care only about a single method, the QuerySomeData method.

This class has a dependency on IResource. It also has a configuration parameter, maximumIdleTimeBeforeClosing of type TimeSpan. This parameter will allow us to configure how much time the resource is allowed to be idle, before it is automatically closed.

In the QuerySomeData method, we first check to see if the resource is already opened and open it if needed. We then invoke the QuerySomeData method on the resource. Then, in the finally block, we want to create a timer that would close the resource after some idle time.

To do so, we do the following:

1. We keep track of the last time a request (a call to QuerySomeData) is made by maintaining a stopwatch, lastRequestStopwatch, and restarting it every time the resource is used.

2. We keep track of the number of requests made since the resource was opened, via the requestNumber field.

3. If requestNumber is 1, i.e., if this is the first request since the resource was opened, then we schedule the closing of the resource in the future. We do this inside the ScheduleResourceClosing method. More specifically, in the following line:

await Task.Delay(delay).ConfigureAwait(false);

..we ask the system to asynchronously wait for some time (specified by the maximumIdleTimeBeforeClosing parameter). The calling thread will return immediately when “await” is executed. When the timer elapses (the task returned by Task.Delay is completed), the ScheduleResourceClosing method will continue execution on a thread pool thread because of the ConfigureAwait(false) call.

If requestNumber is still 1, then this means that no more requests were sent while we were waiting, and therefore, we simply close the resource, reset the stopwatch, and the requestNumber variable.

If requestNumber is not 1, then this means that while we were waiting, other requests were made. We need to wait more so that we end up waiting maximumIdleTimeBeforeClosing from the time of the last request.

This is the reason of the loop inside the ScheduleResourceClosing method. We calculate a new value of delay that takes into account the time that has passed since the last request. This is why we keep track of the time of the last request via the lastRequestStopwatch stopwatch.

In Program.cs, in the TestAutomaticManager method, I have written some code to test this:

IResourceUser resourceUser =
    new AutomaticManager(
        new FakeResource(),
        TimeSpan.FromMilliseconds(500));

resourceUser.QuerySomeData();

Thread.Sleep(400);

resourceUser.QuerySomeData();

Thread.Sleep(600);

resourceUser.QuerySomeData();

Console.WriteLine("Done");

Console.ReadLine();

This code first creates a FakeResource object. This object simply prints to the console when the Open, Close, or QuerySomeData methods are invoked.

The code then wraps this fake resource object with an instance of the AutomaticManager class specifying a value of 500 milliseconds for maximumIdleTimeBeforeClosing.

We then invoke QuerySomeData three times. Intentionally, we wait different periods of time between every two invocations of QuerySomeData to test the behavior of AutomaticManager.

This is what is printed to the console when the application executes:

Opening the resource

Querying the resource

Querying the resource

Closing the resource

Opening the resource

Querying the resource

Done

Closing the resource

When we call QuerySomeData for the first time, the resource is opened. We see two messages in the console indicating that the resource was opened and then queried.

Then after we wait 400 milliseconds and invoke the method again, the resource is queried and a message in the console is printed to indicate this. No messages to indicate that the resource is opened or closed are printed, as expected.

Then we wait 600 milliseconds. Before waiting is complete, a message indicating that the resource was closed is printed. This is because the resource auto-closes after 500 milliseconds.

In order to service the third request, the resource is opened again. We see two messages indicating the opening and the querying of the resource.

We then see the “Done” message printed to the console.

After 500 milliseconds, the resource auto-closes again and a closing message is printed to the console.

Here is how the graph for the imaginary example used earlier would look like after we use this approach:

This graph is very similar to the simple approach graph. We don’t need any objects up in the graph to manage the resource explicitly.

The automatic management aspect

In a real application, you might have many resources, each with different interfaces, and many with multiple usage/querying methods.

The AutomaticManager class works only with the example IResource and IResourceUser interfaces. We can do the same thing for other resource interfaces by creating an aspect.

For more information about aspects, see the Aspect Oriented Programming (AOP) in C# with SOLID article (bit.ly/dnc-aop-solid).

Take a look at the AutomaticManagerAspect class which is very similar to the AutomaticManager class. This aspect is built using the DynamicProxy library.

Also take a look at the TestAutomaticManagerAspect method in the Program.cs file to see how to use it. The aspect expects a dependency of type IOpenClosable in the constructor. You can use such an interface in your resource classes for the Open and Close methods. If you can’t, then you can give the aspect an adapter object that adapts your resource object to IOpenClosable.

Customizing the automatic manager

The approach described above can be modified and customized in many ways.

For example, currently the automatic management classes assume that the resource can only be used by a single thread. You might encounter a scenario where you want the same resource instance to be used by multiple threads.

Or you can have a pool of resources managed automatically by the automatic manager.

For example, you can configure the manager to have a pool of at most three instances of the resource. If three concurrent threads want to use the resource, each of them will get a different instance. A fourth thread that wants to use the resource would have to wait for one of the other three threads to finish using the resource.

Another customization is to enable the resource to auto-renew after some time.

For example, you might want a connection to be closed and opened again every ten minutes. Or you might customize it to make it renew at 12:00 AM every day.

Or you might want the resource to auto-renew every twenty invocations, e.g. after QuerySomeData is called twenty times.

Existing automatic resource management in the .NET framework

For performance reasons, the .NET framework has some automatic management features built into it.

For example, if you use the HttpClient class to send multiple requests to some HTTP server from a single thread, a single connection will be used. The connection will only be closed after some time elapses without any requests being sent. The mechanics of this feature are different from those of the approach I described above, but the general idea is the same.

The consumer of the HttpClient class does not need to worry about opening and closing the connection. All the consumer needs to do is invoke methods that use the client like HttpClient.GetAsync, or HttpClient.PostAsync, etc.

This is very similar to the idea I discussed in the article where the consumer needs to only worry about using the resource, not opening and closing it. Although the HttpClient class implements the IDisposable interface, it is a recommended practice that developers create a single instance of it and use it for the whole duration of the application, i.e., they don’t need to dispose it. See this blog post (bit.ly/dnc-http-best) for more details.

Another example of automatic resource management in the .NET framework is related to ADO.NET and SQL Server.

When using ADO.NET to connect to SQL Server, ADO.NET manages a pool of connections to the SQL server. This means that when you open and close a SqlConnection object, the real connection remains open for some time. If you later (within some frame of time) open another SqlConnection object with the same connection string, you would use the same real connection to the SQL server.

Again, the specifics of this feature are different from the approach I described in the article, but the general idea is the same. For more information about SQL Server Connection Pooling see this article here.

One particular thing to note about SqlConnection object is that you still need to “Close” the connection for it to be marked as not used. You also need to explicitly “Open” the connection to retrieve an existing connection from the pool or to actually create a new one.

This is different from the HttpClient scenario and the automatic management example I described in the article. In both of these cases, the consumer doesn’t need to invoke “Open” or “Close”.

As you can see, the .NET framework already does a lot of work to make sure many kinds of resources are managed in a somewhat automatic way so that developers don’t need to worry about doing such management themselves.

Then when should a developer use the approaches described in this article?

Resource management Examples

In general, you can use the approaches described in this article whenever you work with resources that meet the definition I provided in the beginning of the article. These resources do not have any built-in automatic management features, and you need to use them from multiple objects in the application.

Let’s consider some examples.

The SmtpClient class

The SmtpClient class in the .NET framework allows us to send email messages. We can use the Send method to send messages.

Internally, the SmtpClient class implementation pools SMTP connections. This means that when we invoke Send to send a message, an existing connection might be used.

This is great.

However, the SmtpClient class does not send the QUIT command to end existing SMTP sessions (to free server resources) until we explicitly call the Dispose method. This Dispose method will not be called automatically after some time of inactivity (or after some other condition as described in the Customizing the automatic manager section).

We can easily encapsulate the SmtpClient class inside some SmtpResource object. The Open method would create an instance of the SmtpClient class, and the Close method would call Dispose.

We should also provide some methods to send the email messages.

We can then use the AutomaticManagerAspect to auto-manage this resource allowing multiple objects to use the SMTP client resource without worrying about disposing/closing anything. We can make the resource close after some period of time of inactivity, or maybe after twenty messages have been sent.

A locked file

Another example could be when we might want to log some sensitive information to some file in some folder.

Each day we create a new file in this folder and open it for writing, and for security reasons we specify that we don’t want anyone else to be able to access the file. To do so, we specify the FileShare.None option when we create and open the file, for example via the File.Open method.

The file also needs to be signed before it is closed. This means that special code needs to run before we close the file.

Multiple objects in the application graph need to access the resource object that encapsulates the FileStream returned by the File.Open method.

The Open method of the resource object would first determine the new filename to use (e.g. generate a random file name or generate a name using the current time), and then call File.Open and store the FileStream in some private field.

The resource would declare some Log method that will be used by the direct consumers.

The Close method would sign the content of the file and then close it.

In this example, the resource is the unique file that needs to be shared by multiple objects in the graph.

Third party libraries

Many third-party libraries allow you to access resources, e.g. an SMTP server. They provide some Open and Close methods but they don’t have any built in automatic management features (e.g. pooling). In this case, you can use the approaches described in this article to encapsulate the resources provided by these libraries to automate the process of managing them.

Conclusion:

In this article I discussed the issues we usually face when we deal with resources.

Resources are things that we cannot simply invoke like ordinary single-method operations. A resource needs to be opened, used, and eventually closed. We need to make sure it is always closed even in case of error.

Also, for performance reasons, we might want to open the resource, use it multiple times, and then close it.

When using a resource from multiple objects in an application, dealing with resources becomes a bit harder. In this article I discussed some approaches on how to deal with them in this scope. These approaches allow the consumers of a resource to use it without needing to worry about opening and closing it.

I also gave examples of automatic resource management that exists in the .NET framework and described some examples of when someone might decide to implement an automatic management solution.

This article was technically reviewed by Damir Arh.

This article has been editorially reviewed by Suprotim Agarwal.

C# and .NET have been around for a very long time, but their constant growth means there’s always more to learn.

We at DotNetCurry are very excited to announce The Absolutely Awesome Book on C# and .NET. This is a 500 pages concise technical eBook available in PDF, ePub (iPad), and Mobi (Kindle).

Organized around concepts, this Book aims to provide a concise, yet solid foundation in C# and .NET, covering C# 6.0, C# 7.0 and .NET Core, with chapters on the latest .NET Core 3.0, .NET Standard and C# 8.0 (final release) too. Use these concepts to deepen your existing knowledge of C# and .NET, to have a solid grasp of the latest in C# and .NET OR to crack your next .NET Interview.

Click here to Explore the Table of Contents or Download Sample Chapters!

Author

Yacoub Massad is a software developer and works mainly on Microsoft technologies. Currently, he works at Zeva International where he uses C#, .NET, and other technologies to create eDiscovery solutions. He is interested in learning and writing about software design principles that aim at creating maintainable software. You can view his blog posts at criticalsoftwareblog.com. He is also the creator of DIVEX(https://divex.dev), a dependency injection tool that allows you to compose objects and functions in C# in a way that makes your code more maintainable. Recently he started a YouTube channel about Roslyn, the .NET compiler.