Stressing SignalR by Crank-ing it up
Posted by: Suprotim Agarwal
in Category ASP.NET
Abstract: A quick walkthrough of the Crank utility used by the SignalR team to generate load for their Test Harness Project
In our previous article we saw how to use the Windows PerfMon.exe utility along with custom SignalR performance counters to monitor performance of the SignalR Test Harness sample. However we had only three clients via three web browser clients. Today we’ll see a utility called Crank - again a part of the SignalR codebase that the team uses to test limits of a SignalR app on a given server.
Thanks to Sumit Maitra for sharing his expertise in this article
Types of SignalR Load
Before we get hands on, it’s worth reviewing the types of load that a SignalR application can handle. Quoting from the recent Build talk by Damian Edwards, SignalR can be subjected to four types of loads
1. Server Broadcast – In this scenario, the server broadcasts a message to all clients. Here the server is sending out only one message, but it’s going out to all connected clients. So this relatively less stressful thing to do.
2. Server Push – What’s the difference with Server Broadcast? In case of Server Push, a message is sent to one or a few connections (owned by the same user preferably). So if a server is going to push lots of messages to lots of different people, it increases the load on the server, specially if you have a lot of connections and lot of messages. However, this is server controlled and can be managed with respect to load.
3. User Event Driven – Chats or Collaboration apps are the perfect example. Load on the server depends on both number of connections and the amount of chatter/communication between the clients or client groups. Increasing number of connections will increase load as will increased communication between the connected people. This is a scenario where it becomes difficult to control load on the server only because of the dynamics of load; communication frequency and number of connections are both dynamic.
4. High Frequency – This scenario is typical for realtime interactive games (e.g. http://shootr.signalr.com). Here the server sends out messages at a very high frequency (SignalR team strongly recommends it be < 25Hz). Also messages sent out to each client varies based on their gameplay. So increase in number of simultaneous users increases load proportionally. Due to the high fixed rate of unique messages, this scenario can put a lot of load on the system quickly.
In our previous article we saw the ‘Server Broadcast’ scenario but for a very limited number of clients. Today we will try it out for a much higher number of clients. The other three scenarios will need creative test harness creation from us, we’ll leave that for another day.
As with our previous article, we’ll start off with the SignalR codebase (can download the zip here, or fork on Github from here).
The Crank is a command line tool and the executable is under src\Microsoft.AspNet.SignalR.Crank. So the executable after a successful build is two folders down under bin\Debug
Crank has the following options
So if we setup Micrsoft.AspNet.SignalR.LoadTestHarness as the default project and run the application from Visual Studio, it will by default run at http://localhost:29573/. To setup Crank such that it creates 100 clients that connect to this app, we type the following command in
crank /Clients:100 /Url:http://localhost:29573/TestConnection /BatchSize:10 /Duration:120
This starts Crank with 100 clients pointing to the URL specified (Note: TestConnection is the name of the PersistentConnection in the LoadTestHarness sample). It creates the 100 clients in batches of 10 and the clients remain connected for 120 seconds as set by the Duration flag. But before we start Crank, let’s setup our performance counters.
Start perfmon.exe as Administrator and add all the SignalR performance counters. If you don’t see the loadtestharness object instance, close your Visual Studio, reopen in Administrator mode, reload project and run. It should appear in the Instance Object list.
Once the Performance counters are selected, we are ready for our test. The perf counter at the beginning of the test looks like the following.
Generating Clients and Broadcasting Data
We are ready to run Crank now, so let’s start with 100 clients for two minutes and see how loaded our server gets.
As we can see below, maintaining 100 connections is pretty easy for SignalR on this machine as it roughly uses about 15% CPU. But at this point, there are no messages being exchanged.
Let’s see what happens when we start exchanging messages.
At 6 messages per second we are almost at 80% load
When we push it to 12 messages per second we hit 100% CPU and the frame rate is unable to keep up as it sticks to 11 FPS.
On this rather low powered machine we were able to sustain 100 connections at 11 messages per second where message size was 32 bytes. This can serve as a baseline and when we are deploying to production, we could run a similar test using the Web Application and Crank to approximately determine its performance.
Limitations of Crank
My buddy Sumit told me that Crank can only connect to PersistentConnection based apps and not Hub based apps. Though it makes it unsuitable straight off the bat, the sample can serve as a base for your custom stress testing harness. In future we will see how we can build a custom testing harness using the .NET client object to call our Hub API and stress it out using multiple connections.
We looked at the Crank utility and saw how we could exercise the Test Harness SignalR Application. Crank is not a tool you use out of the box for your SignalR apps, but something we can use as a guideline to build our own test harnesses.