DotNetCurry Logo

Stressing SignalR by Crank-ing it up

Posted by: Suprotim Agarwal , on 7/10/2013, in Category ASP.NET
Views: 41260
Abstract: A quick walkthrough of the Crank utility used by the SignalR team to generate load for their Test Harness Project

In our previous article we saw how to use the Windows PerfMon.exe utility along with custom SignalR performance counters to monitor performance of the SignalR Test Harness sample. However we had only three clients via three web browser clients. Today we’ll see a utility called Crank - again a part of the SignalR codebase that the team uses to test limits of a SignalR app on a given server.

Thanks to Sumit Maitra for sharing his expertise in this article

 

Types of SignalR Load

Before we get hands on, it’s worth reviewing the types of load that a SignalR application can handle. Quoting from the recent Build talk by Damian Edwards, SignalR can be subjected to four types of loads

1. Server Broadcast – In this scenario, the server broadcasts a message to all clients. Here the server is sending out only one message, but it’s going out to all connected clients. So this relatively less stressful thing to do.

2. Server Push – What’s the difference with Server Broadcast? In case of Server Push, a message is sent to one or a few connections (owned by the same user preferably). So if a server is going to push lots of messages to lots of different people, it increases the load on the server, specially if you have a lot of connections and lot of messages. However, this is server controlled and can be managed with respect to load.

3. User Event Driven – Chats or Collaboration apps are the perfect example. Load on the server depends on both number of connections and the amount of chatter/communication between the clients or client groups. Increasing number of connections will increase load as will increased communication between the connected people. This is a scenario where it becomes difficult to control load on the server only because of the dynamics of load; communication frequency and number of connections are both dynamic.

4. High Frequency – This scenario is typical for realtime interactive games (e.g. http://shootr.signalr.com). Here the server sends out messages at a very high frequency (SignalR team strongly recommends it be < 25Hz). Also messages sent out to each client varies based on their gameplay. So increase in number of simultaneous users increases load proportionally. Due to the high fixed rate of unique messages, this scenario can put a lot of load on the system quickly.

In our previous article we saw the ‘Server Broadcast’ scenario but for a very limited number of clients. Today we will try it out for a much higher number of clients. The other three scenarios will need creative test harness creation from us, we’ll leave that for another day.

Using Crank

As with our previous article, we’ll start off with the SignalR codebase (can download the zip here, or fork on Github from here).

The Crank is a command line tool and the executable is under src\Microsoft.AspNet.SignalR.Crank. So the executable after a successful build is two folders down under bin\Debug

Crank Options

Crank has the following options

crank-options

So if we setup Micrsoft.AspNet.SignalR.LoadTestHarness as the default project and run the application from Visual Studio, it will by default run at http://localhost:29573/. To setup Crank such that it creates 100 clients that connect to this app, we type the following command in

crank /Clients:100 /Url:http://localhost:29573/TestConnection /BatchSize:10 /Duration:120

This starts Crank with 100 clients pointing to the URL specified (Note: TestConnection is the name of the PersistentConnection in the LoadTestHarness sample). It creates the 100 clients in batches of 10 and the clients remain connected for 120 seconds as set by the Duration flag. But before we start Crank, let’s setup our performance counters.

signalr-perf-counters

Start perfmon.exe as Administrator and add all the SignalR performance counters. If you don’t see the loadtestharness object instance, close your Visual Studio, reopen in Administrator mode, reload project and run. It should appear in the Instance Object list.

Once the Performance counters are selected, we are ready for our test. The perf counter at the beginning of the test looks like the following.

setting-up-counters

Generating Clients and Broadcasting Data

We are ready to run Crank now, so let’s start with 100 clients for two minutes and see how loaded our server gets.

As we can see below, maintaining 100 connections is pretty easy for SignalR on this machine as it roughly uses about 15% CPU. But at this point, there are no messages being exchanged.

added-connections-only

Let’s see what happens when we start exchanging messages.

six-connections-load

At 6 messages per second we are almost at 80% load

When we push it to 12 messages per second we hit 100% CPU and the frame rate is unable to keep up as it sticks to 11 FPS.

twelve-connections-load

On this rather low powered machine we were able to sustain 100 connections at 11 messages per second where message size was 32 bytes. This can serve as a baseline and when we are deploying to production, we could run a similar test using the Web Application and Crank to approximately determine its performance.

Limitations of Crank

My buddy Sumit told me that Crank can only connect to PersistentConnection based apps and not Hub based apps. Though it makes it unsuitable straight off the bat, the sample can serve as a base for your custom stress testing harness. In future we will see how we can build a custom testing harness using the .NET client object to call our Hub API and stress it out using multiple connections.

Conclusion

We looked at the Crank utility and saw how we could exercise the Test Harness SignalR Application. Crank is not a tool you use out of the box for your SignalR apps, but something we can use as a guideline to build our own test harnesses.

Was this article worth reading? Share it with fellow developers too. Thanks!
Share on LinkedIn
Share on Google+
Further Reading - Articles You May Like!
Author
Suprotim Agarwal, MCSD, MCAD, MCDBA, MCSE, is the founder of DotNetCurry, DNC Magazine for Developers, SQLServerCurry and DevCurry. He has also authored a couple of books 51 Recipes using jQuery with ASP.NET Controls and a new one recently at The Absolutely Awesome jQuery CookBook.

Suprotim has received the prestigious Microsoft MVP award for nine times in a row now. In a professional capacity, he is the CEO of A2Z Knowledge Visuals Pvt Ltd, a digital group that represents premium web sites and digital publications comprising of Professional web, windows, mobile and cloud developers, technical managers, and architects.

Get in touch with him on Twitter @suprotimagarwal, LinkedIn or befriend him on Facebook



Page copy protected against web site content infringement 	by Copyscape




Feedback - Leave us some adulation, criticism and everything in between!
Comment posted by Sundar on Thursday, January 23, 2014 5:34 AM
D:\Drive D\StudyMaterials\SignalR-Full-Material\SignalR-master\src\Microsoft.Asp
Net.SignalR.Crank\bin\Debug>crank /Clients:100 /Url:http://localhost:29573/TestC
onnection /BatchSize:10 /Duration:12

Unhandled Exception: System.TypeInitializationException: The type initializer fo
r 'System.Net.ServicePointManager' threw an exception. ---> System.TypeInitializ
ationException: The type initializer for 'System.Net.ComNetOS' threw an exceptio
n. ---> System.Configuration.ConfigurationErrorsException: Configuration system
failed to initialize ---> System.Configuration.ConfigurationErrorsException: Dat
a at the root level is invalid. Line 10, position 1. (D:\Drive D\StudyMaterials\
SignalR-Full-Material\SignalR-master\src\Microsoft.AspNet.SignalR.Crank\bin\Debu
g\crank.exe.Config line 10) ---> System.Xml.XmlException: Data at the root level
is invalid. Line 10, position 1.
   at System.Xml.XmlTextReaderImpl.Throw(String res, String arg)
   at System.Xml.XmlTextReaderImpl.ParseRootLevelWhitespace()
   at System.Xml.XmlTextReaderImpl.ParseDocumentContent()
   at System.Configuration.XmlUtil.StrictSkipToNextElement(ExceptionAction actio
n)
   at System.Configuration.BaseConfigurationRecord.ScanSectionsRecursive(XmlUtil
xmlUtil, String parentConfigKey, Boolean inLocation, String locationSubPath, Ov
errideModeSetting overrideMode, Boolean skipInChildApps)
   at System.Configuration.BaseConfigurationRecord.ScanSections(XmlUtil xmlUtil)

   at System.Configuration.BaseConfigurationRecord.InitConfigFromFile()
   --- End of inner exception stack trace ---
   at System.Configuration.ConfigurationSchemaErrors.ThrowIfErrors(Boolean ignor
eLocal)
   at System.Configuration.BaseConfigurationRecord.ThrowIfParseErrors(Configurat
ionSchemaErrors schemaErrors)
   at System.Configuration.ClientConfigurationSystem.EnsureInit(String configKey
)
   --- End of inner exception stack trace ---
   at System.Configuration.ClientConfigurationSystem.EnsureInit(String configKey
)
   at System.Configuration.ClientConfigurationSystem.System.Configuration.Intern
al.IInternalConfigSystem.GetSection(String sectionName)
   at System.Configuration.ConfigurationManager.GetSection(String sectionName)
   at System.Configuration.PrivilegedConfigurationManager.GetSection(String sect
ionName)
   at System.Diagnostics.DiagnosticsConfiguration.Initialize()
   at System.Diagnostics.DiagnosticsConfiguration.get_Sources()
   at System.Diagnostics.TraceSource.Initialize()
   at System.Diagnostics.TraceSource.get_Switch()
   at System.Net.Logging.InitializeLogging()
   at System.Net.ComNetOS..cctor()
   --- End of inner exception stack trace ---
   at System.Net.ServicePointManager..cctor()
   --- End of inner exception stack trace ---
   at System.Net.ServicePointManager.set_InternalConnectionLimit(Int32 value)
   at System.Net.ServicePointManager.set_DefaultConnectionLimit(Int32 value)
   at Microsoft.AspNet.SignalR.Crank.Program.Main(String[] args) in d:\Drive D\S
tudyMaterials\SignalR-Full-Material\SignalR-master\src\Microsoft.AspNet.SignalR.
Crank\Program.cs:line 31
Comment posted by Sundar on Thursday, January 23, 2014 5:35 AM
why I am Getting the above Error? how to resolve??
Comment posted by Sundar on Thursday, January 23, 2014 5:59 AM
why I am Getting the above Error? how to resolve??
Comment posted by Suprotim Agarwal on Thursday, January 23, 2014 6:43 AM
@Sundar: Which version of SignalR are you using. This article was written using v1.0
Comment posted by Jijie Chen on Monday, April 7, 2014 9:41 PM
Could you please share your hardware infomation?
Comment posted by Roger on Monday, July 28, 2014 8:30 AM
Hello, You say "100 connections at 11 messages per second where message size was 32 bytes".
This is pretty low. Did you use the Server Broacast or push mode?
Since then have you got better update rates?
Comment posted by Derrick on Thursday, August 14, 2014 1:13 PM
What is the machine spec for your test machine?  What was the server's RAM and CPU?  What was the bandwidth of the network between server and client?