DotNetCurry Logo

Comparison of Automated Testing Tools: Coded UI Test, Selenium and QTP

Posted by: Gouri Sohoni , on 4/28/2014, in Category Product Articles
Views: 190430
Abstract: This article evaluates multiple Test Automation tools viz. Selenium, QTP and CUIT and does a side by side comparison of the features to give us a high level picture of the capabilities of these suites.

Software Testing is a way to validate and verify the working of a particular product or application. It can be incorporated at various points of time in the development process depending upon the methodology and tools used. Testing usually starts after the crystallization of requirements. At a unit level, it starts concurrently with coding; whereas at an integration level, when coding is completed. Testing is used for finding out the bugs in our application. It helps in finding out the failure of the software before it crashes the application. The purpose is to satisfy the stakeholders and ensure the quality of an application with testing.

This article is published from the DotNetCurry .NET Magazine – A Free High Quality Digital Magazine for .NET professionals published once every two months. Subscribe to this eMagazine for Free and get access to hundreds of free tutorials from experts

 

Manual and Automated Testing

There are mainly two broad types of software testing - Manual Testing and Automated Testing.

manual-automated-testing

Manual Testing

Manual Testing involves testing the software without any automation script or any tool. Testers check the application or software by taking up the role of an end user. They try to find out if there is any unexpected behavior or failure in the application. Test Management can be taken care of by using test plans and test cases.

Automation testing

Automation testing process involves testing with the help of automation scripts and executing the scripts to run the application with the help of some automation tool. Once the script is ready then these tests can run quickly and efficiently.

Since the cost of automated testing is in the form of efforts and time required to create the scripts, not all tests can be converted to automated test. There should be a valid reason to pay that cost.

Reasons for Automation

1. Regression testing to confirm that new changes have not affected the application adversely. It considers already existing test cases for execution. This is an efficient process when we need to provide feedback to the developer immediately.

2. The test cases need to be iterated multiple number of times often with varying datasets to cover multiple workflow paths.

3. When we require support for agile methodologies.

4. Customized reports are required for monitoring.

Getting Started with Automated Testing

Once the need for automated testing has been established, it involves creation of relevant test scripts. Test script creation can be done only by a skilled testers having knowledge of testing, the suite of tools as well functionality under development. Such resources are costly and their time is a premium. Considering this fact, it is often not possible to budget the automation of all tests. Some of the major decision points while identifying cases for testing automation are

1. System modules where requirements do not change frequently

2. Ample time is at hand to describe a test via scripts

3. The application/software module is critical enough to justify the upfront cost of automation

4. After functional testing we want to do performance testing with multiple virtual users using the same test script.

With the scope of automation decided, next step is to pick the testing tool. The following checklist can help with the selection.

1. The tool should be able to easy to work with. It should execute test cases in unattended manner. It should provide interface to write scripts, efficient IDE and ease of test execution.

2. The tool should provide support to various technologies. It should support testing using different browsers, languages, and types of applications.

3. It should integrate with a software that does Application Lifecycle Management so that it can be used for running automated Build Verification Tests as well as the reports can be integrated with other reports created by ALM software.

 

testing-automation-process

Automated testing frameworks today

Today we will compare three automated testing tools namely Selenium, QTP (Quality Test Professional) and Coded UI Test (CUIT) with Visual Studio 2012.

We will consider above mentioned main aspects of automation and see how these tools provide the support for each category.

General information about these tools

Selenium was created by Jason Huggins. It is an open source testing tool. Later Simon Stewart started WebDriver (to overcome some limitations of Selenium). Both the tools are now merged to get one awesome testing tool. HP QTP (Quick Test Professional) was originally written by Mercury Interactive. It is a part of HP Quality Centre Suite (QC). Coded UI Test (CUIT) was introduced by Microsoft along with Visual Studio 2010. It integrates with Team Foundation Server.

Ease of Use

- Recording and Playback Functionality
Each of the testing tools has the ability of recording the actions and playback the recorded actions. Selenium provides the plug-in named Selenium IDE with Mozilla Firefox with which the actions can be recorded. QTP provides record button to record a new test. Recording for CUIT can be done using two different tools. Microsoft Test Manager can be used to record actions that can be then converted to CUIT. The fast forward playback is available to run the test case in a semi-automatic mode even before converting to script. With Visual Studio CUIT provides Coded UI Test Builder to record the actions. In my opinion all the 3 tools are very easy for record and playback.

- IDE and tools with which the tester can write the scripts
With Selenium IDE there is no special tool and specific technology to write the script. We can insert commands with ‘Table or Source View’ when required.

QTP provides Keyword View to display test steps graphically or Expert View which shows VB Script lines.

For CUIT we can easily use Visual Studio IDE to write scripts.
 

Selenium IDE comes as a plug-in with Mozilla Firefox. With this we can create a test suite which comprise of various test cases. With Selenium IDE is, you can convert recorded Selenium IDE scripts into different languages and after conversion you can run it in Selenium RC. Selenium RC has two components, one is "Selenium Server" and another is "Selenium Client".

 

With QTP IDE for the first time 3 add-ins are provided ActiveX, Visual Basic and Web. Various links to best practices, new features for the current version are available with start page. We can either open existing test case or create a new one.

 

For CUIT we have a very elaborative IDE as the recording can be done using Visual Studio. All the features of Visual Studio are applicable. The script writing support with Visual Studio is an excellent way of writing and debugging. In my opinion CUIT scores more points in this area.

 

- Ease of Test Case execution
With Selenium IDE we have the option of executing the entire test suite already recorded or a test case at a time.

 

Depending upon the add-ins loaded in QTP IDE the record and run window shows tabs. Windows Application tab is always available. The tests can be executed with run button which in turn opens the run dialog box. We can specify the location for run specific results and provide parameters if any.

 

CUIT can either be executed with Visual Studio or by using Microsoft Test Manager (MTM). With MTM we can provide various settings for the test case execution so as to gather a lot of information while executing the test case behind the scene. Test Execution can be done is more or less in a very straight manner with the 3 tools. CUIT provides various test settings to execute test cases so as to capture different data when we need to create a bug (commonly called as rich bug).

 

All the 3 tools can execute test cases without human intervention.

Platform Support

-Language Support
Selenium uses Selenese, a high-level, cross platform language to write Selenium commands which is a domain specific language. There are 3 basic categories for the commands - named actions, accessors and assertions. To write tests there are a lot of programming languages like C#, Java, Perl, PHP, Python or Ruby.

QTP scripts can be written with VBScript which is a high-level language with support to everything except polymorphism and inheritance.

For CUIT we can write the script with Visual Studio with which we can use all object programming concepts if required. Writing VBScript code is very easy from tester’s perspective. Even though CUIT supports object oriented programming, the testers may not prefer as it involves a lot of skill for writing or customizing the created script.

- Support for various application types
Selenium supports only Web applications.

QTP supports almost any kind of applications.

CUIT supports Windows Applications, Web Application, WPF applications, SharePoint, Office Client Applications Dynamics CRM Web Client applications.
 

Selenium scores fewer points in this regard as it supports only web application. QTP supports almost all kinds of applications as against CUIT.

- Support for various browsers
Selenium supports all versions of IE, Firefox, Safari and Opera and a few more browsers.

QTP supports IE & Firefox. But both do not provide full cross browser support.

CUIT supports only IE8, IE 9 and IE 10 (IE 10 supported only on desktop). There is no support to IE6, IE7, Chrome, Opera or Safari.

 

Selenium is the clear winner in this respect

- Support for Data Driven Testing
Selenium IDE supports xml data source using user extensions.

Data Driven testing is implemented as Excel workbook that can be accessed by QTP. There are 2 types of data sheet global and local. Global sheet is a single one which can be accessed from every action in a test. There can even be a local data sheet associated with every action.

Coded UI Test supports any data source supported by .NET framework which can come in the form of a .CSV file, XML file or any other data source like SQL Server table, Access table etc.

 

In my opinion CUIT provides better ways of data driven testing.

- Exception Handling
Selenium IDE does not support error handling particularly unexpected errors (as it supports only HTML language). Selenium RC will provide support for it (it supports languages with .NET, Java, Perl, Python, PHP, Ruby).

QTP provides VBScript with the help of which we can use On Error statements. As CUIT supports high level languages like C# or VB.Net we can use try catch construct here. In my opinion all the 3 tools have their limitations here.

With CUIT it is suggested to capture the base exception and write code accordingly.

- Validations or Assertions
Selenium assertions can be used in 3 modes assert, verify and waitFor. When an "assert" fails, the test is aborted. When a "verify" (Soft Assertions) fails, the test will continue execution, logging the failure. This facility can be used with TestNg framework. The "waitFor" commands wait for some condition to become true. They will succeed immediately if the condition is already true. However, they will fail and halt the test if the condition does not become true within the current timeout period.

For QTP there are checkpoints: to verify application under test. These are of 10 types – Standard, Table, Image, Bitmap, Database, Text, Text Area, Page, Accessibility, and XML. A checkpoint is a verification point that compares the current value with the expected value. If the current and expected value match it generates a PASS status otherwise FAIL status.
We can use Coded UI Test Builder to add assertions for your UI controls. We need to edit the assertion condition as required (equal to, in between, contains etc.), provide expected value and generate code for it. Mouse hover events can be recorded manually if required.
 

In my opinion QTP and CUIT are better tools to be used in this aspect.

- Support for Objects
Object properties are not supported by Selenium. Selenium objects can be managed by using UI element user extensions. QTP comes with in-built object repository.

QTP objects have user friendly names.

Coded UI Test code is written with 3 main parts for UI controls. UIMap.designer.cs, UIMap.cs and UIMap.uitest. The first 2 are different physical files for same partial class while the third is a XML equivalent of all the actions recorded with CUIT Builder. Any changes required can be incorporated with the help of the partial class file. We can also edit the UIMap with the help of Coded UI Editor and find out object’s properties. The CUIT can be completely hand coded if required. Coded UI Test includes a rich API library to code against and a resilient record and playback tool. Coded UI Test can be extended to support custom controls.

CUIT is the clear winner in this regard.

Integration with Application Lifecycle Management and going beyond

- ALM Integration
Selenium being an Open Source software can be integrated with other Open Source products for Application Lifecycle Management like QMetry. This in turn can provide platform for software development lifecycle platform in the form of Atlassian Jira (project tracking tool), FogBugz or Bugzilla (bug tracking tool).

QTP being a part of Quality Centre it supports requirement traceability matrix. QTP integrates seamlessly with QC. Test management and mapping the manual testing process with automation becomes a lot easier with this integration

For CUIT and MTM we can provide all the ALM support Team Foundation Server provides. It supports work item tracking, source control or version control, build automation, various reports. The support is in-built; we do not have to do anything extra.

QC is still not complete life cycle management tool. It does not provide support for efforts management, build management or support to different process templates. It supports only test management, bug management and requirement management.

CUIT is a winner here as it seamlessly integrates with Team Foundation Server (TFS). TFS in turn supports work item tracking, source or version control, requirements management, project management, build automation and various reports for monitoring.

· Monitor with Customized Reports
Test results can be made available with each tool. Coded UI Test supports all the reports supported by Team foundation Server as well as has the option of creating any custom report. The custom report can be created in any of the 3 ways, by using Report Project with Business Intelligence Development studio (BIDs), by using Microsoft Excel or by using Report Builder facility to create reports on the fly.

- Going beyond
Selenium being Open Source a lot of plugins available. Selenium IDE has plug-ins for customization, for adding new functionality to API, changing existing functionality.

QTP provided plug-ins for ActiveX controls, web application and VB objects. Other than these plug-ins for other objects like Microsoft .NET, multimedia plug-ins and Windows Mobile are also available. These QTP plugins available at an additional cost.

Apart from hand coding complete CUIT there is another feature available. CUITe Coded UI Test enhanced is a thin layer developed on top of Microsoft’s Coded UI engine which helps reduce the code. It also increases readability and maintainability. It is very easy to install and will be referred with CUITe.dll in the project. CUITe provides simple object repository.
Each of the tools keeps on adding features as per need. Selenium being open source there are a lot of plug-ins.
 

I have evaluated these tools from different angles and each has got its strength and weaknesses. You may choose a tool depending upon your need and the support the tool provides.

Summarizing the Comparison

The following table provides a bird’s eye view for the categories and the tools’ support for it.

comparison

Legend

 

 

legend

Conclusion

To conclude, we did a quick overview of what is automation testing and when it a good time is to start thinking about test automation in a software development cycle.

We looked at three popular automation tools, Selenium, CUIT and QTP and gauged their strengths and weaknesses. Final selection of tool is almost always based on budgeting and team strengths (tool familiarity), however for Web Application testing all three have compelling strengths. For desktop application testing the choice gets reduced to two with Selenium dropping out.

Editorial Note: Abhishek from Telerik extended this article to add Telerik Test Studio to the mix. Read his article at http://telerikhelper.net/2014/07/10/comparison-of-automated-testing-tools-throwing-in-telerik-test-studio/

Was this article worth reading? Share it with fellow developers too. Thanks!
Share on LinkedIn
Share on Google+
Further Reading - Articles You May Like!
Author
Gouri Sohoni is a Trainer and Consultant for over two decades. She specializes in Visual Studio - Application Lifecycle Management (ALM) and Team Foundation Server (TFS). She is a Microsoft MVP in VS ALM, MCSD (VS ALM) and has conducted several corporate trainings and consulting assignments. She has also created various products that extend the capability of Team Foundation Server.


Page copy protected against web site content infringement 	by Copyscape




Feedback - Leave us some adulation, criticism and everything in between!
Comment posted by Martin Hinshelwood on Tuesday, April 29, 2014 7:50 AM
Good article.... just a few notes!

1) CUIT supports Chrome and Firefox (Selenuin still wins but not as well)
2) Selenium can be fully integrated into TFS easily

Other than that this is awesome :)
Comment posted by Gouri Sohoni on Wednesday, April 30, 2014 10:23 PM
Thanks Martin,

Your comments are very valuable.
Comment posted by Dan Porat on Wednesday, May 14, 2014 5:33 AM
Great article.
I love it.
Just wanted to remind Applitools as a great complementary tools to the above.
Be it with Selenium , QTP or coded UI - Applitools helps you do the visual comparion an regression.
Sure saved us a lot of time during development.

Dan
Comment posted by Shabeer Qadri on Tuesday, June 24, 2014 4:43 PM
Very good article. Would like to talk to you about Coded UI training. Could you please send me your contact info?
Comment posted by Prasanna on Wednesday, June 25, 2014 9:19 AM
Great article. We are able to seen a clear view as we are trying with CUIT. Thank you.

In future expect many more of such.
Comment posted by Abhishek Kant on Thursday, July 10, 2014 2:42 AM
Great article. I have extended this article by adding Telerik Test Studio capabilities in the mix as well. It is interesting to compare what "Visionary" tool (as awarded by Gartner magic quadrant) like Test Studio has to offer in comparison to market leaders: http://telerikhelper.net/2014/07/10/comparison-of-automated-testing-tools-throwing-in-telerik-test-studio/
Comment posted by Suprotim Agarwal on Monday, July 14, 2014 9:23 AM
Thanks Abhishek, updated and added the link to the article.
Comment posted by Waqas Bajwa on Thursday, July 17, 2014 9:24 AM
Excellent article. I recently started learning Coded UI Tests. As a Microsoft user, I would recommend this to all those who develop or test .NET apps.
Comment posted by Nishant on Thursday, July 24, 2014 8:57 AM
Hi,
QTP support objects.
Person who is automating can use VBscript to create object. QTP supports OOPS concepts.In case you need more information feel free to email me.
Comment posted by suresh on Sunday, August 17, 2014 11:22 AM
As I have worked with QTP, Selenium and CUIT as well, I can say that, some are completely wrong judgements (obviously this is biased towards VSTS):

1. Objects not supported - QTP and Selenium.

Question - What are objects?

String str= new String() // this is Java object
--------
This is supported by Selenium.
QTP also supports.


2. Ease of IDE (with IDE).
QTP takes the number one position - Next maybe, VSTS and Eclipse both.

3. Integration with ALM - QTP completely can be integrated into QC. Ths scripts are run from ALM/QC.

Comment posted by Nishant on Thursday, October 9, 2014 3:30 AM
Hello
Descriptive information. Can you please let me know if there is any open source available for testing Form based applications like we have selenium for web based applications.
Thanks
Comment posted by Arvind on Friday, November 14, 2014 6:22 AM
This is an extremely Coded UI biased article . Does not add any value
Comment posted by Rajesh on Monday, March 2, 2015 6:29 AM
Hello,
In our mvc application, we have C#, Knockoutjs and Silverlight coding. We want automate our test cases, is CUIT suites for our application? Please help,

Thanks.
Comment posted by Rajesh on Monday, March 2, 2015 6:31 AM
Hello,
In our mvc application, we have C#, Knockoutjs and Silverlight coding. We want automate our test cases, is CUIT suites for our application? Please help,

Thanks.
Comment posted by H20Dragon on Thursday, March 5, 2015 12:25 PM
HP UFT/QTP supports VBScript as their language of choice, however the commentary in regards to OOP per VBScript is odd.  VBScript doesn't provide you the ability of class inheritance and polymorphism.  Futher, it doesn't have the ability to provide namespaces.  These are the basics for writing OOD/P programs.

The other tools that were mentioned, such as Selenium, Telerik Test Studio and CodedUI all leverage a programming language that provides OOD/P.  A good software programmer would be able to extend the automation tool's technology to improve the framework.  Unfortunately, HP's choice of VBScript minimizes the ability to extend the tool .. too many examples on why SDET's (e.g. not QA functional testers), with programming experience, dislike HP's UFT/QTP.
Comment posted by Testomato on Thursday, March 19, 2015 4:39 AM
Thanks for a great info on testing tools, for automated website testing we use Testomato from http://www.testomato.com Would be great to hear thoughts when you have time.
Comment posted by Diti Joshi on Thursday, March 26, 2015 2:29 AM
Great article.
I dote it.
Just wanted to remind TestingWhiz as a great implements to the above.
Be it with Selenium , QTP or coded UI - TestingWhiz avails you do Automated Database testing as well.
Preserving lot of time while testing is the advantage as it is partially code-less as well. Do have a visual examination of TestingWhiz @ http://www.testing-whiz.com/solutions
Comment posted by Irina on Thursday, April 30, 2015 3:57 AM
Thank Gouri for a great article, would be great to hear your opinion on Testomato automated testing tool from http://www.testomato.com - is it possible to hear your thoughts on it?
Comment posted by Shripad on Friday, May 22, 2015 6:42 AM
How selenium is integrated with MTM 2013?