In a previous article , I had compared Automated Testing Tools Selenium, Coded UI Test and QTP in a broader perspective without going into much details for any of them. In this article, we will do a thorough comparison between two of these tools - Coded UI Test and Unified Functional Testing (QTP).
Quick Test Professional (QTP) was introduced by Mercury which was later acquired by HP as an automated testing tool for testing various software’s and environments. The first version was released way back in 1998 with the name as Astra QuickTest. In 2003, QTP 6.5 was released and now we have a combination of QTP and Service Test as Unified Functional Testing with the current version as 11.5. Note: Service Test is used to test Web services.
Coded UI Test (CUIT) is comparatively a new kid on the block. It was made available as a part of Visual Studio 2010 and since then a lot of enhancements have been made till the current version with Visual Studio 2013. Even though Visual Studio provides other automated test types like Unit Tests, Web Performance Test, Load Test; we are not getting into the details of any one of them other than CUIT. Note: Web Performance Test can be used to test Web Services.
This article is published from the DotNetCurry .NET Magazine – A Free High Quality Digital Magazine for .NET professionals published once every two months. Subscribe to this eMagazine for Free and get access to hundreds of free tutorials from experts
Setting up the Sample
While doing the comparison, we are going to consider an example of an application that I developed as part of the Employee Management System. This application has two interfaces, one for the desktop (Windows) and the other for the web. With the Windows IDE, the HR personnel can login and enter or edit details of Employees. With the web interface, Employees can find their own information and maintain their passwords.
I am including some steps to start the desktop version of the app. You can download a zip file named “SSGS EMS App” from the source code mentioned at the end of this article.
Follow these steps to run the application which will help to reproduce the screenshots I have included in this article.
1. Attach the database files to SQL Server by copying it in the folder Program Files\Microsoft SQL Server\MSSQL11.MSSQLSERVER\MSSQL\DATA. Your SQL Server version may be different, so choose the folder accordingly.
2. Double click on “SSGS EMS WinForm App.exe” to start Windows Application
3. The user name and password for the HR Personnel is “Shalaka” and “Shalaka”. Click on the Login button.
4. Enter Employee Id as 2 in the next screen and click on Get Basic Data, followed by Get Extended Data to get following details
5. The Reset button will clear all textboxes and you can enter any other Employee code.
6. In order to enter a new Employee, enter all details except employee id and click on Add New. As you may have observed, I have not created any validation as this application is solely created to help you use the two testing tools we will be discussing shortly.
Note: Employee Id will be automatically generated. I have used the application only up to the first 4 steps in the article.
Manual testing is a slow process as human factor is involved in it, for execution as well as determining whether the test case passes or fails. Manual testing requires more testers to execute, but less skill. Automated testing is less time consuming but requires more resources for creating it. With Automation testing, the tool helps in execution and the assertion can be checked programmatically. When a test is needed to be executed in recurrence for checking quality of build or for regression, it is more productive to run it as an automated test rather than running it manually every time.
Record and Playback
UFT as well as CUIT support record and playback feature. Application under test is executed by the tester and every action of the tester as well as changes in the UI objects, is recorded by the testing framework. This recording is then played-back number of times to ensure that every time for the same action, the change is the same in the UI objects. The actions performed are recorded with a tool built into the testing framework and the playback is done by another tool from the same framework to execute the automated tests.
With UFT, we can use the Record option, while CUIT provides Coded UI Test Builder to do the same. The settings for Windows application or Web application can be set using UFT’s record and run settings. There are 4 icons with CUIT Builder as record, show steps, add assertions and generate code. The code will be automatically generated in high level languages like C# or VB.NET. There is another option for creating CUIT by using the already recorded actions while doing manual testing. Usually any test is first run as a manual test and when we realize that it now needs to be re-executed quite a number of times, say as a regression test, it is then automated. In Microsoft Test Manager (MTM), it is possible to record the actions of testers when they are executing a test. While creating CUIT, we can select an option to use that recording, for generating the test code.
I personally like the feature of MTM for converting manual actions to its code equivalent. It reduces the efforts required by the automated test developer to record the test. Other than that, Record and playback feature is equally easy to use in both the tools.
Languages used with UFT and CUIT
The first and foremost difference between these two Automated Testing Tools is the underlying code that is generated or written. While UFT uses a scripting language (VBScript), CUIT is supported by high level language like C# or VB.NET. UFT also provides an additional option of descriptive programming. As we all know VBScript supports classes but does not provide polymorphism and inheritance, which is the basis of object oriented technology. VBScript also does not have inherent event handlers.
Another difference is the way the test code can be debugged. VBScript code cannot be debugged as such. Though HP added a debugger in its package of UFT, it comes with very limited functionality as compared to integrated IDE, which we have with CUIT as a part of Visual Studio. Visual Studio also provides excellent IntelliSense which makes code writing much faster. The IntelliSense with UFT is a minimal feature.
I like the high level language support provided by CUIT. In my experience, hard core testers prefer VBScript as the efforts for them to learn the language like C# or VB.NET are considerably higher, but those who have little knowledge of coding in object oriented languages, appreciate CUIT more. Support of object oriented programming clearly gives more flexibility although it is more difficult if not impossible to write such a code from scratch,h which many testers have got accustomed to do, rather than using a recorder.
UFT stores all application related objects in Object Repository and modifying it is quite simple. Once the object repository is ready, it can be exported to XML. We can even see the objects getting stored while recording. The object repository can either be local or shared. As the name suggests, shared repository can be shared amongst multiple tests if the tests have properties which can be re-used. Shared repository can also be edited if required. UFT provides a feature called as object spy which helps to list runtime properties of GUI and test objects. It also lists the methods which can be used in implementing the particular object.
Visual Studio stores the application related objects in its own repository called UIMap. The CUIT Editor is provided in Visual Studio 2010 Feature Pack 2 onwards, which edits UI Map. You can select any object and view its properties. You can change some of the search properties or expressions. You can also expand recorded action and view the steps. It is possible to remove the steps; split a bigger method into another but you cannot add a step to an action. You can use CUIT Builder to record all previous steps and your new step and also overwrite the changes in action. These actions are re-usable within the project. For re-using repository across the projects, you can set a reference to the assembly from which you want to use the code. The default UIMap becomes very heavy if we add a number of large methods to it. In order to overcome this, we can add non-default UIMap. If we add another UIMap, we need to take care of calling the methods after we have recorded the actions with CUIT Builder.
Object spy is a great feature provided by UFT. This clearly shows the stability of the tool.
The following two images show the object repository with both the tools:
Integration with Test Management Tool
Test Management Tool provides a container for storing information about testing to be done, planning of various artifacts and helps in providing reports for quality. The tool can provide different configurations for testing. The tool can be used to execute manual test cases, provide data in terms of parameters and gather information about the result. It can also be used to run automated tests, manage different environments under which tests need to be executed etc. This works as a collaborative platform for communications within teams. The defects can be filed and the traceability from defect/ bug to test case to requirement can be established.
UFT can be connected with HP ALM (Application Lifecycle Management) which is an enhanced version of Quality Centre (QC) with a complete application lifecycle support. HP ALM provides option for converting Test Case to script (provided you have HP ALM and UFT installed). Once the script is generated you can open it with UFT.
CUIT can be connected with Team Foundation Server, a product provided for application lifecycle management. Microsoft Test Manager is a tool for test case management. Using this tool, any test case can be recorded and then converted to its code (CUIT) by using Visual Studio. Visual Studio provides us with automated build and CUIT can be included as a Build Verification Test. We can use distributed testing by using Test Controller and Test Agent.
Visual Studio can be integrated seamlessly with Team Foundation Server with the help of Team Explorer.
Checkpoint or assertion is used to find out if the Application Under Test (AUT) is providing a particular value or a property. The value can be asserted against expected value to decide whether to pass or fail the test. These assertions or checkpoints can be added while recording the application or it can be added after the recording is completed.
Once a checkpoint is added in UFT, it gets added to the current row in the keyboard view. There are various types of checkpoints like - standard, bitmap, image, text, database and table.
Standard checkpoint can be added in a keyboard view, expert view or using active screen. It checks objects like radio buttons, tables, combo boxes, links etc. The Bitmap checkpoint is used in an application or a web page to check its area. It checks the area pixel by pixel, either partially or completely. The Text checkpoint checks to see if the expected text matches. The Database checkpoint creates a query and the values are verified against the database values stored as expected values.
Normally checkpoints are called as assertion in Coded UI Test. Assertions can be added using Coded UI Test Builder. The cross hair for assertion can be followed by the property to be selected for assertion.
In the UI Control Map, we can add control, delete a control or rename a control. Once the assertion is added, we can set the comparison operator and the value to compare. We can create assertion by directly adding a method for it in the partial class UIMap which is in file UIMap.cs.
UFT provides more types of checkpoints when compared to CUIT.
Integration with Source Control
Application Lifecycle Intelligence (ALI) is a technology which is embedded in HP ALM which helps in providing complete ALM traceability. ALI provides support to Source Control and Build Management. Since I was working with the evaluation version, this feature could not be tried; however those having a licensed version of the product can test this feature.
Coded UI Test has complete traceability with ALM features of Team Foundation Server (TFS) as it totally integrates with it. We can convert a recorded test case from Microsoft Test Manager to CUIT, for which the recording is stored in TFS. We also have the source control service of TFS at our disposal.
As I could not try the support with UFT, I prefer CUIT in this scenario, although my choice here is biased.
Here’s a quick recap of a comparison of CUIT with UFT (QTP)
I hope you enjoyed going through these feature comparisons of both these tools. UFT clearly is the market leader and has been there since a long time. Coded UI Test (CUIT) is being made feature rich with each new version of Visual Studio, thus making it a great competitor tool, that I look forward to!
Download the app used in this article (Github)