Smack down!! WatiN vs Selenium vs VS2010 Test Lab Manager

*Updated* 21 March 2011: Updated how "Coded UI tests" finds a control with info from Balagans Blog - How does “Coded UI test” finds a control? 


We need Web functionality tests!!! (and a lot of people are already doing them)

  1. Easier to find bugs
  2. No need for a lot of manual testing
  3. Maintainability of your code (THIS IS BIG!!)
    1. When we change something (Code, DB, scripts, HTML,...) we want to verify if our user stories working as expected
    2. We setup web tests for each user story that we have.
  4. We can use web tests in load tests or stress tests
  5. … lots more

Note from Wikipedia
Stress testing: Simulates increased load, more than expected
Load testing: End to end performance testing under anticipated load


I tried the following 3 players (recommendations from work mates, and others..)

  1. Selenium
  2. WatiN
  3. Visual Studio 2010 & the Test & Lab Manager

All of these 3 tools provide a recording tool, where you can record your browsing actions and generate code from that.
This is important!!!
No one wants to write these tests manually!


Do the test recording

apcheck_thumb1 Selenium Test recorder just works.

Record your session. Copy & paste to Visual Studio. Done

ap--delete Watin Test recorder v2 throw exception

Bug reported here:

ap--delete Watin Test recorder V1 generates unsupported code -> manual fixing needed :-(


Code output of recorder

selenium.Open("/Public/default.aspx"); selenium.Click("//div[@id='userlogin']/div/div[2]"); selenium.Click("//div[@id='userstatusLoggedOut']/div/a/span"); selenium.WaitForPageToLoad("30000");
selenium.Type("ctl10_MainContentArea_ctl00_ctl00_CreateUserForm_CreateUserStepContainer_FirstName", "o"); selenium.Type("ctl10_MainContentArea_ctl00_ctl00_CreateUserForm_CreateUserStepContainer_LastName", "g"); selenium.Type("ctl10_MainContentArea_ctl00_ctl00_CreateUserForm_CreateUserStepContainer_Email", ""); selenium.Type("ctl10_MainContentArea_ctl00_ctl00_CreateUserForm_CreateUserStepContainer_ConfirmEmail", ""); selenium.Type("ctl10_MainContentArea_ctl00_ctl00_CreateUserForm_CreateUserStepContainer_Password", "password!"); selenium.Type("ctl10_MainContentArea_ctl00_ctl00_CreateUserForm_CreateUserStepContainer_ConfirmPassword", "password!"); selenium.Click("ctl10_MainContentArea_ctl00_ctl00_CreateUserForm___CustomNav0_StepNextButtonButton"); selenium.WaitForPageToLoad("30000");
selenium.Click("link=Log Out"); selenium.WaitForPageToLoad("30000");

Figure: Simulating a user registration with Selenium


IE ie = new IE("about:blank"); ie.GoTo(""); ie.Div(Find.ByCustom("innertext", "Member Login")).Click(); ie.Link(Find.ByUrl("")).Click();
ie.TextField(Find.ByName("ctl10$MainContentArea$ctl00$ctl00$CreateUserForm$CreateUserStepContainer$FirstName")).TypeText("Peter"); ie.TextField(Find.ByName("ctl10$MainContentArea$ctl00$ctl00$CreateUserForm$CreateUserStepContainer$LastName")).TypeText("Gfader"); ie.TextField(Find.ByName("ctl10$MainContentArea$ctl00$ctl00$CreateUserForm$CreateUserStepContainer$Email")).TypeText(""); ie.TextField(Find.ByName("ctl10$MainContentArea$ctl00$ctl00$CreateUserForm$CreateUserStepContainer$Password")).TypeText(""); ie.TextField(Find.ByName("ctl10$MainContentArea$ctl00$ctl00$CreateUserForm$CreateUserStepContainer$Password")).TypeText("passmenumenuqword!"); ie.TextField(Find.ByName("ctl10$MainContentArea$ctl00$ctl00$CreateUserForm$CreateUserStepContainer$ConfirmPassword")).TypeText("passmenuqword!"); ie.Button(Find.ByName("ctl10$MainContentArea$ctl00$ctl00$CreateUserForm$__CustomNav0$StepNextButtonButton")).Click();

Figure: Simulating a user registration with WatiN


apcheck_thumb1 Selenium is a tool to create tests for web apps

apcheck_thumb1 Selenium’s output is a whole test class

apcheck_thumb1 Selenium’s output code is using relative URL’s (that you define in your test base URL)

apcheck_thumb1 Selenium code: Easy to read

ap--delete WatiN: The code output from the test recorder v1 is quite complicated to read.

I prefer Seleniums: “selenium.Click” instead of the long string and then seeing the “Click()”

// WatiN
watin.TextField(Find.ByName("ctl10$MainContentArea$ctl00$ctl00$CreateUserForm$CreateUserStepContainer$ConfirmPassword")).TypeText("password, f17");

// Selenium
selenium.Type("ctl10_MainContentArea_ctl00_ctl00_CreateUserForm_CreateUserStepContainer_ConfirmPassword", "password");

Figure: Code comparison of WatiN with Selenium. Enter text in a password field and click on button

ap--delete WatiN : Output code from Watin Test recorder v1 was wrong.

Some input control TypeText() was missing
Finding of elements didn't always work E.g.  <a href="xx"> <span> My funkty test </span> </a>  


Running the tests

ap--delete Selenium needs a JAVA server to execute the tests

apcheck_thumb1 WatiN doesn't need a proxy server!

ap--delete Selenium is quite slow to run

Output from the Selenium Java server (see the 2 seconds of start up gap)

19:11:54.341 INFO - Command request: getNewBrowserSession[*chrome,, ] on session null
19:11:54.341 INFO - creating new remote session
19:11:54.341 INFO - Allocated session 69581bdd207a4684b61f6c368af906f0 for, launching...
19:11:54.370 INFO - Preparing Firefox profile...
19:11:56.895 INFO - Launching Firefox...


Code maintainance

apcheck_thumb1 Selenium makes working with Firefox perfectly together: uses the same name as in HTML

E.g. ctl10_MainContentArea_ctl00_ctl00_SubTotal

That means you can copy & paste from FireBug directly to Visual Studio

string priceSubTotalOnCheckout = selenium.GetText("ctl10_MainContentArea_ctl00_ctl00_SubTotal");

ap--delete Watin replaces the underscores with a dollar sign (This is annoying when copy paste from FireBug)

    needs to be converted to

string resultText = browser.Element(Find.ByName("ctl10$MainContentArea$ctl00$ctl00$SubTotal")).Text;


Figure: Selenium is coming from the Java world, as you can see this here. No properties :-) --> GetXXX


What about  Visual Studio 2010 and Test & Lab Manager

  1. In Test & Lab Manager you define a test case that links to a user story
    1. User story: As a customer I want to search and buy a bag so that I satisfy my needs for a bag
    2. Test case below
  2. Now you run you test with an Action recording
    1. This is easy. Just follow the instructions from the test case
    2. Done
  3. Close down Test Lab Manager and open Visual Studio, add "Coded UI test" with an "Existing action recording"
  4. Code that you get

[DataSource("Microsoft.VisualStudio.TestTools.DataSource.TestCase", "http://win-peterG:8080/tfs/DefaultCollection;Peter-Gfader-WebTest", "140", DataAccessMethod.Sequential), TestMethod] public void CodedUITestMethod1() { this.UIMap.Openbrowser(); this.UIMap.Gotostartpage();
this.UIMap.ClickonRegisterNew(); this.UIMap.EnterdetailsfirstnamelastnameemailpasswordParams.Ctl10MainContentAreaEditText = TestContext.DataRow["firstname"].ToString(); this.UIMap.EnterdetailsfirstnamelastnameemailpasswordParams.Ctl10MainContentAreaEdit1Text = TestContext.DataRow["lastname"].ToString(); this.UIMap.EnterdetailsfirstnamelastnameemailpasswordParams.Ctl10MainContentAreaEdit2Text1 = TestContext.DataRow["email"].ToString(); this.UIMap.EnterdetailsfirstnamelastnameemailpasswordParams.Ctl10MainContentAreaEdit3Text1 = TestContext.DataRow["email"].ToString(); this.UIMap.EnterdetailsfirstnamelastnameemailpasswordParams.Ctl10MainContentAreaEdit4Text = TestContext.DataRow["password"].ToString(); this.UIMap.EnterdetailsfirstnamelastnameemailpasswordParams.Ctl10MainContentAreaEdit5Text = TestContext.DataRow["password"].ToString(); this.UIMap.Enterdetailsfirstnamelastnameemailpassword(); this.UIMap.ClickonRegister();
this.UIMap.Searchforbag(); this.UIMap.Add1stbagfoundtoshoppingcart();
this.UIMap.EnterAddressdetails(); this.UIMap.Placeorder(); }

Figure: Code generated by VS2010 for a registration of a new user, searching for a bag, adding this bad and going through the checkout

apcheck_thumb1 VS2010 code is very easy to read. And there are a lot of things happening behind the scenes

UIMap is a huge file generated for you that does all the magic behind the scenes, like finding controls, clicking buttons, …

UIMap: Strongly typed representation of controls


How is Visual Studio finding controls?

        public HtmlInputButton RegisterButton
                if ((this.mRegisterButton == null))
                    this.mRegisterButton = new HtmlInputButton(this);

                    #region Search Criteria

                    this.mRegisterButton.SearchProperties[HtmlProperties.Button.Id] = "ctl10_MainContentArea_ctl00_ctl00_CreateUserForm___CustomNav0_StepNextButtonButton";
                    this.mRegisterButton.SearchProperties[HtmlProperties.Button.Name] = "ctl10$MainContentArea$ctl00$ctl00$CreateUserForm$__CustomNav0$StepNextButtonButton";
                    this.mRegisterButton.FilterProperties[HtmlProperties.Button.DisplayText] = "Register";
                    this.mRegisterButton.FilterProperties[HtmlProperties.Button.Type] = "submit";
                    this.mRegisterButton.FilterProperties[HtmlProperties.Button.Title] = null;
                    this.mRegisterButton.FilterProperties[HtmlProperties.Button.Class] = null;
                    this.mRegisterButton.FilterProperties[HtmlProperties.Button.ControlDefinition] = "id=ctl10_MainContentArea_ctl00_ctl00_Cre";
                    this.mRegisterButton.FilterProperties[HtmlProperties.Button.TagInstance] = "40";
                    this.mRegisterButton.WindowTitles.Add("Peter Gfader | Sign Up - Windows Internet Explorer");


                return this.mRegisterButton;

Figure: VS 2010 Beta2 generated code for finding a button on a web page. I think this is way TOO STRICT

ap--delete VS2010: All the "SearchProperties"above have to match, otherwise the button won’t be found. If too many controls are found from the "SearchProperties", the "FilterProperties" are applied one by one. All details on Balagans Blog - How does “Coded UI test” finds a control?

ap--delete There is no option to customize the default generated SearchProperties


I would probably change that to

        public HtmlInputButton RegisterButton
                if ((this.mRegisterButton == null))
                    this.mRegisterButton = new HtmlInputButton(this);
                    #region Search Criteria

                    this.mRegisterButton.FilterProperties[HtmlProperties.Button.DisplayText] = "Register";
                    this.mRegisterButton.FilterProperties[HtmlProperties.Button.Type] = "submit";

                return this.mRegisterButton;

apcheck_thumb1 Better way to search for the Register button. Filter on DisplayText and type of the control.
Note: This would fail if there are 2 buttons with "Register" and of type "Submit" on the page. Which is excellent!


Can VS2010 simulate a FireFox?
See this image for details about the VS2010 platform support
Figure: VS2010 platform support from


 WebAII looks very promising. I will definitely have a closer look on that tool in the next weeks, months…


Conclusion: Please count the apcheck_thumb1 green ticks and  ap--delete red crosses

Other interesting opinions about Watin and Selenium on testdrivendeveloper


I would not automate the UI too much because that makes your test very fragile for future changes. That you will have sooner or later!

ap--delete BAD EXAMPLE
Create a coded UI test for a whole user interaction with your web app 
e.g. Register, Login, Browse, Find, Add to Shopping cart, Update Billing Address, Update Shipping Address, Checkout, Pay, See Order successful

Makes your UI test very fragile!
But could be a “Done” criteria for your SCRUM team: "An acceptance UI test must be written for each user story"


apcheck_thumb1 GOOD EXAMPLE
1. Create a coded UI tests for single screens
e.g. Logon screen:
  Assert "Username", "Password" boxes and "Login" button are there

2. Create a functional test for testing the actual Login functionality.

Latest Posts

Popular Posts