Smack down!! WatiN vs Selenium vs VS2010 Test Lab Manager

*Updated* 21 March 2011: Updated how "Coded UI tests" finds a control with info from Balagans Blog - How does “Coded UI test” finds a control? 

 

We need Web functionality tests!!! (and a lot of people are already doing them)
Reasons?

1. Easier to find bugs
2. No need for a lot of manual testing
3. Maintainability of your code (THIS IS BIG!!)
   1. When we change something (Code, DB, scripts, HTML,...) we want to verify if our user stories working as expected
   2. We setup web tests for each user story that we have.
4. We can use web tests in load tests or stress tests
5. … lots more

Note from Wikipedia
Stress testing: Simulates increased load, more than expected
Load testing: End to end performance testing under anticipated load

 

I tried the following 3 players (recommendations from work mates, and others..)

1. Selenium
2. WatiN
3. Visual Studio 2010 & the Test & Lab Manager

All of these 3 tools provide a recording tool, where you can record your browsing actions and generate code from that.
This is important!!!
No one wants to write these tests manually!

 

Do the test recording

Selenium Test recorder just works.

Record your session. Copy & paste to Visual Studio. Done

Watin Test recorder v2 throw exception

Bug reported here: http://sourceforge.net/tracker/index.php?func=detail&aid=2934193&group_id=193118&atid=944137

Watin Test recorder V1 generates unsupported code -> manual fixing needed :-(

 

Code output of recorder

selenium.Open("/Public/default.aspx");
selenium.Click("//div[@id='userlogin']/div/div[2]");
selenium.Click("//div[@id='userstatusLoggedOut']/div/a/span");
selenium.WaitForPageToLoad("30000");

selenium.Type("ctl10_MainContentArea_ctl00_ctl00_CreateUserForm_CreateUserStepContainer_FirstName", "o");
selenium.Type("ctl10_MainContentArea_ctl00_ctl00_CreateUserForm_CreateUserStepContainer_LastName", "g");
selenium.Type("ctl10_MainContentArea_ctl00_ctl00_CreateUserForm_CreateUserStepContainer_Email", "peter@gfader.com");
selenium.Type("ctl10_MainContentArea_ctl00_ctl00_CreateUserForm_CreateUserStepContainer_ConfirmEmail", "peter@gfader.com");
selenium.Type("ctl10_MainContentArea_ctl00_ctl00_CreateUserForm_CreateUserStepContainer_Password", "password!");
selenium.Type("ctl10_MainContentArea_ctl00_ctl00_CreateUserForm_CreateUserStepContainer_ConfirmPassword", "password!");
selenium.Click("ctl10_MainContentArea_ctl00_ctl00_CreateUserForm___CustomNav0_StepNextButtonButton");
selenium.WaitForPageToLoad("30000");

selenium.Click("link=Log Out");
selenium.WaitForPageToLoad("30000");

Figure: Simulating a user registration with Selenium

 

IE ie = new IE("about:blank");
ie.GoTo("http://REMOVED.ssw.com.au/Public");
ie.Div(Find.ByCustom("innertext", "Member Login")).Click();
ie.Link(Find.ByUrl("http://REMOVED.ssw.com.au/Public/Profile/Register.aspx")).Click();

ie.TextField(Find.ByName("ctl10$MainContentArea$ctl00$ctl00$CreateUserForm$CreateUserStepContainer$FirstName")).TypeText("Peter");
ie.TextField(Find.ByName("ctl10$MainContentArea$ctl00$ctl00$CreateUserForm$CreateUserStepContainer$LastName")).TypeText("Gfader");
ie.TextField(Find.ByName("ctl10$MainContentArea$ctl00$ctl00$CreateUserForm$CreateUserStepContainer$Email")).TypeText("petermenuqggfader.com");
ie.TextField(Find.ByName("ctl10$MainContentArea$ctl00$ctl00$CreateUserForm$CreateUserStepContainer$Password")).TypeText("petermenuqggfader.com");
ie.TextField(Find.ByName("ctl10$MainContentArea$ctl00$ctl00$CreateUserForm$CreateUserStepContainer$Password")).TypeText("passmenumenuqword!");
ie.TextField(Find.ByName("ctl10$MainContentArea$ctl00$ctl00$CreateUserForm$CreateUserStepContainer$ConfirmPassword")).TypeText("passmenuqword!");
ie.Button(Find.ByName("ctl10$MainContentArea$ctl00$ctl00$CreateUserForm$__CustomNav0$StepNextButtonButton")).Click();

Figure: Simulating a user registration with WatiN

 

Selenium is a tool to create tests for web apps

Selenium’s output is a whole test class

Selenium’s output code is using relative URL’s (that you define in your test base URL)

Selenium code: Easy to read

WatiN: The code output from the test recorder v1 is quite complicated to read.

I prefer Seleniums: “selenium.Click” instead of the long string and then seeing the “Click()”

// WatiN
watin.TextField(Find.ByName("ctl10$MainContentArea$ctl00$ctl00$CreateUserForm$CreateUserStepContainer$ConfirmPassword")).TypeText("password, f17");
watin.Button(Find.ByName("ctl10$MainContentArea$ctl00$ctl00$CreateUserForm$__CustomNav0$StepNextButtonButton")).Click();

// Selenium
selenium.Type("ctl10_MainContentArea_ctl00_ctl00_CreateUserForm_CreateUserStepContainer_ConfirmPassword", "password");
selenium.Click("ctl10_MainContentArea_ctl00_ctl00_CreateUserForm___CustomNav0_StepNextButtonButton");
         

Figure: Code comparison of WatiN with Selenium. Enter text in a password field and click on button

WatiN : Output code from Watin Test recorder v1 was wrong.

Some input control TypeText() was missing
Finding of elements didn't always work E.g.  <a href="xx"> <span> My funkty test </span> </a>  

 

Running the tests

Selenium needs a JAVA server to execute the tests

WatiN doesn't need a proxy server!

Selenium is quite slow to run

Output from the Selenium Java server (see the 2 seconds of start up gap)

19:11:54.341 INFO - Command request: getNewBrowserSession[*chrome, http://REMOVED.ssw.com.au/Public/, ] on session null
19:11:54.341 INFO - creating new remote session
19:11:54.341 INFO - Allocated session 69581bdd207a4684b61f6c368af906f0 for http://REMOVED.ssw.com.au/Public/, launching...
19:11:54.370 INFO - Preparing Firefox profile...
19:11:56.895 INFO - Launching Firefox...

 

Code maintainance

Selenium makes working with Firefox perfectly together: uses the same name as in HTML

E.g. ctl10_MainContentArea_ctl00_ctl00_SubTotal

That means you can copy & paste from FireBug directly to Visual Studio

string priceSubTotalOnCheckout = selenium.GetText("ctl10_MainContentArea_ctl00_ctl00_SubTotal");

Watin replaces the underscores with a dollar sign (This is annoying when copy paste from FireBug)

ctl10_MainContentArea_ctl00_ctl00_SubTotal
    needs to be converted to
ctl10$MainContentArea$ctl00$ctl00$SubTotal

string resultText = browser.Element(Find.ByName("ctl10$MainContentArea$ctl00$ctl00$SubTotal")).Text; 

 

Note

Figure: Selenium is coming from the Java world, as you can see this here. No properties :-) --> GetXXX

 

What about  Visual Studio 2010 and Test & Lab Manager

1. In Test & Lab Manager you define a test case that links to a user story
   1. User story: As a customer I want to search and buy a bag so that I satisfy my needs for a bag
   2. Test case below
      
      
2. Now you run you test with an Action recording
   1. This is easy. Just follow the instructions from the test case
   2. Done
      
3. Close down Test Lab Manager and open Visual Studio, add "Coded UI test" with an "Existing action recording"
4. Code that you get

[DataSource("Microsoft.VisualStudio.TestTools.DataSource.TestCase", "http://win-peterG:8080/tfs/DefaultCollection;Peter-Gfader-WebTest", "140", DataAccessMethod.Sequential), TestMethod]
public void CodedUITestMethod1()
{
    this.UIMap.Openbrowser();
    this.UIMap.Gotostartpage();

    this.UIMap.ClickonRegisterNew();
    this.UIMap.EnterdetailsfirstnamelastnameemailpasswordParams.Ctl10MainContentAreaEditText = TestContext.DataRow["firstname"].ToString();
    this.UIMap.EnterdetailsfirstnamelastnameemailpasswordParams.Ctl10MainContentAreaEdit1Text = TestContext.DataRow["lastname"].ToString();
    this.UIMap.EnterdetailsfirstnamelastnameemailpasswordParams.Ctl10MainContentAreaEdit2Text1 = TestContext.DataRow["email"].ToString();
    this.UIMap.EnterdetailsfirstnamelastnameemailpasswordParams.Ctl10MainContentAreaEdit3Text1 = TestContext.DataRow["email"].ToString();
    this.UIMap.EnterdetailsfirstnamelastnameemailpasswordParams.Ctl10MainContentAreaEdit4Text = TestContext.DataRow["password"].ToString();
    this.UIMap.EnterdetailsfirstnamelastnameemailpasswordParams.Ctl10MainContentAreaEdit5Text = TestContext.DataRow["password"].ToString();
    this.UIMap.Enterdetailsfirstnamelastnameemailpassword();
    this.UIMap.ClickonRegister();

    this.UIMap.Searchforbag();
    this.UIMap.Add1stbagfoundtoshoppingcart();

    this.UIMap.Gotoshoppingcart();

    this.UIMap.DoCheckout();

    this.UIMap.EnterAddressdetails();
    this.UIMap.Placeorder();
}

Figure: Code generated by VS2010 for a registration of a new user, searching for a bag, adding this bad and going through the checkout

VS2010 code is very easy to read. And there are a lot of things happening behind the scenes

UIMap is a huge file generated for you that does all the magic behind the scenes, like finding controls, clicking buttons, …

UIMap: Strongly typed representation of controls

 

How is Visual Studio finding controls?

        public HtmlInputButton RegisterButton
        {
            get
            {
                if ((this.mRegisterButton == null))
                {
                    this.mRegisterButton = new HtmlInputButton(this);

                    #region Search Criteria

                    this.mRegisterButton.SearchProperties[HtmlProperties.Button.Id] = "ctl10_MainContentArea_ctl00_ctl00_CreateUserForm___CustomNav0_StepNextButtonButton";
                    this.mRegisterButton.SearchProperties[HtmlProperties.Button.Name] = "ctl10$MainContentArea$ctl00$ctl00$CreateUserForm$__CustomNav0$StepNextButtonButton";
                    this.mRegisterButton.FilterProperties[HtmlProperties.Button.DisplayText] = "Register";
                    this.mRegisterButton.FilterProperties[HtmlProperties.Button.Type] = "submit";
                    this.mRegisterButton.FilterProperties[HtmlProperties.Button.Title] = null;
                    this.mRegisterButton.FilterProperties[HtmlProperties.Button.Class] = null;
                    this.mRegisterButton.FilterProperties[HtmlProperties.Button.ControlDefinition] = "id=ctl10_MainContentArea_ctl00_ctl00_Cre";
                    this.mRegisterButton.FilterProperties[HtmlProperties.Button.TagInstance] = "40";
                    this.mRegisterButton.WindowTitles.Add("Peter Gfader | Sign Up - Windows Internet Explorer");

                    #endregion

                }
                return this.mRegisterButton;
            }
        }

Figure: VS 2010 Beta2 generated code for finding a button on a web page. I think this is way TOO STRICT

VS2010: All the "SearchProperties"above have to match, otherwise the button won’t be found. If too many controls are found from the "SearchProperties", the "FilterProperties" are applied one by one. All details on Balagans Blog - How does “Coded UI test” finds a control?

There is no option to customize the default generated SearchProperties

 

I would probably change that to

        public HtmlInputButton RegisterButton
        {
            get
            {
                if ((this.mRegisterButton == null))
                {
                    this.mRegisterButton = new HtmlInputButton(this);
                    #region Search Criteria

                    this.mRegisterButton.FilterProperties[HtmlProperties.Button.DisplayText] = "Register";
                    this.mRegisterButton.FilterProperties[HtmlProperties.Button.Type] = "submit";

                    #endregion
                }
                return this.mRegisterButton;
            }
        }

Better way to search for the Register button. Filter on DisplayText and type of the control.
Note: This would fail if there are 2 buttons with "Register" and of type "Submit" on the page. Which is excellent!

 

Can VS2010 simulate a FireFox?
See this image for details about the VS2010 platform support

Figure: VS2010 platform support from http://videos.visitmix.com/MIX09/T83M

 

 WebAII looks very promising. I will definitely have a closer look on that tool in the next weeks, months…

 

Conclusion: Please count the green ticks and  red crosses

Other interesting opinions about Watin and Selenium on testdrivendeveloper

 

PERSONAL
I would not automate the UI too much because that makes your test very fragile for future changes. That you will have sooner or later!

BAD EXAMPLE
Create a coded UI test for a whole user interaction with your web app 
e.g. Register, Login, Browse, Find, Add to Shopping cart, Update Billing Address, Update Shipping Address, Checkout, Pay, See Order successful

Makes your UI test very fragile!
But could be a “Done” criteria for your SCRUM team: "An acceptance UI test must be written for each user story"

 

GOOD EXAMPLE
1. Create a coded UI tests for single screens
e.g. Logon screen:
  Assert "Username", "Password" boxes and "Login" button are there

2. Create a functional test for testing the actual Login functionality.