Smack down!! WatiN vs Selenium vs VS2010 Test Lab Manager

*Updated* 21 March 2011: Updated how "Coded UI tests" finds a control with info from Balagans Blog - How does “Coded UI test” finds a control? 

 

We need Web functionality tests!!! (and a lot of people are already doing them)
Reasons?

  1. Easier to find bugs
  2. No need for a lot of manual testing
  3. Maintainability of your code (THIS IS BIG!!)
    1. When we change something (Code, DB, scripts, HTML,...) we want to verify if our user stories working as expected
    2. We setup web tests for each user story that we have.
  4. We can use web tests in load tests or stress tests
  5. … lots more

Note from Wikipedia
Stress testing: Simulates increased load, more than expected
Load testing: End to end performance testing under anticipated load

 

I tried the following 3 players (recommendations from work mates, and others..)

  1. Selenium
  2. WatiN
  3. Visual Studio 2010 & the Test & Lab Manager

All of these 3 tools provide a recording tool, where you can record your browsing actions and generate code from that.
This is important!!!
No one wants to write these tests manually!

 

Do the test recording

apcheck_thumb1 Selenium Test recorder just works.

Record your session. Copy & paste to Visual Studio. Done

ap--delete Watin Test recorder v2 throw exception

Bug reported here: http://sourceforge.net/tracker/index.php?func=detail&aid=2934193&group_id=193118&atid=944137

ap--delete Watin Test recorder V1 generates unsupported code -> manual fixing needed :-(

 

Code output of recorder

selenium.Open("/Public/default.aspx"); selenium.Click("//div[@id='userlogin']/div/div[2]"); selenium.Click("//div[@id='userstatusLoggedOut']/div/a/span"); selenium.WaitForPageToLoad("30000");
selenium.Type("ctl10_MainContentArea_ctl00_ctl00_CreateUserForm_CreateUserStepContainer_FirstName", "o"); selenium.Type("ctl10_MainContentArea_ctl00_ctl00_CreateUserForm_CreateUserStepContainer_LastName", "g"); selenium.Type("ctl10_MainContentArea_ctl00_ctl00_CreateUserForm_CreateUserStepContainer_Email", "peter@gfader.com"); selenium.Type("ctl10_MainContentArea_ctl00_ctl00_CreateUserForm_CreateUserStepContainer_ConfirmEmail", "peter@gfader.com"); selenium.Type("ctl10_MainContentArea_ctl00_ctl00_CreateUserForm_CreateUserStepContainer_Password", "password!"); selenium.Type("ctl10_MainContentArea_ctl00_ctl00_CreateUserForm_CreateUserStepContainer_ConfirmPassword", "password!"); selenium.Click("ctl10_MainContentArea_ctl00_ctl00_CreateUserForm___CustomNav0_StepNextButtonButton"); selenium.WaitForPageToLoad("30000");
selenium.Click("link=Log Out"); selenium.WaitForPageToLoad("30000");

Figure: Simulating a user registration with Selenium

 

IE ie = new IE("about:blank"); ie.GoTo("http://REMOVED.ssw.com.au/Public"); ie.Div(Find.ByCustom("innertext", "Member Login")).Click(); ie.Link(Find.ByUrl("http://REMOVED.ssw.com.au/Public/Profile/Register.aspx")).Click();
ie.TextField(Find.ByName("ctl10$MainContentArea$ctl00$ctl00$CreateUserForm$CreateUserStepContainer$FirstName")).TypeText("Peter"); ie.TextField(Find.ByName("ctl10$MainContentArea$ctl00$ctl00$CreateUserForm$CreateUserStepContainer$LastName")).TypeText("Gfader"); ie.TextField(Find.ByName("ctl10$MainContentArea$ctl00$ctl00$CreateUserForm$CreateUserStepContainer$Email")).TypeText("petermenuqggfader.com"); ie.TextField(Find.ByName("ctl10$MainContentArea$ctl00$ctl00$CreateUserForm$CreateUserStepContainer$Password")).TypeText("petermenuqggfader.com"); ie.TextField(Find.ByName("ctl10$MainContentArea$ctl00$ctl00$CreateUserForm$CreateUserStepContainer$Password")).TypeText("passmenumenuqword!"); ie.TextField(Find.ByName("ctl10$MainContentArea$ctl00$ctl00$CreateUserForm$CreateUserStepContainer$ConfirmPassword")).TypeText("passmenuqword!"); ie.Button(Find.ByName("ctl10$MainContentArea$ctl00$ctl00$CreateUserForm$__CustomNav0$StepNextButtonButton")).Click();

Figure: Simulating a user registration with WatiN

 

apcheck_thumb1 Selenium is a tool to create tests for web apps

apcheck_thumb1 Selenium’s output is a whole test class

apcheck_thumb1 Selenium’s output code is using relative URL’s (that you define in your test base URL)

apcheck_thumb1 Selenium code: Easy to read

ap--delete WatiN: The code output from the test recorder v1 is quite complicated to read.

I prefer Seleniums: “selenium.Click” instead of the long string and then seeing the “Click()”

// WatiN
watin.TextField(Find.ByName("ctl10$MainContentArea$ctl00$ctl00$CreateUserForm$CreateUserStepContainer$ConfirmPassword")).TypeText("password, f17");
watin.Button(Find.ByName("ctl10$MainContentArea$ctl00$ctl00$CreateUserForm$__CustomNav0$StepNextButtonButton")).Click();

// Selenium
selenium.Type("ctl10_MainContentArea_ctl00_ctl00_CreateUserForm_CreateUserStepContainer_ConfirmPassword", "password");
selenium.Click("ctl10_MainContentArea_ctl00_ctl00_CreateUserForm___CustomNav0_StepNextButtonButton");
         

Figure: Code comparison of WatiN with Selenium. Enter text in a password field and click on button

ap--delete WatiN : Output code from Watin Test recorder v1 was wrong.

Some input control TypeText() was missing
Finding of elements didn't always work E.g.  <a href="xx"> <span> My funkty test </span> </a>  

 

Running the tests

ap--delete Selenium needs a JAVA server to execute the tests

apcheck_thumb1 WatiN doesn't need a proxy server!

ap--delete Selenium is quite slow to run

Output from the Selenium Java server (see the 2 seconds of start up gap)

19:11:54.341 INFO - Command request: getNewBrowserSession[*chrome, http://REMOVED.ssw.com.au/Public/, ] on session null
19:11:54.341 INFO - creating new remote session
19:11:54.341 INFO - Allocated session 69581bdd207a4684b61f6c368af906f0 for http://REMOVED.ssw.com.au/Public/, launching...
19:11:54.370 INFO - Preparing Firefox profile...
19:11:56.895 INFO - Launching Firefox...

 

Code maintainance

apcheck_thumb1 Selenium makes working with Firefox perfectly together: uses the same name as in HTML

E.g. ctl10_MainContentArea_ctl00_ctl00_SubTotal

That means you can copy & paste from FireBug directly to Visual Studio

string priceSubTotalOnCheckout = selenium.GetText("ctl10_MainContentArea_ctl00_ctl00_SubTotal");

ap--delete Watin replaces the underscores with a dollar sign (This is annoying when copy paste from FireBug)

ctl10_MainContentArea_ctl00_ctl00_SubTotal
    needs to be converted to
ctl10$MainContentArea$ctl00$ctl00$SubTotal

string resultText = browser.Element(Find.ByName("ctl10$MainContentArea$ctl00$ctl00$SubTotal")).Text;

 

Note
image
Figure: Selenium is coming from the Java world, as you can see this here. No properties :-) --> GetXXX

 

What about  Visual Studio 2010 and Test & Lab Manager

  1. In Test & Lab Manager you define a test case that links to a user story
    1. User story: As a customer I want to search and buy a bag so that I satisfy my needs for a bag
    2. Test case below
      image
  2. Now you run you test with an Action recording
    1. This is easy. Just follow the instructions from the test case
    2. Done
  3. Close down Test Lab Manager and open Visual Studio, add "Coded UI test" with an "Existing action recording"
  4. Code that you get

[DataSource("Microsoft.VisualStudio.TestTools.DataSource.TestCase", "http://win-peterG:8080/tfs/DefaultCollection;Peter-Gfader-WebTest", "140", DataAccessMethod.Sequential), TestMethod] public void CodedUITestMethod1() { this.UIMap.Openbrowser(); this.UIMap.Gotostartpage();
this.UIMap.ClickonRegisterNew(); this.UIMap.EnterdetailsfirstnamelastnameemailpasswordParams.Ctl10MainContentAreaEditText = TestContext.DataRow["firstname"].ToString(); this.UIMap.EnterdetailsfirstnamelastnameemailpasswordParams.Ctl10MainContentAreaEdit1Text = TestContext.DataRow["lastname"].ToString(); this.UIMap.EnterdetailsfirstnamelastnameemailpasswordParams.Ctl10MainContentAreaEdit2Text1 = TestContext.DataRow["email"].ToString(); this.UIMap.EnterdetailsfirstnamelastnameemailpasswordParams.Ctl10MainContentAreaEdit3Text1 = TestContext.DataRow["email"].ToString(); this.UIMap.EnterdetailsfirstnamelastnameemailpasswordParams.Ctl10MainContentAreaEdit4Text = TestContext.DataRow["password"].ToString(); this.UIMap.EnterdetailsfirstnamelastnameemailpasswordParams.Ctl10MainContentAreaEdit5Text = TestContext.DataRow["password"].ToString(); this.UIMap.Enterdetailsfirstnamelastnameemailpassword(); this.UIMap.ClickonRegister();
this.UIMap.Searchforbag(); this.UIMap.Add1stbagfoundtoshoppingcart();
this.UIMap.Gotoshoppingcart();
this.UIMap.DoCheckout();
this.UIMap.EnterAddressdetails(); this.UIMap.Placeorder(); }

Figure: Code generated by VS2010 for a registration of a new user, searching for a bag, adding this bad and going through the checkout

apcheck_thumb1 VS2010 code is very easy to read. And there are a lot of things happening behind the scenes

UIMap is a huge file generated for you that does all the magic behind the scenes, like finding controls, clicking buttons, …

UIMap: Strongly typed representation of controls

 

How is Visual Studio finding controls?

        public HtmlInputButton RegisterButton
        {
            get
            {
                if ((this.mRegisterButton == null))
                {
                    this.mRegisterButton = new HtmlInputButton(this);

                    #region Search Criteria

                    this.mRegisterButton.SearchProperties[HtmlProperties.Button.Id] = "ctl10_MainContentArea_ctl00_ctl00_CreateUserForm___CustomNav0_StepNextButtonButton";
                    this.mRegisterButton.SearchProperties[HtmlProperties.Button.Name] = "ctl10$MainContentArea$ctl00$ctl00$CreateUserForm$__CustomNav0$StepNextButtonButton";
                    this.mRegisterButton.FilterProperties[HtmlProperties.Button.DisplayText] = "Register";
                    this.mRegisterButton.FilterProperties[HtmlProperties.Button.Type] = "submit";
                    this.mRegisterButton.FilterProperties[HtmlProperties.Button.Title] = null;
                    this.mRegisterButton.FilterProperties[HtmlProperties.Button.Class] = null;
                    this.mRegisterButton.FilterProperties[HtmlProperties.Button.ControlDefinition] = "id=ctl10_MainContentArea_ctl00_ctl00_Cre";
                    this.mRegisterButton.FilterProperties[HtmlProperties.Button.TagInstance] = "40";
                    this.mRegisterButton.WindowTitles.Add("Peter Gfader | Sign Up - Windows Internet Explorer");

                    #endregion

                }
                return this.mRegisterButton;
            }
        }

Figure: VS 2010 Beta2 generated code for finding a button on a web page. I think this is way TOO STRICT

ap--delete VS2010: All the "SearchProperties"above have to match, otherwise the button won’t be found. If too many controls are found from the "SearchProperties", the "FilterProperties" are applied one by one. All details on Balagans Blog - How does “Coded UI test” finds a control?

ap--delete There is no option to customize the default generated SearchProperties

 

I would probably change that to

        public HtmlInputButton RegisterButton
        {
            get
            {
                if ((this.mRegisterButton == null))
                {
                    this.mRegisterButton = new HtmlInputButton(this);
                    #region Search Criteria

                    this.mRegisterButton.FilterProperties[HtmlProperties.Button.DisplayText] = "Register";
                    this.mRegisterButton.FilterProperties[HtmlProperties.Button.Type] = "submit";

                    #endregion
                }
                return this.mRegisterButton;
            }
        }

apcheck_thumb1 Better way to search for the Register button. Filter on DisplayText and type of the control.
Note: This would fail if there are 2 buttons with "Register" and of type "Submit" on the page. Which is excellent!

 

Can VS2010 simulate a FireFox?
See this image for details about the VS2010 platform support
image
Figure: VS2010 platform support from http://videos.visitmix.com/MIX09/T83M

 

 WebAII looks very promising. I will definitely have a closer look on that tool in the next weeks, months…

 

Conclusion: Please count the apcheck_thumb1 green ticks and  ap--delete red crosses

Other interesting opinions about Watin and Selenium on testdrivendeveloper

 

PERSONAL
I would not automate the UI too much because that makes your test very fragile for future changes. That you will have sooner or later!

ap--delete BAD EXAMPLE
Create a coded UI test for a whole user interaction with your web app 
e.g. Register, Login, Browse, Find, Add to Shopping cart, Update Billing Address, Update Shipping Address, Checkout, Pay, See Order successful

Makes your UI test very fragile!
But could be a “Done” criteria for your SCRUM team: "An acceptance UI test must be written for each user story"

 

apcheck_thumb1 GOOD EXAMPLE
1. Create a coded UI tests for single screens
e.g. Logon screen:
  Assert "Username", "Password" boxes and "Login" button are there

2. Create a functional test for testing the actual Login functionality.

16 comments:

Unknown said...

Hi,
nice comparison. Used Selenium in my last project and I love it for having fast UI tests. Especially when you're reusing/modifying UI user controls it's just great to ensure that the other pages still work fine (well, that's a test's purpose after all).

Another use of Selenium and its Firefox IDE is as a tool for filling out large forms during dev. Example: you work on a large form's (also wizard's) save functionality and you have to check whether the validators work correctly. So instead of filling out all the required fields each time you may just use Selenium, record it the first time and then replay as often as you wish. Really comfortable.

Peter Gfader said...

I started with Selenium the same way.
Automating my user creation process via the web.

I couldn't directly insert new users to the DB because it was dependent from a lot of tables.

So I had to use a Selenium extension at that time, to do the GOTO (a loop of 1 user creation)

Unknown said...

I use watiN and it works just fine, but not with the recorder. Also, we all know recorder just makes thing a bit faster but in all cases you have to edit the code to make it work always.. like e.g. you cant register the same user twice so you need to have random values. It's better not to use the full id of a field in case it changes during edits and so many other examples

Peter Gfader said...

Hi Sherine

Do you have a blog where you share your experiences?

Anonymous said...

My experience was pretty similar to yours. But the Watin Recorder that I found rendered HTML so poorly that our site wasn't usable; it's probably using IE6. Looks like the Watin Recorder project is dead in the water.

Unknown said...

Just a thought:

How can any automation tool help you find bug easier?? :O ->It can only help you find bugs that you thought might occur and will not help you find those bugs which of you haven't thought about.

Designing your automation is the key factor here.

Manual testing done by skilled test engineer is always helpful, no matter how much you have automated.

Peter Gfader said...

Hi Mohammad

#1
>>How can any automation tool help you find bug easier?
Cause writing/developing/recording tests puts you in the tester perspective and you see your software from a completely different point of view. This different view leads you to spot certain things and avoid certain things in the future.

#2
Agree!
Automation is the key!
Manual testing will always be needed! Automated tests can help, and it is not easy to find the balance...

Anonymous said...

Selenium looks great but I just want to make a comment on WatiN. We've been using WatiN in my current project with great success, we do not use any sort of recorder (and frankly, I've never missed it). We wrap all the html details in WatiN page objects so that we can easily change our GUI without breaking our workflows (which operate on the page objects, not the html details). Specflow + WatiN has been just smiles and laughter so far :-)

Peter Gfader said...

Hi Anonymous,

I use the recorder to get started with the tests, but then I refactor them out to have an abstraction of the controls. I guess similar as you do.
Record+Play is nice for demos but hard to maintain and cause brittle tests.


How long does it take on avg to execute 1 of your Specflow+WatiN tests?

Anonymous said...

Our (current) 417 specflow/watin tests ran last night for approx 1 hour and 30 mins (5233 seconds) at an average of 12,55 seconds pr test.

Anonymous said...

Hi Peter,

please help me on this case i used the selenium and it's great but now i need some recorder automation like selenium but working with silverlight and HTML5 and selenium doesn't working with.
awni.baggio@hotmail.com

Peter Gfader said...

Hi awni.baggio

Testing Silverlight is not supported by Selenium, because Selenium works on the HTML DOM level.

There is http://code.google.com/p/silverlight-selenium/ although I havent tried it yet


For testing Silverlight you might look into Telerik Test Studio http://www.telerik.com/automated-testing-tools.aspx
OR
Microsoft Visual Studio 2010 Feature Pack 2

There is also a video recording from Roy Osherove about Testing Silverlight
http://osherove.com/videos/2009/8/25/unit-testing-in-silverlight-silverunit-and-ms-sliverlight-te.html

Anonymous said...

Hi Guys, whatwould you suggest for wep applications running on IE?

Peter Gfader said...

>> wep applications running on IE

What do you mean by wep applications?

Anit Patel said...

Hi Peter,

Very nice post on comparing these three Automation tools.
I would like to add a little in this.

>>Selenium needs a JAVA server to execute the tests
This is needed only when you use Selenium RC and every one now have started using Selenium WebDriver which doesn't require to have Java Server running. It was an old requirment for Selenium RC.

>>Selenium is quite slow to run
I think this is true agin in case of Selenium RC which can be fixed by using some workaround.
I wrote a little blog on "Why Selenium RC is so slow" which can be found at : http://www.anitpatel.net/2012/02/24/selenium-rc-why-selenium-rc-is-so-slow/


Thanks again for this cool post. Cheers.

-Anit Patel

Anonymous said...

Hi Anit

Thx for your additions.
I guess I need to update this entry a little.

Post a Comment

Latest Posts

Popular Posts