*Updated* 21 March 2011: Updated how "Coded UI tests" finds a control with info from Balagans Blog - How does “Coded UI test” finds a control?
We need Web functionality tests!!! (and a lot of people are already doing them)
Reasons?
- Easier to find bugs
- No need for a lot of manual testing
- Maintainability of your code (THIS IS BIG!!)
- When we change something (Code, DB, scripts, HTML,...) we want to verify if our user stories working as expected
- We setup web tests for each user story that we have.
- We can use web tests in load tests or stress tests
- … lots more
Note from Wikipedia
Stress testing: Simulates increased load, more than expected
Load testing: End to end performance testing under anticipated load
I tried the following 3 players (recommendations from work mates, and others..)
- Selenium
- WatiN
- Visual Studio 2010 & the Test & Lab Manager
All of these 3 tools provide a recording tool, where you can record your browsing actions and generate code from that.
This is important!!!
No one wants to write these tests manually!
Do the test recording
Selenium Test recorder just works.
Record your session. Copy & paste to Visual Studio. Done
Watin Test recorder v2 throw exception
Bug reported here: http://sourceforge.net/tracker/index.php?func=detail&aid=2934193&group_id=193118&atid=944137
Watin Test recorder V1 generates unsupported code -> manual fixing needed :-(
Code output of recorder
selenium.Open("/Public/default.aspx");
selenium.Click("//div[@id='userlogin']/div/div[2]");
selenium.Click("//div[@id='userstatusLoggedOut']/div/a/span");
selenium.WaitForPageToLoad("30000");
selenium.Type("ctl10_MainContentArea_ctl00_ctl00_CreateUserForm_CreateUserStepContainer_FirstName", "o");
selenium.Type("ctl10_MainContentArea_ctl00_ctl00_CreateUserForm_CreateUserStepContainer_LastName", "g");
selenium.Type("ctl10_MainContentArea_ctl00_ctl00_CreateUserForm_CreateUserStepContainer_Email", "peter@gfader.com");
selenium.Type("ctl10_MainContentArea_ctl00_ctl00_CreateUserForm_CreateUserStepContainer_ConfirmEmail", "peter@gfader.com");
selenium.Type("ctl10_MainContentArea_ctl00_ctl00_CreateUserForm_CreateUserStepContainer_Password", "password!");
selenium.Type("ctl10_MainContentArea_ctl00_ctl00_CreateUserForm_CreateUserStepContainer_ConfirmPassword", "password!");
selenium.Click("ctl10_MainContentArea_ctl00_ctl00_CreateUserForm___CustomNav0_StepNextButtonButton");
selenium.WaitForPageToLoad("30000");
selenium.Click("link=Log Out");
selenium.WaitForPageToLoad("30000");
Figure: Simulating a user registration with
Selenium
IE ie = new IE("about:blank");
ie.GoTo("http://REMOVED.ssw.com.au/Public");
ie.Div(Find.ByCustom("innertext", "Member Login")).Click();
ie.Link(Find.ByUrl("http://REMOVED.ssw.com.au/Public/Profile/Register.aspx")).Click();
ie.TextField(Find.ByName("ctl10$MainContentArea$ctl00$ctl00$CreateUserForm$CreateUserStepContainer$FirstName")).TypeText("Peter");
ie.TextField(Find.ByName("ctl10$MainContentArea$ctl00$ctl00$CreateUserForm$CreateUserStepContainer$LastName")).TypeText("Gfader");
ie.TextField(Find.ByName("ctl10$MainContentArea$ctl00$ctl00$CreateUserForm$CreateUserStepContainer$Email")).TypeText("petermenuqggfader.com");
ie.TextField(Find.ByName("ctl10$MainContentArea$ctl00$ctl00$CreateUserForm$CreateUserStepContainer$Password")).TypeText("petermenuqggfader.com");
ie.TextField(Find.ByName("ctl10$MainContentArea$ctl00$ctl00$CreateUserForm$CreateUserStepContainer$Password")).TypeText("passmenumenuqword!");
ie.TextField(Find.ByName("ctl10$MainContentArea$ctl00$ctl00$CreateUserForm$CreateUserStepContainer$ConfirmPassword")).TypeText("passmenuqword!");
ie.Button(Find.ByName("ctl10$MainContentArea$ctl00$ctl00$CreateUserForm$__CustomNav0$StepNextButtonButton")).Click();
Figure: Simulating a user registration with WatiN
Selenium is a tool to create tests for web apps
Selenium’s output is a whole test class
Selenium’s output code is using relative URL’s (that you define in your test base URL)
Selenium code: Easy to read
WatiN: The code output from the test recorder v1 is quite complicated to read.
I prefer Seleniums: “selenium.Click” instead of the long string and then seeing the “Click()”
// WatiN
watin.TextField(Find.ByName("ctl10$MainContentArea$ctl00$ctl00$CreateUserForm$CreateUserStepContainer$ConfirmPassword")).TypeText("password, f17");
watin.Button(Find.ByName("ctl10$MainContentArea$ctl00$ctl00$CreateUserForm$__CustomNav0$StepNextButtonButton")).Click();
// Selenium
selenium.Type("ctl10_MainContentArea_ctl00_ctl00_CreateUserForm_CreateUserStepContainer_ConfirmPassword", "password");
selenium.Click("ctl10_MainContentArea_ctl00_ctl00_CreateUserForm___CustomNav0_StepNextButtonButton");
Figure: Code comparison of WatiN with Selenium. Enter text in a password field and click on button
WatiN : Output code from Watin Test recorder v1 was wrong.
Some input control TypeText() was missing
Finding of elements didn't always work E.g. <a href="xx"> <span> My funkty test </span> </a>
Running the tests
Selenium needs a JAVA server to execute the tests
WatiN doesn't need a proxy server!
Selenium is quite slow to run
Output from the Selenium Java server (see the 2 seconds of start up gap)
19:11:54.341 INFO - Command request: getNewBrowserSession[*chrome, http://REMOVED.ssw.com.au/Public/, ] on session null
19:11:54.341 INFO - creating new remote session
19:11:54.341 INFO - Allocated session 69581bdd207a4684b61f6c368af906f0 for http://REMOVED.ssw.com.au/Public/, launching...
19:11:54.370 INFO - Preparing Firefox profile...
19:11:56.895 INFO - Launching Firefox...
Code maintainance
Selenium makes working with Firefox perfectly together: uses the same name as in HTML
E.g. ctl10_MainContentArea_ctl00_ctl00_SubTotal
That means you can copy & paste from FireBug directly to Visual Studio
string priceSubTotalOnCheckout = selenium.GetText("ctl10_MainContentArea_ctl00_ctl00_SubTotal");
Watin replaces the underscores with a dollar sign (This is annoying when copy paste from FireBug)
ctl10_MainContentArea_ctl00_ctl00_SubTotal
needs to be converted to
ctl10$MainContentArea$ctl00$ctl00$SubTotal
string resultText = browser.Element(Find.ByName("ctl10$MainContentArea$ctl00$ctl00$SubTotal")).Text;
Note
Figure: Selenium is coming from the Java world, as you can see this here. No properties :-) --> GetXXX
What about Visual Studio 2010 and Test & Lab Manager
- In Test & Lab Manager you define a test case that links to a user story
- User story: As a customer I want to search and buy a bag so that I satisfy my needs for a bag
- Test case below
- Now you run you test with an Action recording
- This is easy. Just follow the instructions from the test case
- Done
- Close down Test Lab Manager and open Visual Studio, add "Coded UI test" with an "Existing action recording"
- Code that you get
[DataSource("Microsoft.VisualStudio.TestTools.DataSource.TestCase", "http://win-peterG:8080/tfs/DefaultCollection;Peter-Gfader-WebTest", "140", DataAccessMethod.Sequential), TestMethod]
public void CodedUITestMethod1()
{
this.UIMap.Openbrowser();
this.UIMap.Gotostartpage();
this.UIMap.ClickonRegisterNew();
this.UIMap.EnterdetailsfirstnamelastnameemailpasswordParams.Ctl10MainContentAreaEditText = TestContext.DataRow["firstname"].ToString();
this.UIMap.EnterdetailsfirstnamelastnameemailpasswordParams.Ctl10MainContentAreaEdit1Text = TestContext.DataRow["lastname"].ToString();
this.UIMap.EnterdetailsfirstnamelastnameemailpasswordParams.Ctl10MainContentAreaEdit2Text1 = TestContext.DataRow["email"].ToString();
this.UIMap.EnterdetailsfirstnamelastnameemailpasswordParams.Ctl10MainContentAreaEdit3Text1 = TestContext.DataRow["email"].ToString();
this.UIMap.EnterdetailsfirstnamelastnameemailpasswordParams.Ctl10MainContentAreaEdit4Text = TestContext.DataRow["password"].ToString();
this.UIMap.EnterdetailsfirstnamelastnameemailpasswordParams.Ctl10MainContentAreaEdit5Text = TestContext.DataRow["password"].ToString();
this.UIMap.Enterdetailsfirstnamelastnameemailpassword();
this.UIMap.ClickonRegister();
this.UIMap.Searchforbag();
this.UIMap.Add1stbagfoundtoshoppingcart();
this.UIMap.Gotoshoppingcart();
this.UIMap.DoCheckout();
this.UIMap.EnterAddressdetails();
this.UIMap.Placeorder();
}
Figure: Code generated by VS2010 for a registration of a new user, searching for a bag, adding this bad and going through the checkout
VS2010 code is very easy to read. And there are a lot of things happening behind the scenes
UIMap is a huge file generated for you that does all the magic behind the scenes, like finding controls, clicking buttons, …
UIMap: Strongly typed representation of controls
How is Visual Studio finding controls?
public HtmlInputButton RegisterButton
{
get
{
if ((this.mRegisterButton == null))
{
this.mRegisterButton = new HtmlInputButton(this);
#region Search Criteria
this.mRegisterButton.SearchProperties[HtmlProperties.Button.Id] = "ctl10_MainContentArea_ctl00_ctl00_CreateUserForm___CustomNav0_StepNextButtonButton";
this.mRegisterButton.SearchProperties[HtmlProperties.Button.Name] = "ctl10$MainContentArea$ctl00$ctl00$CreateUserForm$__CustomNav0$StepNextButtonButton";
this.mRegisterButton.FilterProperties[HtmlProperties.Button.DisplayText] = "Register";
this.mRegisterButton.FilterProperties[HtmlProperties.Button.Type] = "submit";
this.mRegisterButton.FilterProperties[HtmlProperties.Button.Title] = null;
this.mRegisterButton.FilterProperties[HtmlProperties.Button.Class] = null;
this.mRegisterButton.FilterProperties[HtmlProperties.Button.ControlDefinition] = "id=ctl10_MainContentArea_ctl00_ctl00_Cre";
this.mRegisterButton.FilterProperties[HtmlProperties.Button.TagInstance] = "40";
this.mRegisterButton.WindowTitles.Add("Peter Gfader | Sign Up - Windows Internet Explorer");
#endregion
}
return this.mRegisterButton;
}
}
Figure: VS 2010 Beta2 generated code for finding a button on a web page. I think this is way TOO STRICT
VS2010: All the "SearchProperties"above have to match, otherwise the button won’t be found. If too many controls are found from the "SearchProperties", the "FilterProperties" are applied one by one. All details on Balagans Blog - How does “Coded UI test” finds a control?
There is no option to customize the default generated SearchProperties
I would probably change that to
public HtmlInputButton RegisterButton
{
get
{
if ((this.mRegisterButton == null))
{
this.mRegisterButton = new HtmlInputButton(this);
#region Search Criteria
this.mRegisterButton.FilterProperties[HtmlProperties.Button.DisplayText] = "Register";
this.mRegisterButton.FilterProperties[HtmlProperties.Button.Type] = "submit";
#endregion
}
return this.mRegisterButton;
}
}
Better way to search for the Register button. Filter on DisplayText and type of the control.
Note: This would fail if there are 2 buttons with "Register" and of type "Submit" on the page. Which is excellent!
Can VS2010 simulate a FireFox?
See this image for details about the VS2010 platform support
Figure: VS2010 platform support from http://videos.visitmix.com/MIX09/T83M
WebAII looks very promising. I will definitely have a closer look on that tool in the next weeks, months…
Conclusion: Please count the 
green ticks and red crosses
Other interesting opinions about Watin and Selenium on testdrivendeveloper
PERSONAL
I would not automate the UI too much because that makes your test very fragile for future changes. That you will have sooner or later!
BAD EXAMPLE
Create a coded UI test for a whole user interaction with your web app
e.g. Register, Login, Browse, Find, Add to Shopping cart, Update Billing Address, Update Shipping Address, Checkout, Pay, See Order successful
Makes your UI test very fragile!
But could be a “Done” criteria for your SCRUM team: "An acceptance UI test must be written for each user story"
GOOD EXAMPLE
1. Create a coded UI tests for single screens
e.g. Logon screen:
Assert "Username", "Password" boxes and "Login" button are there
2. Create a functional test for testing the actual Login functionality.
