I tried different ways to run nunit on our build server.
From "NUnit for Team Build", to "executing nunit-console on cmd" and eventually customizing our project files…
None of those methods made me happy. 
Figure: Sad - Broken builds and mucking around with XML make dev's unhappy
The best way (and most important *working* way) is by customizing the build template.
By using the customization from Shawn Wallace I was able to run nunit tests on the build server and get a nice output if a test fails.
http://blog.shawnewallace.com/2011/02/running-nunit-tests-in-tfs-2010.html
I am writing this up, because I have 3 comments on his build template change
- I removed all 'sap:' attributes from his XML in order to get it working on our TFS 2010
- Make sure to customize the path of the nunit-console runner
Search for nunit-console in the template and replace the path - Add the below XML before the 1st occurrence of '<If Condition="[Not DisableTests]"'
The resulting template looks like this
<!-- *********** START: Unit Testing *********** -->
<If Condition="[Not DisableTests]" DisplayName="If Not DisableTests" >
<If.Then>
<Sequence DisplayName="Run Tests" >
<Sequence.Variables>
<Variable x:TypeArguments="scg:IEnumerable(x:String)" Name="testAssemblies" />
<Variable x:TypeArguments="mtbwa:TestAssemblySpec" Name="testAssembly" />
<Variable x:TypeArguments="mtbwa:TestAssemblySpec" Name="testAssemblySpec" />
</Sequence.Variables>
<ForEach x:TypeArguments="mtbwa:TestSpec" DisplayName="ForEach<TestSpec>" Values="[TestSpecs]">
<ActivityAction x:TypeArguments="mtbwa:TestSpec">
<ActivityAction.Argument>
<DelegateInArgument x:TypeArguments="mtbwa:TestSpec" Name="testSpec" />
</ActivityAction.Argument>
<Sequence >
<Sequence.Variables>
<Variable x:TypeArguments="x:String" Name="testAssemblyMatchPattern" />
</Sequence.Variables>
<mtbwa:WriteBuildMessage DisplayName="Write outputDirectory" Importance="[Microsoft.TeamFoundation.Build.Client.BuildMessageImportance.High]" Message="["outputDirectory is: " + outputDirectory]" mva:VisualBasic.Settings="Assembly references and imported namespaces serialized as XML namespaces" />
<Cast x:TypeArguments="mtbwa:TestSpec, mtbwa:TestAssemblySpec" Operand="[testSpec]" Result="[testAssemblySpec]" />
<mtbwa:WriteBuildMessage DisplayName="Write testAssemblySpec.AssemblyFileSpec" Importance="[Microsoft.TeamFoundation.Build.Client.BuildMessageImportance.High]" Message="["testAssemblySpec.AssemblyFileSpec is: " + testAssemblySpec.AssemblyFileSpec]" mva:VisualBasic.Settings="Assembly references and imported namespaces serialized as XML namespaces" />
<Assign >
<Assign.To>
<OutArgument x:TypeArguments="x:String">[testAssemblyMatchPattern]</OutArgument>
</Assign.To>
<Assign.Value>
<InArgument x:TypeArguments="x:String">[String.Format("{0}\{1}", outputDirectory, testAssemblySpec.AssemblyFileSpec)]</InArgument>
</Assign.Value>
</Assign>
<mtbwa:WriteBuildMessage Importance="[Microsoft.TeamFoundation.Build.Client.BuildMessageImportance.High]" Message="["testAssemblyMatchPattern: " + testAssemblyMatchPattern]" mva:VisualBasic.Settings="Assembly references and imported namespaces serialized as XML namespaces" />
<mtbwa:FindMatchingFiles DisplayName="Find Test Assemblies --> testAssemblies" MatchPattern="[testAssemblyMatchPattern]" Result="[testAssemblies]" />
<!-- Assign value to treatTestFailureAsBuildFailure -->
<Assign >
<Assign.To>
<OutArgument x:TypeArguments="x:Boolean">[treatTestFailureAsBuildFailure]</OutArgument>
</Assign.To>
<Assign.Value>
<InArgument x:TypeArguments="x:Boolean">[testSpec.FailBuildOnFailure]</InArgument>
</Assign.Value>
</Assign>
<If Condition="[testAssemblies.Count() > 0]" DisplayName="If Test Assemblies Found" >
<If.Then>
<Sequence DisplayName="Run NUnit" >
<ForEach x:TypeArguments="x:String" DisplayName="Loop through all Test Assemblies" Values="[testAssemblies]">
<ActivityAction x:TypeArguments="x:String">
<ActivityAction.Argument>
<DelegateInArgument x:TypeArguments="x:String" Name="item" />
</ActivityAction.Argument>
<Sequence >
<Sequence.Variables>
<Variable x:TypeArguments="x:String" Name="assemblyName" />
<Variable x:TypeArguments="x:String" Name="testResultXmlPath" />
<Variable x:TypeArguments="x:String" Name="variable1" />
</Sequence.Variables>
<Assign DisplayName="Get Assembly Name" >
<Assign.To>
<OutArgument x:TypeArguments="x:String">[assemblyName]</OutArgument>
</Assign.To>
<Assign.Value>
<InArgument x:TypeArguments="x:String">[item.Substring(item.LastIndexOf("\") + 1)]</InArgument>
</Assign.Value>
</Assign>
<Assign DisplayName="Get Test Result Path" >
<Assign.To>
<OutArgument x:TypeArguments="x:String">[testResultXmlPath]</OutArgument>
</Assign.To>
<Assign.Value>
<InArgument x:TypeArguments="x:String">[String.Format("{0}\{1}.ResultXml.xml", logFileDropLocation, assemblyName)]</InArgument>
</Assign.Value>
</Assign>
<mtbwa:WriteBuildMessage DisplayName="Write testResultXmlPath" Importance="[Microsoft.TeamFoundation.Build.Client.BuildMessageImportance.High]" Message="["testResultXmlPath: " + testResultXmlPath]" mva:VisualBasic.Settings="Assembly references and imported namespaces serialized as XML namespaces" />
<!-- Execute NUnit tests -->
<mtbwa:InvokeProcess Arguments="[String.Format("""{0}"" /xml:""{1}"" /nologo /nodots", item, testResultXmlPath)]" DisplayName="Execute NUnit" FileName="c:\Program Files (x86)\NUnit 2.5.10\bin\net-2.0\nunit-console.exe" >
<mtbwa:InvokeProcess.ErrorDataReceived>
<ActivityAction x:TypeArguments="x:String">
<ActivityAction.Argument>
<DelegateInArgument x:TypeArguments="x:String" Name="errOutput" />
</ActivityAction.Argument>
<mtbwa:WriteBuildError Message="[errOutput]" />
</ActivityAction>
</mtbwa:InvokeProcess.ErrorDataReceived>
<mtbwa:InvokeProcess.OutputDataReceived>
<ActivityAction x:TypeArguments="x:String">
<ActivityAction.Argument>
<DelegateInArgument x:TypeArguments="x:String" Name="stdOutput" />
</ActivityAction.Argument>
<Sequence >
<mtbwa:WriteBuildMessage Importance="[Microsoft.TeamFoundation.Build.Client.BuildMessageImportance.High]" Message="[	 stdOutput]" mva:VisualBasic.Settings="Assembly references and imported namespaces serialized as XML namespaces" />
<!-- if test output contains errors or failures -->
<If Condition="[(Not stdOutput.Contains("Failures: 0") And stdOutput.Contains("Failures:")) Or
(Not stdOutput.Contains("Errors: 0") And stdOutput.Contains("Errors:"))]" DisplayName="If there were failed tests" >
<If.Then>
<Sequence >
<!-- put build test status in failed state -->
<Assign >
<Assign.To>
<OutArgument x:TypeArguments="mtbc:BuildPhaseStatus">[BuildDetail.TestStatus]</OutArgument>
</Assign.To>
<Assign.Value>
<InArgument x:TypeArguments="mtbc:BuildPhaseStatus">[Microsoft.TeamFoundation.Build.Client.BuildPhaseStatus.Failed]</InArgument>
</Assign.Value>
</Assign>
<!-- if treatTestFailureAsBuildFailure create build error, else create build warning -->
<If Condition="[treatTestFailureAsBuildFailure]" >
<If.Then>
<mtbwa:WriteBuildError Message="[assemblyName & " -> " & stdOutput]" />
</If.Then>
<If.Else>
<mtbwa:WriteBuildWarning Message="[assemblyName & " -> " & stdOutput]" />
</If.Else>
</If>
</Sequence>
</If.Then>
</If>
</Sequence>
</ActivityAction>
</mtbwa:InvokeProcess.OutputDataReceived>
</mtbwa:InvokeProcess>
<!-- end EXECUTE NUnit tests -->
</Sequence>
</ActivityAction>
</ForEach>
</Sequence>
</If.Then>
</If>
</Sequence>
</ActivityAction>
</ForEach>
</Sequence>
</If.Then>
</If>
<!-- *********** END: Unit Testing *********** -->
Thanks Shawn!!
1 comment:
A couple of people asked me about why nUnit over mstest...
1. I love trying new things
2. Usage of datarow driven tests (TestCase attribute) is just awesome
3. Execution speed ("nunit-console runner" beats "mstest" by miles) Not an issue if you use testdriven.net
4. Readability. Assert.That is nice syntax. Note this can be achieved by some extensions methods as well e.g. "Shouldly"
Post a Comment