Skip to main content

2 posts tagged with "chutzpah"

View All Tags

Running JavaScript Unit Tests in AppVeyor

With a little help from Chutzpah...#

AppVeyor (if you're not aware of it) is a Continuous Integration provider. If you like, it's plug-and-play CI for .NET developers. It's lovely. And what's more it's "free for open-source projects with public repositories hosted on GitHub and BitBucket". Boom! I recently hooked up 2 of my GitHub projects with AppVeyor. It took me all of... 10 minutes. If that? It really is *that* good.

But.... There had to be a "but" otherwise I wouldn't have been writing the post you're reading. For a little side project of mine called Proverb there were C# unit tests and there were JavaScript unit tests. And the JavaScript unit tests weren't being run... No fair!!!

Chutzpah is a JavaScript test runner which at this point runs QUnit, Jasmine and Mocha JavaScript tests. I use the Visual Studio extension to run Jasmine tests on my machine during development. I've also been able to use Chutzpah for CI purposes with Visual Studio Online / Team Foundation Server. So what say we try and do the triple and make it work with AppVeyor too?

NuGet me?#

In order that I could run Chutzpah I needed Chutzpah to be installed on the build machine. So I had 2 choices:

  1. Add Chutzpah direct to the repo
  2. Add the Chutzpah Nuget package to the solution

Unsurprisingly I chose #2 - much cleaner.

Now to use Chutzpah#

Time to dust down the PowerShell. I created myself a "before tests script" and added it to my build. It looked a little something like this:

# Locate Chutzpah
$ChutzpahDir = get-childitem chutzpah.console.exe -recurse | select-object -first 1 | select -expand Directory
# Run tests using Chutzpah and export results as JUnit format to chutzpah-results.xml
$ChutzpahCmd = "$($ChutzpahDir)\chutzpah.console.exe $($env:APPVEYOR_BUILD_FOLDER)\AngularTypeScript\Proverb.Web.Tests.JavaScript /junit .\chutzpah-results.xml"
Write-Host $ChutzpahCmd
Invoke-Expression $ChutzpahCmd
# Upload results to AppVeyor one by one
$testsuites = [xml](get-content .\chutzpah-results.xml)
$anyFailures = $FALSE
foreach ($testsuite in $testsuites.testsuites.testsuite) {
write-host " $($"
foreach ($testcase in $testsuite.testcase){
$failed = $testcase.failure
$time = $testsuite.time
if ($testcase.time) { $time = $testcase.time }
if ($failed) {
write-host "Failed $($ $($testcase.failure.message)"
Add-AppveyorTest $ -Outcome Failed -FileName $ -ErrorMessage $testcase.failure.message -Duration $time
$anyFailures = $TRUE
else {
write-host "Passed $($"
Add-AppveyorTest $ -Outcome Passed -FileName $ -Duration $time
if ($anyFailures -eq $TRUE){
write-host "Failing build as there are broken tests"

What this does is:

  1. Run Chutzpah from the installed NuGet package location, passing in the location of my Jasmine unit tests. In the case of my project there is a chutzpah.json file in the project which dictates how Chutzpah should run the tests. Also, the JUnit flag is also passed in order that Chutzpah creates a chutzpah-results.xml file of test results in the JUnit format.
  2. We iterate through test results and tell AppVeyor about the the test passes and failures using the Build Worker API.
  3. If there have been any failed tests then we fail the build. If you look here you can see a deliberately failed build which demo's that this works as it should.

That's a wrap - We now have CI which includes our JavaScript tests! That's right we get to see beautiful screens like these:

Thanks to...#

Thanks to Dan Jones, whose comments on this discussion provided a number of useful pointers which moved me in the right direction. And thanks to Feador Fitzner who has generously said AppVeyor will support JUnit in the future which may simplify use of Chutzpah with AppVeyor even further.

Team Foundation Server, Continuous Integration and separate projects for JavaScript unit tests

Do you like to separate out your unit tests from the project you are testing? I imagine so. My own practice when creating a new project in Visual Studio is to create a separate unit test project alongside whose responsibility is to house unit tests for that new project.

When I check in code for that project I expect the continuous integration build to kick off and, as part of that, the unit tests to be run. When it comes to running .NET tests then Team Foundation Server (and it's cloud counterpart Visual Studio Online) has your back. When it comes to running JavaScript tests then... not so much.

This post will set out:

  1. How to get JavaScript tests to run on TFS / VSO in a continuous integration scenario.
  2. How to achieve this *without* having to include your tests as part of web project.

To do this I will lean heavily (that's fancy language for "rip off entirely") on an excellent blog post by Mathew Aniyan which covers point #1. My contribution is point #2.

Points #1 and #2 in short order#

First of all, install Chutzpah on TFS / VSO. You can do this by following Steps 1 - 6 from Mathew Aniyan's post. Instead of following steps 7 and 8 create a new unit test project in your solution.

Edit 29/05/2014: Matthew Manela (creator of Chutzpah) has confirmed that this is the correct approach - thanks chap!

@johnny_reilly Nope that is pretty much what you need to do.

โ€” Matthew Manela (@mmanela) May 15, 2014

To our unit test project add your JavaScript unit tests. These should be marked in Visual Studio with a Build Action of "Content" and a Copy to Output Directory of "Do not copy". You should be able to run these tests locally using the Visual Studio Chutzpah extension - or indeed in some other JavaScript test runner. Then, and this is the important part, edit the csproj file of your unit test project and add this Import Project statement:

<Import Project="$(VSToolsPath)\WebApplications\Microsoft.WebApplication.targets" Condition="'$(VSToolsPath)' != ''" />

Ordering is important in this case. It matters that this new statement sits after the other Import Project statements. So you should end up with a csproj file that looks in part like this: (comments added by me for clarity)

<!-- Pre-existing Import Project statements start -->
<Import Project="$(VSToolsPath)\TeamTest\Microsoft.TestTools.targets" Condition="Exists('$(VSToolsPath)\TeamTest\Microsoft.TestTools.targets')" />
<Import Project="$(MSBuildToolsPath)\Microsoft.CSharp.targets" />
<!-- Pre-existing Import Project statements end -->
<!-- New addition start -->
<Import Project="$(VSToolsPath)\WebApplications\Microsoft.WebApplication.targets" Condition="'$(VSToolsPath)' != ''" />
<!-- New addition end -->

Check in your amended csproj and the unit tests to TFS / VSO. You should see the JavaScript unit tests being run as part of the build.