4 posts tagged with "npm"

To my lasting regret, TFS 2012 has no direct support for PowerShell. Such a shame as PowerShell scripts can do a lot of heavy lifting in a build process. Well, here we're going to brute force TFS 2012 into running PowerShell scripts. And along the way we'll also get Karma test results publishing into TFS 2012 as an example usage. Nice huh? Let's go!

PowerShell via csproj#

It's time to hack the csproj (or whatever project file you have) again. We're going to add an AfterBuild target to the end of the file. This target will be triggered after the build completes (as the name suggests):

There's 2 things happening in this target:

1. A message is printed out during compilation which contains details of the various compile time variables. This is nothing more than a console.log statement really; it's useful for debugging and so I keep it around. You'll notice one of them is called $(BuildNumber); more on that later.
2. A command is executed; PowerShell! This invokes PowerShell with the -NonInteractive and -ExecutionPolicy RemoteSigned flags. It passes a script to be executed called AfterBuild.ps1 that lives in the root of the project directory. To that script a number of parameters are supplied; compile time variables that we may use in the script.

Where's my BuildNumber and BuildDefinitionName?#

So you've checked in your changes and kicked off a build on the server. You're picking over the log messages and you're thinking: "Where's my BuildNumber?". Well, TFS 2012 does not have that set as a variable at compile time by default. This stumped me for a while but thankfully I wasn't the only person wondering... As ever, Stack Overflow had the answer. So, deep breath, it's time to hack the TFS build template file.

Checkout the DefaultTemplate.11.1.xaml file from TFS and open it in your text editor of choice. It's find and replace time! (There are probably 2 instances that need replacement.) Perform a find for the below

And replace it with this:

Pretty long line eh? Let's try breaking that up to make it more readable: (but remember in the XAML it needs to be a one liner)

We're just adding 2 extra parameters of BuildNumber and BuildDefinitionName to the invocation of MSBuild. Once we've checked this back in, BuildNumber and BuildDefinitionName will be available on future builds. Yay! Important! You must have a build name that does not feature spaces; probably there's a way to pass spaces here but I'm not sure what it is.

AfterBuild.ps1#

You can use your AfterBuild.ps1 script to do any number of things. In my case I'm going to use MSTest to publish some test results which have been generated by Karma into TFS:

Assuming we have a build number this script kicks off the PublishTestResults function above. So we won't attempt to publish test results when compiling in Visual Studio on our dev machine. The script looks for MSTest.exe in a certain location on disk (the default VS 2015 installation location in fact; it may be installed elsewhere on your build machine). MSTest is then invoked and passed a file called test-results.trx which is is expected to live in the root of the project. All being well, the test results will be registered with the running build and will be visible when you look at test results in TFS.

Finally in FailBuildIfThereAreTestFailures we parse the test-results.trx file (and for this I'm totally in the debt of David Robert's helpful Gist). We write out the results to the host so it'll show up in the MSBuild logs. Also, and this is crucial, if there are any failures we fail the build by exiting PowerShell with a code of 1. We are deliberately swallowing any error that Karma raises earlier when it detects failed tests. We do this because we want to publish the failed test results to TFS before we kill the build.

Bonus Karma: test-results.trx#

If you've read a previous post of mine you'll be aware that it's possible to get MSBuild to kick off npm build tasks. Specifically I have MSBuild kicking off an npm run build. My package.json looks like this:

You can see that the postbuild hook kicks off the test script in turn. And that test script kicks off a single run of karma. I'm not going to go over setting up Karma at all here; there are other posts out there that cover that admirably. But I wanted to share news of the karma trx reporter. This reporter is the thing that produces our test-results.trx file; trx being the format that TFS likes to deal with.

So now we've got a PowerShell hook into our build process (something very useful in itself) which we are using to publish Karma test results into TFS 2012. They said it couldn't be done. They were wrong. Huzzah!!!!!!!

OK - the title's total clickbait but stay with me; there's a point here.

I'm a member of the Definitely Typed team - and hopefully I won't be kicked out for writing this. My point is this: .d.ts files should live with the package they provide typing information for, in npm / GitHub etc. Not separately. TypeScript 1.6 has just been released. Yay! In the release blog post it says this:

We’ve changed module resolution when doing CommonJS output to work more closely to how Node does module resolution. If a module name is non-relative, we now follow these steps to find the associated typings:

1. Check in node_modules for <module name>.d.ts
2. Search node_modules\<module name>\package.json for a typings field
3. Look for node_modules\<module name>\index.d.ts
4. Then we go one level higher and repeat the process

Please note: when we search through node_modules, we assume these are the packaged node modules which have type information and a corresponding .js file. As such, we resolve only .d.ts files (not .ts file) for non-relative names.

Previously, we treated all module names as relative paths, and therefore we would never properly look in node_modules... We will continue to improve module resolution, including improvements to AMD, in upcoming releases.

The TL;DR is this: consuming npm packages which come with definition files should JUST WORK™... npm is now a first class citizen in TypeScriptLand. So everyone who has a package on npm should now feel duty bound to include a .d.ts when they publish and Definitely Typed can shut up shop. Simple right?

Wrong!#

Yeah, it's never going to happen. Surprising as it is, there are many people who are quite happy without TypeScript in their lives (I know - mad right?). These poor unfortunates are unlikely to ever take the extra steps necessary to write definition files. For this reason, there will probably always be a need for a provider of typings such as Definitely Typed. As well as that, the vast majority of people using TypeScript probably don't use npm to manage dependencies. There are, however, an increasing number of users who are using npm. Some (like me) may even be using tools like Browserify (with the TSIFY plugin) or WebPack (with the TS loader) to bring it all together. My feeling is that, over time, using npm will become more common; particularly given the improvements being made to module resolution in the language.

An advantage of shipping typings with an npm package is this: those typings should accurately describe their accompanying package. In Definitely Typed we only aim to support the latest and greatest typings. So if you find yourself looking for the typings of an older version of a package you're going to have to pick your way back through the history of a .d.ts file and hope you happen upon the version you're looking for. Not a fantastic experience.

So I guess what I'm saying is this: if you're an npm package author then it would be fantastic to start shipping a package with typings in the box. If you're using npm to consume packages then using Definitely Typed ought to be the second step you might take after installing a package; the step you only need to take if the package doesn't come with typings. Using DT should be a fallback, not a default.

Authoring npm modules with TypeScript#

Yup - that's what this post is actually about. See how I lured you in with my mild trolling and pulled the old switcheroo? That's edutainment my friend. So, how do we write npm packages in TypeScript and publish them with their typings? Apparently Gandhi didn't actually say "Be the change you wish to see in the world." Which is a shame. But anyway, I'm going to try and embrace the sentiment here.

Not so long ago I wrote a small npm module called globalize-so-what-cha-want. It is used to determine what parts of Globalize 1.x you need depending on the modules you're planning to use. It also, contains a little demo UI / online tool written in React which powers this.

For this post, the purpose of the package is rather irrelevant. And even though I've just told you about it, I want you to pretend that the online tool doesn't exist. Pretend I never mentioned it.

What is relevant, and what I want you to think about, is this: I wrote globalize-so-what-cha-want in plain old, honest to goodness JavaScript. Old school.

But, my love of static typing could be held in abeyance for only so long. Once the initial package was written, unit tested and published I got the itch. THIS SHOULD BE WRITTEN IN TYPESCRIPT!!! Well, it didn't have to be but I wanted it to be. Despite having used TypeScript since the early days I'd only been using it for front end work; not for writing npm packages. My mission was clear: port globalize-so-what-cha-want to TypeScript and re-publish to npm.

Port, port, port!!!#

At this point globalize-so-what-cha-want consisted of a single index.js file in the root of the package. My end goal was to end up with that file still sat there, but now generated from TypeScript. Alongside it I wanted to see a index.d.ts which was generated from the same TypeScript.

index.jsbefore looked like this:

You can even kind of tell that it was written in JavaScript thanks to the jshint rules at the top.

I fired up Atom and created a new folder src/lib and inside there I created index.ts (yes, index.js renamed) and tsconfig.json. By the way, you'll notice I'm not leaving Atom - I'm making use of the magnificent atom-typescript which you should totally be using too. It rocks.

Now I'm not going to bore you with what I had to do to port the JS to TS (not much). If you're interested, the source is here. What's more interesting is the tsconfig.json - as it's this that is going to lead the generation of the JS and TS that we need:

The things to notice are:

module

Publishing a commonjs module means it will play well with npm

declaration

This is what makes TypeScript generate index.d.ts

outDir

We want to regenerate the index.js in the root (2 directories above this)

So now, what do we get when we build in Atom? Well, we're generating an <a href="https://github.com/johnnyreilly/globalize-so-what-cha-want/blob/master/index.js">index.js</a> file which looks like this:

Aside from one method moving internally and me adding some JSDoc, the only really notable change is the end of the file. TypeScript, when generating commonjs, doesn't use the module.exports = {} approach. Rather, it drops exported functions onto the exports object as functions are exported. Functionally this is identical.

Now for our big finish: happily sat alongside is index.js is the <a href="https://github.com/johnnyreilly/globalize-so-what-cha-want/blob/master/index.d.ts">index.d.ts</a> file:

We're there, huzzah! This has been now published to npm - anyone consuming this package can use TypeScript straight out of the box. I really hope that publishing npm packages in this fashion becomes much more commonplace. Time will tell.

PS I'm not the only one#

I was just about to hit "publish" when I happened upon Basarat's ts-npm-module which is a project on GitHub which demo's how to publish and consume TypeScript using npm. I'd say great minds think alike but I'm pretty sure Basarat's mind is far greater than mine! (Cough, atom-typescript, cough.) Either way, it's good to see validation for the approach I'm suggesting.

PPS Update 23/09/2015 09:51#

One of the useful things about writing a blog is that you get to learn. Since I published I've become aware of a few things somewhat relevant to this post. First of all, there is still work ongoing in TypeScript land around this topic. Essentially there are problems resolving dependency conflicts when different dependencies have different versions - you can take part in the ongoing discussion here. There's also some useful resources to look at:

• https://github.com/Microsoft/TypeScript/wiki/Typings-for-npm-packages
• https://basarat.gitbooks.io/typescript/content/docs/node/nodejs.html

I don't know about you but I personally feel that the following sentence may well be the saddest in the English language:

2>ASPNETCOMPILER : error ASPRUNTIME: The specified path, file name, or both are too long. The fully qualified file name must be less than 260 characters, and the directory name must be less than 248 characters.

The message above would suggest there is some kind of ASP.Net issue going on. There isn't - the problem actually lies with Windows. It's not the first time it's come up but for those of you not aware there is something you need to know about Windows: It handles long paths badly.

There's a number of caveats which people may attach the above sentence. But essentially what I have said is true. And it becomes brutally apparent to you the moment you start using a few node / npm powered tools in your workflow. You will likely see that horrible message and you won't be able to get much further forward. Sigh. I thought this was the future...

This post is about how to deal with the long path issue when using npm with Visual Studio. This should very much be a short term workaround as npm 3.0 is planned to make long paths with npm a thing of the past. But until that golden dawn....

The Latest Infraction#

I'm a big fan of Gulp and Bower. They rock. Steve Cadwallader wrote an excellent blog post about integrating Gulp into your Visual Studio build. Essentially the Gist of his post is this: forget using Task Runner Explorer to trigger your Gulp / Grunt jobs. No, actually plug it into the build process by tweaking your .csproj file. The first time I used this approach it was a dream come true. It just worked and I was a very happy man.

Since this approach was so marvellous I took a look at the demo / docs part of jQuery Validation Unobtrusive Native with a view to applying it there. I originally wrote this back in 2013 and at the time used NuGet for both server and client side package management. I decided to migrate it to use Bower for the client side packages (which I planned to combine with a Gulp script which was going to pull out the required JS / CSS etc as needed). However it wasn't the plain sailing I'd imagined. The actual switchover from NuGet to Bower was simple. Just a case of removing NuGet packages and adding their associated Bower counterpart. The problem came when the migration was done and I hit "compile". That's when I got to see 2>ASPNETCOMPILER : error ASPRUNTIME: The specified path, file name, or both are too long... etc

For reasons that I don't fully understand, Visual Studio is really upset by the presence in the project structure of one almighty long path. Oddly enough, not a path that's actually part of the Visual Studio project in question at all. Rather one that has come along as a result of our Gulp / Bower / npm shenanigans. Quick as a flash, I whipped out Daniel Schroeder's Path Length Checker to see where the problem lay:

And lo, the fault lay with Bower. Poor show, Bower, poor show.

rimraf to the Rescue#

rimraf is "the UNIX commandrm -rf for node". (By the way, what is it with node and the pathological hatred of capital letters?)

What this means is: rimraf can delete. Properly. So let's get it: npm install -g rimraf. Then at any time at the command line we can dispose of a long path in 2 shakes of lamb's tail.

In my current situation the contents of the node_modules folder is causing me heartache. But with rimraf in play I can get rid of it with the magic words: rimraf ./node_modules. Alakazam! So let's poke this command into the extra commands that I've already shoplifted from Steve's blog post. I'll end up with the following section of XML at the end of my .csproj:

So let's focus on the important bits in the GulpBuild target:

• <Exec Command="npm install" /> - install the node packages our project uses as specified in package.json. This will include Gulp and Bower. The latter package is going to contain super-long, Windows wrecking paths.
• <Exec Command="bower install" /> - install the bower packages specified in bower.json using Bower (which was installed by npm just now).
• <Exec Command="gulp" /> - do a little dance, make a little love, copy a few files, get down tonight.
• <Exec Command="rimraf ./node_modules" /> - remove the node_modules folder populated by the npm install command.

With that addition of rimraf ./node_modules to the build phase the problem goes away. During each build a big, big Windows path is being constructed but then it's wiped again before it has chance to upset anyone. I've also added the same to the GulpClean target.

You are very welcome.

_{How I managed to gulp-angular-templatecache working inside Visual Studio}#

Every now and then something bites you unexpectedly. After a certain amount of pain, the answer comes to you and you know you want to save others from falling into the same deathtrap.

There I was minding my own business and having a play with a Gulp plugin called gulp-angular-templatecache. If you're not aware of it, it "Concatenates and registers AngularJS templates in the $templateCache". I was planning to use it so that all the views in an Angular app of mine were loaded up-front rather than on demand. (It's a first step in making an "offline-first" version of that particular app.)

I digress already. No sooner had I tapped in:

Then I noticed my Visual Studio project was no longer compiling. It was dying a death on build with this error:

I was dimly aware that there were issues with the nested node_modules leading to Windows-killing paths. This sounded just like that.... And it was! gulp-angular-templatecache had a dependency on gulp-footer which had a dependency on lodash.assign which had a dependency on lodash._basecreatecallback which had.... You see where I'm going? It seems that the lovely lodash has created the path from hell.

For reasons that aren't particularly clear this kills Visual Studio's build process. This is slightly surprising given that our rogue path is sat in the node_modules directory which isn't part of the project in Visual Studio. That being the case you'd imagine that you could do what you liked there. But no, it seems VS is a delicate flower and we must be careful not to offend. Strange.

It's Workaround Time!#

After a great deal of digging I found the answer nestled in the middle of an answer on Stack Overflow. To quote:

If you will add "lodash.bind" module to your project's package.json as dependency it will be installed in one level with gulp and not as gulp's dependency

That's right, I just needed to tap enter this at the root of my project:

And all was sweetness and light once more - no more complaints from VS.

The Future#

It looks like lodash are on course to address this issue. So one day this this workaround won't be necessary anymore which is good.

However, the general long path issue concerning node / npm hasn't vanished for Windows users. Given VS 2015 is planning to make Gulp and Grunt 1st class citizens of Visual Studio I'm going to guess that sort of issue is likely to arise again and again for other packages. I'm hoping that means that someone will actually fix the underlying path issues that upset Windows with npm.

It sounds like npm are planning to take some steps which is great. But I can't be alone in having a slightly nagging feeling that Windows isn't quite a first class citizen for node / io / npm yet. I really hope it will become one.