Friday, 27 February 2015

Hey tsconfig.json, where have you been all my life?

Sometimes, you just miss things. Something seismic happens and you had no idea. So it was with tsconfig.json.

This blog post started life with the name "TypeScript: Some IDEs are more equal than others". I'd intended to use it summarise a discussion on the TypeScript GitHub repo about implicit referencing including a fist shaken at the sky at the injustice of it all. But whilst I was writing it I dicovered things had changed without my knowledge. That's a rather wonderful thing.

Implicit Referencing

Implicit referencing, if you're not aware, is the thing that separates Visual Studio from all other IDEs / text editors. Implicit referencing means that in Visual Studio you don't need to make use of comments at the head of each TypeScript file in order to tell the compiler where it can find the related TypeScript files.

The reference comments aren't necessary when using Visual Studio because the VS project file is used to drive the files passed to the TypeScript compiler (tsc).

The upshot of this is that, at time of writing, you can generally look at a TypeScript codebase and tell whether it was written using Visual Studio by opening it up a file at random and eyeballing for something like this at the top:

/// <reference path="other-file.ts" />

"A-ha! They're using "reference" comments Watson. From this I deduce that the individuals in question are using the internal module approach and using Visual Studio as their IDE. Elementary, my dear fellow, quite elementary."

This has important implications. Important I tell you, yes important! Well, important if you want to reduce the barriers between Visual Studio and everyone else. And I do. Whilst I love Visual Studio - it's been my daily workhorse for many years - I also love stepping away from it and using something more stripped down. I also like working with other people without mandating that they need to use Visual Studio as well. In the words of Rodney King, "can't we all get along?".

Cross-IDE TypeScript projects

I feel I should be clear - you can already set up TypeScript projects to work regardless of IDE. But there's friction. It's not clear cut. You can see a full on discussion around this here but in the end it comes down to making a choice between these 3 options:

  1. Set <TypeScriptEnabled>false</TypeScriptEnabled> in a project file. This flag effectively deactivates implicit referencing. This approach requires that all developers (regardless of IDE) use /// <references to build context. Compiler options in VS can be controlled using the project file as is.
  2. Using Visual Studio without any csproj tweaks. This approach requires that all files will need /// <references at their heads in order to build compilation context outside of Visual Studio. It's possible that /// <references and the csproj could get out of line - care is required to avoid this. Compiler options in VS can be controlled using the project file as is.
  3. Using just files in Visual Studio with /// <references to build compilation context. This scenario also requires that all developers (regardless of IDE) use /// <references to build context. In Visual Studio there will be no control over compiler options.

As you can see - this is sub-optimal. But don't worry - there's a new sheriff in town....

tsconfig.json

I'd decided to give Atom TypeScript plugin a go as I heard much enthusiastic noise about it. I fired it up and pointed it at a a TypeScript AngularJS project built in Visual Studio. I was mentally preparing myself for the job of adding all the /// references in when I suddenly noticed a file blinking at me:

tsconfig.json? What's that? Time to read the docs:

Supported via tsconfig.json (read more) which is going to be the defacto Project file format for the next versions of TypeScript.

"read more"? Oh yes indeedy - I think I will "read more"!

A unified project format for TypeScript (see merged PR on Microsoft/TypeScript). The TypeScript compiler (1.4 and above) only cares about compilerOptions and files. We add additional features to this with the typescript team's approval to extend the file as long as we don't conflict:

  • compilerOptions similar to what you would pass on the commandline to tsc.
  • filesGlob: To make it easier for you to just add / remove files in your project we add filesGlob which accepts an array of glob / minimatch / RegExp patterns (similar to grunt)to specify source files.
  • format: Code formatting options
  • version: The TypeScript version

That's right folks, we don't need /// <references comments anymore. In a blinding flash of light it all changes. We're going from the dark end of the street, to the bright side of the road. tsconfig.json is here to ease away the pain and make it all better. Let's enjoy it while we can.

This change should ship with TypeScript 1.5 (hopefully) for those using Visual Studio. For those using Atom TypeScript (and as of today that's includes me) the carnival celebrations can begin now!

Thanks to @basarat who have quoted at length and Daniel Earwicker who is the reason that I came to discover tsconfig.json.

Tuesday, 17 February 2015

Using Gulp to inject scripts and styles tags directly into your HTML

This is very probably the dullest title for a blog post I've ever come up with. Read on though folks - it's definitely going to pick up...

I wrote last year about my first usage of Gulp in an ASP.Net project. I used Gulp to replace the Web Optimization functionality that is due to disappear when ASP.Net v5 ships. What I came up with was an approach that provided pretty much the same functionality; raw source in debug mode, bundling + minification in release mode.

It worked by having a launch page which was straight HTML. Embedded within this page was JavaScript that would, at runtime, load the required JavaScript / CSS and inject it dynamically into the document. This approach worked but it had a number of downsides:

  1. Each time you fired up the app the following sequence of events would happen:
    • jQuery would load (purely there to simplify the making of various startup AJAX calls)
    • the page would make an AJAX call to the server to load various startup data, including whether the app is running in debug or release mode
    • Depending on the result of the startup data either the debug or release package manifest would be loaded.
    • For each entry in the package manifest script and link tags would be created and added to the document. These would generate further requests to the server to load the resources.
    Quite a lot going on here isn't there? Accordingly, initial startup time was slower than you might hope.
  2. The "F" word: FOUC. Flash Of Unstyled Content - whilst all the hard work of the page load was going on (before the CSS had been loaded) the page would look rather ... bare. Not a terrible thing but none too slick either.
  3. The gulpfile built both the debug and the release package each time it was run. This meant the gulp task generally did double the work that it needed to do.

I wanted to see if I could tackle these issues. I've recently been watching John Papa's excellent Pluralsight course on Gulp and picked up a number of useful tips. With that in hand let's see what we can come up with...

Death to dynamic loading

The main issue with the approach I've been using is the dynamic loading. It makes the app slower and more complicated. So the obvious solution is to have my gulpfile inject scripts and css into the template. To that end it's wiredep & gulp-inject to the rescue!

gulp-inject (as the name suggests) is used to inject script and link tags into source code. I'm using Bower as my client side package manager and so I'm going to use wiredep to determine the vendor scripts I need. It will determine what packages my app is using from looking at my bower.json, and give me a list of file paths in dependency order (which I can then pass on to gulp-inject in combination with my own app script files). This means I don't have to think about ordering bower dependencies myself and I no longer need to separately maintain a list of these files within my gulpfile.

So, let's get the launch page (index.html) ready for gulp-inject:

<!DOCTYPE html>
<html>
<head>
    <meta http-equiv="X-UA-Compatible" content="IE=edge, chrome=1" />
    <style>
        .ng-hide {
            display: none !important;
        }
    </style>
    <title ng-bind="title">Proverb</title>
    <meta charset="utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1, minimum-scale=1, user-scalable=no" />

    <!-- inject:css -->
    <!-- endinject -->

    <link rel="icon" type="image/png" href="content/images/icon.png">
</head>
<body>
    <div>
        <div ng-include="'app/layout/shell.html'"></div>
        <div id="splash-page" ng-show="false" class="dissolve-animation">
            <div class="page-splash">
                <div class="page-splash-message">
                    Proverb
                </div>

                <div class="progress">
                    <div class="progress-bar progress-bar-striped active" role="progressbar" style="width: 20%;">
                        <span class="sr-only">loading...</span>
                    </div>
                </div>
            </div>
        </div>
    </div>

    <script src="https://ajax.googleapis.com/ajax/libs/jquery/2.1.3/jquery.min.js"></script>
    <script>window.jQuery || document.write('<script src="/build/jquery.min.js">\x3C/script>')</script>

    <!-- inject:js -->
    <!-- endinject -->

    <script>
        (function () {

            // Load startup data from the server
            $.getJSON("api/Startup")
                .done(function (startUpData) {

                    angularApp.start({
                        thirdPartyLibs: {
                            moment: window.moment,
                            toastr: window.toastr,
                            underscore: window._
                        },
                        appConfig: startUpData
                    });
                });
        })();
    </script>
</body>
</html>

The important thing to notice here are the <!-- inject:css --> and <!-- inject:js --> injection placeholders. It's here that our script and style tags will be injected into the template. You'll notice that jQuery is not being injected - and that's because I've opted to use a CDN for jQuery and then only fallback to serving jQuery myself if the CDN fails.

The other thing to notice here is that our launch page has become oh so much simpler in comparison with the dynamic loading approach. Which is fab.

Now before we start looking at our gulpfile I want to split out the configuration into a standalone file called gulpfile.config.js:

var tsjsmapjsSuffix = ".{ts,js.map,js}";

var bower = "bower_components/";
var app = "app/";

var config = {

    base: ".",
    buildDir: "./build/",
    debug: "debug",
    release: "release",
    css: "css",

    bootFile: app + "index.html",
    bootjQuery: bower + "jquery/dist/jquery.min.js",

    // The fonts we want Gulp to process
    fonts: [bower + "fontawesome/fonts/*.*"],

    images: "images/**/*.{gif,jpg,png}",

    // The scripts we want Gulp to process
    scripts: [
        // Bootstrapping
        app + "app" + tsjsmapjsSuffix,
        app + "config.route" + tsjsmapjsSuffix,

        // common Modules
        app + "common/common" + tsjsmapjsSuffix,
        app + "common/logger" + tsjsmapjsSuffix,
        app + "common/spinner" + tsjsmapjsSuffix,

        // common.bootstrap Modules
        app + "common/bootstrap/bootstrap.dialog" + tsjsmapjsSuffix,

        // directives
        app + "directives/**/*" + tsjsmapjsSuffix,

        // services
        app + "services/**/*" + tsjsmapjsSuffix,

        // controllers
        app + "about/**/*" + tsjsmapjsSuffix,
        app + "admin/**/*" + tsjsmapjsSuffix,
        app + "dashboard/**/*" + tsjsmapjsSuffix,
        app + "layout/**/*" + tsjsmapjsSuffix,
        app + "sages/**/*" + tsjsmapjsSuffix,
        app + "sayings/**/*" + tsjsmapjsSuffix
    ],

    // The styles we want Gulp to process
    styles: [
        "content/styles.css"
    ],

    wiredepOptions: {
        exclude: [/jquery/],
        ignorePath: ".."
    }
};

config.debugFolder = config.buildDir + config.debug + "/";
config.releaseFolder = config.buildDir + config.release + "/";

config.templateFiles = [
    app + "**/*.html",
    "!" + config.bootFile // Exclude the launch page
];

module.exports = config;

Now to the meat of the matter - let me present the gulpfile:

/// <vs AfterBuild='default' />
var gulp = require("gulp");

// Include Our Plugins
var concat = require("gulp-concat");
var ignore = require("gulp-ignore");
var minifyCss = require("gulp-minify-css");
var uglify = require("gulp-uglify");
var rev = require("gulp-rev");
var del = require("del");
var path = require("path");
var templateCache = require("gulp-angular-templatecache");
var eventStream = require("event-stream");
var order = require("gulp-order");
var gulpUtil = require("gulp-util");
var wiredep = require("wiredep");
var inject = require("gulp-inject");

// Get our config
var config = require("./gulpfile.config.js");

/**
 * Get the scripts or styles the app requires by combining bower dependencies and app dependencies
 * 
 * @param {string} jsOrCss Should be "js" or "css"
 */
function getScriptsOrStyles(jsOrCss) {

    var bowerScriptsAbsolute = wiredep(config.wiredepOptions)[jsOrCss];

    var bowerScriptsRelative = bowerScriptsAbsolute.map(function makePathRelativeToCwd(file) {
        return path.relative('', file); 
    });

    var appScripts = bowerScriptsRelative.concat(jsOrCss === "js" ? config.scripts : config.styles);

    return appScripts;
}

/**
 * Get the scripts the app requires
 */
function getScripts() {

    return getScriptsOrStyles("js");
}

/**
 * Get the styles the app requires
 */
function getStyles() {

    return getScriptsOrStyles("css");
}

/**
 * Get the scripts and the templates combined streams
 * 
 * @param {boolean} isDebug
 */
function getScriptsAndTemplates(isDebug) {

    var options = isDebug ? { base: config.base } : undefined;
    var appScripts = gulp.src(getScripts(), options);

    //Get the view templates for $templateCache
    var templates = gulp.src(config.templateFiles)
        .pipe(templateCache({ module: "app", root: "app/" }));

    var combined = eventStream.merge(appScripts, templates);

    return combined;
}

gulp.task("clean", function (cb) {

    gulpUtil.log("Delete the build folder");

    return del([config.buildDir], cb);
});

gulp.task("boot-dependencies", ["clean"], function () {

    gulpUtil.log("Get dependencies needed for boot (jQuery and images)");

    var jQuery = gulp.src(config.bootjQuery);
    var images = gulp.src(config.images, { base: config.base });

    var combined = eventStream.merge(jQuery, images)
        .pipe(gulp.dest(config.buildDir));

    return combined;
});

gulp.task("inject-debug", ["styles-debug", "scripts-debug"], function () {

    gulpUtil.log("Inject debug links and script tags into " + config.bootFile);

    var scriptsAndStyles = [].concat(getScripts(), getStyles());

    return gulp
        .src(config.bootFile)
        .pipe(inject(
                gulp.src([
                        config.debugFolder + "**/*.{js,css}",
                        "!build\\debug\\bower_components\\spin.js" // Exclude weird spin js path
                    ], { read: false })
                    .pipe(order(scriptsAndStyles))
            ))
        .pipe(gulp.dest(config.buildDir));
});

gulp.task("inject-release", ["styles-release", "scripts-release"], function () {

    gulpUtil.log("Inject release links and script tags into " + config.bootFile);

    return gulp
        .src(config.bootFile)
        .pipe(inject(gulp.src(config.releaseFolder + "**/*.{js,css}", { read: false })))
        .pipe(gulp.dest(config.buildDir));
});

gulp.task("scripts-debug", ["clean"], function () {

    gulpUtil.log("Copy across all JavaScript files to build/debug");

    return getScriptsAndTemplates(true)
        .pipe(gulp.dest(config.debugFolder));
});

gulp.task("scripts-release", ["clean"], function () {

    gulpUtil.log("Concatenate & Minify JS for release into a single file");

    return getScriptsAndTemplates(false)
        .pipe(ignore.exclude("**/*.{ts,js.map}")) // Exclude ts and js.map files - not needed in release mode
        .pipe(concat("app.js"))                   // Make a single file                                                         
        .pipe(uglify())                           // Make the file titchy tiny small
        .pipe(rev())                              // Suffix a version number to it
        .pipe(gulp.dest(config.releaseFolder));   // Write single versioned file to build/release folder
});

gulp.task("styles-debug", ["clean"], function () {

    gulpUtil.log("Copy across all CSS files to build/debug");

    return gulp
        .src(getStyles(), { base: config.base })
        .pipe(gulp.dest(config.debugFolder));
});

gulp.task("styles-release", ["clean"], function () {

    gulpUtil.log("Copy across all files in config.styles to build/debug");

    return gulp
        .src(getStyles())
        .pipe(concat("app.css"))                                   // Make a single file
        .pipe(minifyCss())                                         // Make the file titchy tiny small
        .pipe(rev())                                               // Suffix a version number to it
        .pipe(gulp.dest(config.releaseFolder + "/" + config.css)); // Write single versioned file to build/release folder
});

gulp.task("fonts-debug", ["clean"], function () {

    gulpUtil.log("Copy across all fonts in config.fonts to debug location");

    return gulp
        .src(config.fonts, { base: config.base })
        .pipe(gulp.dest(config.debugFolder));
});

gulp.task("fonts-release", ["clean"], function () {

    gulpUtil.log("Copy across all fonts in config.fonts to release location");

    return gulp
        .src(config.fonts)
        .pipe(gulp.dest(config.releaseFolder + "/fonts"));
});

gulp.task("build-debug", [
    "boot-dependencies", "inject-debug", "fonts-debug"
]);

gulp.task("build-release", [
    "boot-dependencies", "inject-release", "fonts-release"
]);

// Use the web.config to determine whether the default task should create a debug or a release build
// If the web.config contains this: '<compilation debug="true"' then we do a default build, otherwise
// we do a release build.  It's a little hacky but generally works
var fs = require('fs');
var data = fs.readFileSync(__dirname + "/web.config", "UTF-8");
var inDebug = !!data.match(/<compilation debug="true"/); 

gulp.task("default", [(inDebug ? "build-debug" : "build-release")]);

That's a big old lump of code. So let's go through this a task by task...

clean

Deletes the build folder so we have a clean slate to build into.

boot-dependencies

Copy across all files that are needed to allow the page to "boot" / startup. At present this is only jQuery and images.

inject-debug and inject-release

This is the magic. This picks up the launch page (index.html), takes the JavaScript and CSS and injects the corresponding script and link tags into the page and writing it to the build folder. Either the original source code or the bundled / minified equivalent will be used depending on whether it's debug or release.

scripts-debug and scripts-release

Here we collect up the following:

  • the Bower specified JavaScript files
  • the TypeScript + associated JavaScript files
  • and we use our template files to construct a templates.js file to prime the Angular template cache

If it's the scripts-debug task we copy all these files into the build/debug folder. If it's the scripts-release task we also bundle, minify and strip the TypeScript out too and copy into the build/release folder.

styles-debug and styles-release

Here we collect up the following:

  • the Bower specified CSS files
  • our own app CSS

If it's the styles-debug task we copy all these files into the build/debug folder. If it's the styles-release task we also bundle and minify and copy into the build/release folder.

fonts-debug and fonts-release

Whether it's the debug or the release build we copy across the font-awesome assets and place them in a location which works for the associated CSS (as the CSS will depend upon font-awesome).

build-debug, build-release and default

build-debug and build-release (as their name suggests) either perform a build for release or a build for debug. If you remember, the web optimization library in ASP.Net serves up the raw code ("debug" code) if the compilation debug flag in the web.config is set to true. If it is set to false then we get the bundled and minified code ("release" code) instead. Our default task tries its best to emulate this behaviour by doing a very blunt regex against the web.config. Simply, if it can match <compilation debug="true" then it runs the debug build. Otherwise, the release build. It could be more elegant but there's a dearth of XML readers on npm that support synchronous parsing (which you kinda need for this scenario).

What I intend to do soon is switch from using the web.config to drive the gulp build to using the approach outlined here. Namely plugging the build directly into Visual Studio's build process and using the type of build there.

Hopefully what I've written here makes it fairly clear how to use Gulp to directly inject scripts and styles directly into your HTML. If you want to look directly at the source then check out the Proverb.Web folder in this repo.

Wednesday, 11 February 2015

The Convent with Continuous Delivery

I've done it. I've open sourced the website that I maintain for my aunt what is a nun. Because I think we can all agree that nuns need open source and continuous integration about as much as anyone else.

For a long time now I've been maintaining a website for one of my (many) aunts that is a Poor Clare. (That's a subtype of "nun" you OO enthusiasts.) It's not a terribly exciting site - it's mostly static content. It's built with a combination of AngularJS / TypeScript / Bootstrap and ASP.Net MVC. It's hosted on Azure Websites. In fact I have written about it (slightly more cagily) before here.

I'll say up front: presentation-wise the site is not a work of art. However the nuns seem pretty happy with it. (Or perhaps secretly they're forgiving me the shonkiness and sparing my feelings - who can say?) If I put my mind to it the site could look much more lovely. But there's only so much time I can spare - and that's actually one of the reasons I've set up Continuous Delivery.

Why on earth did you bother?

Well, you'd be surprised how often tweaks can be requested. Sometimes it appears to be forgotten for months at a time, and then all of a sudden my inbox is daily filled with a list of minor alterations. You know, slight text changes and the like.

So what I was generally doing was getting home of an evening, waiting until the children were in bed, chomping down some food and then firing up Visual Studio to make the changes and hit "Publish". Yes that's right; I was essentially using Visual Studio to edit text files and push a website out to Azure. The very definition of using a sledgehammer to crack a nut I think we can all agree.

It occurred to me that if I had Continuous Delivery set up then I could make these tweaks and not have to worry about the site being published. Which would be nice. I wouldn't need Visual Studio anymore - any text editor would do. Also nice. Finally, if the source control was accessible online then I could probably get away with doing most tweaks on my mobile phone whilst I was travelling home. Timesaver!

How did you go about it?

Since Visual Studio Online (then "Team Foundation Service") was released I have been using it to host the source code. So the obvious solution was to use the tools offered there to do the deployment. However, this wasn't the smooth experience you might have hoped for. I had quite a frustrating afternoon trying things out before deciding it was becoming more trouble than it was worth. VSO appeared to make it supremely hard to customise builds.

Just recently though I have been having the most wonderful experience with AppVeyor. AppVeyor market themselves as "#1 Continuous Delivery service for Windows" - I think they're right. Their build process is entirely flexible and customisable. It is, in short, a joy to use. (The support is fantastic too - very helpful indeed. Go Feodor!)

If you look just below the header you'll read a very important sentence: "Free for open-source projects". You hear that? By the time I'd finished reading that sentence I'd decided that the Poor Clares website was about to become an open source project.

And now it is.

Where is it?

The source on GitHub. The builds and deployment are taken care of by AppVeyor.

Will you take pull requests?

If they're serious, then yes, certainly! My long term plan is to try and get the nuns set up as collaborators in GitHub. That way they can make their own minor tweaks without me getting involved.

On another front, I do wonder if open-sourcing Poor Clares, Arundel might have other hidden benefits. There's a number of things I'm not too keen on in the code. Up until now I think my attitude was possibly "it works so that's good enough". It was only me aware of the shortcomings. Now it's public I'll probably have more of an incentive to tidy up the rough edges. That's the theory anyway - Embarrassment Driven Development anyone? :-)