Skip to main content

2 posts tagged with "thread-loader"

View All Tags

You Might Not Need thread-loader

It all started with a GitHub issue. Ernst Ammann reported:

Without the thread-loader, compilation takes three to four times less time on changes. We could remove it.

If you're not aware of the webpack-config-plugins project then I commend it to you. Famously, webpack configuration can prove tricky. webpack-config-plugins borrows the idea of presets from Babel. It provides a number of pluggable webpack configurations which give a best practice setup for different webpack use cases. So if you're no expert with webpack and you want a good setup for building your TypeScript / Sass / JavaScript then webpack-config-plugins has got your back.

One of the people behind the project is the very excellent Jan Nicklas who is well known for his work on the html-webpack-plugin.

It was Jan who responded to Ernst's issue and decided to look into it.

All I Want For Christmas is Faster Builds#

Everyone wants fast builds. I do. You do. We all do. webpack-config-plugins is about giving these to the user in a precooked package.

There's a webpack loader called thread-loader which spawns multiple processes and splits up work between them. It was originally inspired by the work in the happypack project which does a similar thing.

I wrote a blog post some time ago which gave details about ways to speed up your TypeScript builds by combining the ts-loader project (which I manage) with the fork-ts-checker-webpack-plugin project (which I'm heavily involved with).

That post was written back in the days of webpack 2 / 3. It advocated use of both happypack / thread-loader to drop your build times even further. As you'll see, now that we're well into the world of webpack 4 (with webpack 5 waiting in the wings) the advantage of happypack / thread-loader are no longer so profound.

webpack-config-plugins follows the advice I set out in my post; it uses thread-loader in its pluggable configurations. Now, back to Ernst's issue.

thread-loader: Infinity War#

Jan quickly identified the problem. He did that rarest of things; he read the documentation which said:

// timeout for killing the worker processes when idle
// defaults to 500 (ms)
// can be set to Infinity for watching builds to keep workers alive
poolTimeout: 2000,

The webpack-config-plugins configurations (running in watch mode) were subject to the thread loaders being killed after 500ms. They got resurrected when they were next needed; but that's not as instant as you might hope. Jan then did a test:

(default pool - 30 runs - 1000 components ) average: 2.668068965517241
(no thread-loader - 30 runs - 1000 components ) average: 1.2674137931034484
(Infinity pool - 30 runs - 1000 components ) average: 1.371827586206896

This demonstrates that using thread-loader in watch mode with poolTimeout: Infinity performs significantly better than when it defaults to 500ms. But perhaps more significantly, not using thread-loader performs even better still.

"Maybe You've Thread Enough"#

When I tested using thread-loader in watch mode with poolTimeout: Infinity on my own builds I got the same benefit Jan had. I also got even more benefit from dropping thread-loader entirely.

A likely reason for this benefit is that typically when you're developing, you're working on one file at a time. Hence you only transpile one file at a time:

So there's not a great deal of value that thread-loader can add here; mostly it's twiddling thumbs and adding an overhead. To quote the docs:

Each worker is a separate node.js process, which has an overhead of ~600ms. There is also an overhead of inter-process communication.

Use this loader only for expensive operations!

Now, my build is not your build. I can't guarantee that you'll get the same results as Jan and I experienced; but I would encourage you to investigate if you're using thread-loader correctly and whether it's actually helping you. In these days of webpack 4+ perhaps it isn't.

There are still scenarios where thread-loader still provides an advantage. It can speed up production builds. It can speed up the initial startup of watch mode. In fact Jan has subsequently actually improved the thread-loader to that specific end. Yay Jan!

If this is all too much for you, and you want to hand off the concern to someone else then perhaps all of this serves as a motivation to just sit back, put your feet up and start using webpack-config-plugins instead of doing your own configuration.

TypeScript + Webpack: Super Pursuit Mode

This post also featured as a webpack Medium publication.

If you're like me then you'll like TypeScript and you'll like module bundling with webpack. You may also like speedy builds. That's completely understandable. The fact of the matter is, you sacrifice a bit of build speed to have webpack in the mix. Wouldn't it be great if we could even up the difference?

I'm the primary maintainer of ts-loader, a TypeScript loader for webpack. Just recently a couple of PRs were submitted that said, in other words: ts-loader is like this:

But it could be like this:

Apologies for the image quality above; there appear to be no high quality pictures out there of KITT in Super Pursuit Mode for me to defame with Garan Jenkin's atrocious puns.

fork-ts-checker-webpack-plugin#

"Faster type checking with forked process" read the enticing name of the issue. It turned out to be Piotr Oleś (@OlesDev) telling the world about his beautiful creation. He'd put together a mighty fine plugin that can be used alongside ts-loader called the fork-ts-checker-webpack-plugin. The name is a bit of a mouthful but the purpose is mouth-watering. To quote the README, it is a:

Webpack plugin that runs typescript type checker on a separate process.

What does this mean and how does this fit with ts-loader? Well, ts-loader does 2 jobs:

  1. It transpiles your TypeScript into JavaScript and hands it off to webpack
  2. It collects any TypeScript compilation errors and reports them to webpack

What this plugin does is say, "forget about #2 - we've got this." It removes the responsibility for type checking from ts-loader, so the only work ts-loader does is transpilation. In the meantime, the all important type checking is still happening. To be honest, there would be little reason to recommend this approach otherwise. The difference is fork-ts-checker-webpack-plugin is doing the heavy lifting in a separate process. This provides a nice performance boost to your workflow. ts-loader is doing less and that's a good thing

.

The approach used here is similar to that employed by awesome-typescript-loader. ATL is another TypeScript loader for webpack by the excellent Stanislav Panferov. ATL also has a technique for performing typechecking in a forked process. fork-ts-checker-webpack-plugin was an effort by Piotr to implement something similar but with improved incremental build performance.

How do we use it? Add fork-ts-checker-webpack-plugin as a devDependency of your project and then amend the webpack.config.js to set ts-loader into transpileOnly mode and drop the plugin into the mix:

var ForkTsCheckerWebpackPlugin = require('fork-ts-checker-webpack-plugin');
var webpackConfig = {
// other config...
context: __dirname, // to automatically find tsconfig.json
module: {
rules: [
{
test: /\.tsx?$/,
loader: 'ts-loader',
options: {
// disable type checker - we will use it in fork plugin
transpileOnly: true
}
}
]
},
plugins: [
new ForkTsCheckerWebpackPlugin()
]
};

If you'd like to see an example of how to use the plugin then take a look at a simple example and a more involved one.

HappyPack#

Not so long ago I didn't know what happyness

HappyPack was. "Happiness in the form of faster webpack build times." That's what it is.

HappyPack makes webpack builds faster by allowing you to transform multiple files in parallel.

It does this by spinning up multiple threads, each with their own loaders inside. We wanted to do this with ts-loader; to have multiple instances of ts-loader running. Work can then be divided up across these separate loaders. Isn't multi-threading great?

ts-loader did not initially play nicely with HappyPack; essentially this is because ts-loader touches parts of webpack's API that HappyPack replaces. The entirely wonderful Artem Kozlov submitted a PR which added HappyPack support to ts-loader. Support essentially amounts to switching ts-loader to run in transpileOnly mode and ensuring that there is no attempt to talk to parts of the webpack API that HappyPack removes.

It would be hard to recommend using HappyPack as is because, as with transpileOnly mode you lose all typechecking. Where it becomes worthwhile is where it is combined with the fork-ts-checker-webpack-plugin so you keep the typechecking.

Enough with the chitter chatter; how can we achieve this? Add HappyPack as a devDependency of your project and then amend the webpack.config.js as follows:

var HappyPack = require('happypack');
var ForkTsCheckerWebpackPlugin = require('fork-ts-checker-webpack-plugin');
module.exports = {
// other config...
context: __dirname, // to automatically find tsconfig.json
module: {
rules: [
{
test: /\.tsx?$/,
exclude: /node_modules/,
loader: 'happypack/loader?id=ts'
}
]
},
plugins: [
new HappyPack({
id: 'ts',
threads: 2,
loaders: [
{
path: 'ts-loader',
query: { happyPackMode: true }
}
]
}),
new ForkTsCheckerWebpackPlugin({ checkSyntacticErrors: true })
]
};

Note that the ts-loader options are now configured via the HappyPack query and that we're setting ts-loader with the happyPackMode option set.

There's one other thing to note which is important; we're now passing the checkSyntacticErrors option to the fork plugin. This ensures that the plugin checks for both syntactic errors (eg const array = [{} {}];) and semantic errors (eg const x: number = '1';). By default the plugin only checks for semantic errors. This is because when ts-loader is used with transpileOnly set, ts-loader will still report syntactic errors. But when used in happyPackMode it does not.

If you'd like to see an example of how to use HappyPack then once again we have a simple example and a more involved one.

thread-loader + cache-loader#

You might have some reservations about using HappyPack. First of all the quirky configuration required makes your webpack config rather less comprehensible. Also, HappyPack is not officially blessed by webpack. It is a side project developed externally from webpack and there's no guarantees that new versions of webpack won't break it. Neither of these are reasons not to use HappyPack but they are things to bear in mind.

What if there were a way to parallelise our builds which dealt with these issues? Well, there is! By using thread-loader and cache-loader in combination you can both feel happy that you're using an official webpack workflow and you can have a config that's less confusing.

What would that config look like? This:

var ForkTsCheckerWebpackPlugin = require('fork-ts-checker-webpack-plugin');
module.exports = {
// other config...
context: __dirname, // to automatically find tsconfig.json
module: {
rules: {
test: /\.tsx?$/,
use: [
{ loader: 'cache-loader' },
{
loader: 'thread-loader',
options: {
// there should be 1 cpu for the fork-ts-checker-webpack-plugin
workers: require('os').cpus().length - 1,
},
},
{
loader: 'ts-loader',
options: {
happyPackMode: true // IMPORTANT! use happyPackMode mode to speed-up compilation and reduce errors reported to webpack
}
}
]
}
},
plugins: [
new ForkTsCheckerWebpackPlugin({ checkSyntacticErrors: true })
]
};

As you can see the configuration is much cleaner than with HappyPack. Interestingly ts-loader still needs to run in "happyPackMode" and that's because thread-loader is essentially behaving in the same fashion as with HappyPack and so ts-loader needs to behave in the same way. Probably ts-loader should have a more generic flag name than "happyPackMode". (Famously, naming things is hard; so if you've a good idea, tell me!)

These loaders are new and so tread carefully. My own experiences have been pretty positive but your mileage may vary. Do note that, as with HappyPack, the thread-loader is highly configurable.

If you'd like to see an example of how to use thread-loader and cache-loader then once again we have a simple example and a more involved one.

All This Could Be Yours...#

Wow! It looks like we can cut our build time by 4 minutes! #webpack@typescriptlang // cc @johnny_reillypic.twitter.com/gjvy9SLBAT

— Donald Pipowitch (@PipoPeperoni) June 23, 2017

In this post we're improving build speeds with TypeScript and webpack in 3 ways:

fork-ts-checker-webpack-plugin
With this plugin in play ts-loader only performs transpilation. ts-loader is doing less so the build is faster.
HappyPack
With HappyPack in the mix, the build is parallelised. That parallelisation means the build is faster.
thread-loader / cache-loader
With thread-loader and cache-loader, again the build is parallelised and the build is faster.