Saturday, 14 November 2020

Bulletproof uniq with TypeScript generics (yay code reviews!)

Never neglect the possibilities of a code review. There are times when you raise a PR and all you want is for everyone to hit approve so you can merge, merge and ship, ship! This can be a missed opportunity. For as much as I'd like to imagine my code is perfect, it's patently not. There's always scope for improvement.

"What's this?"

This week afforded me that opportunity. I was walking through a somewhat complicated PR on a call and someone said "what's this?". They'd spotted an expression much like this in my code:

const myValues = [...new Set(allTheValuesSupplied)];

What is that? Well, it's a number of things:

  1. It's a way to get the unique values in a collection.
  2. It's a pro-tip and a coding BMX trick.

What do I mean? Well, this is indeed a technique for getting the unique values in a collection. But it relies upon you knowing a bunch of things:

  • Set contains unique values. If you add multiple identical values, only a single value will be stored.
  • The Set constructor takes iterable objects. This means we can new up a Set with an array that we want to "unique-ify" and we will have a Set that contains those unique values.
  • If you want to go on to do filtering / mapping etc on your unique values, you'll need to get them out of the Set. This is because (regrettably) ECMAScript iterables don't implicitly support these operations and neither are methods such as these part of the Set API. The easiest way to do that is to spread into a new array which you can then operate upon.

I have this knowledge. Lots of people have this knowledge. But whilst this may be the case, using this technique goes against what I would generally consider to be a good tenet of programming: comprehensibility. When you read this code above, it doesn't immediately tell you what it's doing. This is a strike against it.

Further to that, it's "noisy". Even if the reader does have this knowledge, as they digest the code, they have to mentally unravel it. "Oh it's a Set, we're passing in values, then spreading it out, it's probably intended to get the unique values.... Right, cool, cool.... Continue!"

Margarida Pereira explicitly called this out and I found myself agreeing. Let's make a uniq function!

uniq v1

I wrote a very simple uniq function which looked like this:

/**
 * Return the unique values found in the passed iterable
 */
function uniq<TElement>(iterableToGetUniqueValuesOf: Iterable<TElement>) {
    return [...new Set(iterableToGetUniqueValuesOf)];
}

Usage of this was simple:

uniq([1,1,1,3,1,1,2]) // produces [1, 3, 2]
uniq(["John", "Guida", "Ollie", "Divya", "John"]) // produces ["John", "Guida", "Ollie", "Divya"]

And I thought this was tremendous. I committed and pushed. I assumed there was no more to be done. Guida (Margarida) then made this very helpful comment:

BTW, I found a big bold warning that new Set() compares objects by reference (unless they're primitives) so it might be worth adding a comment to warn people that uniq/distinct compares objects by reference:
https://codeburst.io/javascript-array-distinct-5edc93501dc4

She was right! If a caller was to, say, pass a collection of objects to uniq then they'd end up highly disappointed. Consider:

uniq([{ name: "John" }, { name: "John" }]) // produces [{ name: "John" }, { name: "John" }]

We can do better!

uniq v2

I like compilers shouting at me. Or more accurately, I like compilers telling me when something isn't valid / supported / correct. I wanted uniq to mirror the behaviour of Set - to only support primitives such as string, number etc. So I made a new version of uniq that hardened up the generic contraints:

/**
 * Return the unique values found in the passed iterable
 */
function uniq<TElement extends string | number | bigint | boolean | symbol>(
    iterableToGetUniqueValuesOf: Iterable<TElement>
) {
    return [...new Set(iterableToGetUniqueValuesOf)];
}

With this in place, the compiler started shouting in the most helpful way. When I re-attemped [{ name: "John" }, { name: "John" }] the compiler hit me with:

Argument of type '{ name: string; }[]' is not assignable to parameter of type 'Iterable<string | number | bigint | boolean | symbol>'.

Take a look.

This is good. This is descriptive code that only allows legitimate inputs. It should lead to less confusion and a reduced likelihood of issues in Production. It's also a nice example of how code review can result in demonstrably better code. Thanks Guida!

Tuesday, 10 November 2020

Throttling data requests with React Hooks

When an application loads data, typically relatively few HTTP requests will be made. For example, if we imagine we're making a student administration application, then a "view" screen might make a single HTTP request to load that student's data before displaying it.

Occasionally there's a need for an application to make a large number of HTTP requests. Consider a reporting application which loads data and then aggregates it for presentation purposes.

This need presents two interesting problems to solve:

  1. how do we load data gradually?
  2. how do we present loading progress to users?

This post will talk about how we can tackle these and demonstrate using a custom React Hook.

Let's bring Chrome to its knees

We'll begin our journey by spinning up a TypeScript React app with Create React App:

npx create-react-app throttle-requests-react-hook --template typescript

Because we're going to be making a number of asynchronous calls, we're going to simplify the code by leaning on the widely used react-use for a useAsync hook.

cd throttle-requests-react-hook
yarn add react-use

We'll replace the App.css file with this:

.App {
  text-align: center;
}

.App-header {
  background-color: #282c34;
  min-height: 100vh;
  display: flex;
  flex-direction: column;
  align-items: center;
  justify-content: center;
  font-size: calc(10px + 2vmin);
  color: white;
}

.App-labelinput > * {
  margin: 0.5em;
  font-size:24px;
}

.App-link {
  color: #61dafb;
}

.App-button {
  font-size: calc(10px + 2vmin);
  margin-top: 0.5em;
  padding: 1em;
  background-color: cornflowerblue;
  color: #ffffff;
  text-align: center;
}

.App-progress {
  padding: 1em;
  background-color: cadetblue;
  color: #ffffff;
}

.App-results {
  display: flex;
  flex-wrap: wrap;
}

.App-results > * {
  padding: 1em;
  margin: 0.5em;
  background-color: darkblue;
  flex: 1 1 300px;
}

Then we'll replace the App.tsx contents with this:

import React, { useState } from "react";
import { useAsync } from "react-use";
import "./App.css";

function use10_000Requests(startedAt: string) {
  const responses = useAsync(async () => {
    if (!startedAt) return;

    // make 10,000 unique HTTP requests
    const results = await Promise.all(
      Array.from(Array(10_000)).map(async (_, index) => {
        const response = await fetch(
          `/manifest.json?querystringValueToPreventCaching=${startedAt}_request-${index}`
        );
        const json = await response.json();
        return json;
      })
    );

    return results;
  }, [startedAt]);

  return responses;
}


function App() {
  const [startedAt, setStartedAt] = useState("");
  const responses = use10_000Requests(startedAt);

  return (
    <div className="App">
      <header className="App-header">
        <h1>The HTTP request machine</h1>
        <button
          className="App-button"
          onClick={(_) => setStartedAt(new Date().toISOString())}
        >
          Make 10,000 requests
        </button>
        {responses.loading && <div>{progressMessage}</div>}
        {responses.error && <div>Something went wrong</div>}
        {responses.value && (
          <div className="App-results">
            {responses.value.length} requests completed successfully
          </div>
        )}
      </header>
    </div>
  );
}

export default App;

The app that we've built is very simple; it's a button which, when you press it, fires 10,000 HTTP requests in parallel using the Fetch API. The data being requested in this case is an arbitrary JSON file; the manifest.json. If you look closely you'll see we're doing some querystring tricks with our URL to avoid getting cached data.

In fact, for this demo we're not interested in the results of these HTTP requests; rather we're interested in how the browser copes with this approach. (Spoiler: not well!) It's worth considering that requesting a text file from a server running on the same machine as the browser should be fast.

So we'll run yarn start and go to http://localhost:3000 to get to the app. Running with Devtools open results in the following unhappy affair:

The GIF above has been edited significantly for length. In reality it took 20 seconds for the first request to be fired, prior to that Chrome was unresponsive. When requests did start to fire, a significant number failed with net::ERR_INSUFFICIENT_RESOURCES. Further to that, those requests that were fired sat in "Stalled" state prior to being executed. This is a consequence of Chrome limiting the number of connections - all browsers do this:

There are already six TCP connections open for this origin, which is the limit. Applies to HTTP/1.0 and HTTP/1.1 only.

In summary, the problems with the current approach are:

  1. the browser becoming unresponsive
  2. failing HTTP requests due to insufficient resources
  3. no information displayed to the user around progress

Throttle me this

Instead of hammering the browser by firing all the requests at once, we could instead implement a throttle. A throttle is a mechanism which allows you to limit the rate at which operations are performed. In this case we want to limit the rate at which HTTP requests are made. A throttle will tackle problems 1 and 2 - essentially keeping the browser free and easy and ensuring that requests are all successfully sent. We also want to keep our users informed around how progress is going. It's time to unveil the useThrottleRequests hook:

import { useMemo, useReducer } from "react";
import { AsyncState } from "react-use/lib/useAsync";

/** Function which makes a request */
export type RequestToMake = () => Promise<void>;

/**
 * Given an array of requestsToMake and a limit on the number of max parallel requests
 * queue up those requests and start firing them
 * - inspired by Rafael Xavier's approach here: https://stackoverflow.com/a/48007240/761388
 *
 * @param requestsToMake
 * @param maxParallelRequests the maximum number of requests to make - defaults to 6
 */
async function throttleRequests(
  requestsToMake: RequestToMake[],
  maxParallelRequests = 6
) {
  // queue up simultaneous calls
  const queue: Promise<void>[] = [];
  for (let requestToMake of requestsToMake) {
    // fire the async function, add its promise to the queue,
    // and remove it from queue when complete
    const promise = requestToMake().then((res) => {
      queue.splice(queue.indexOf(promise), 1);
      return res;
    });
    queue.push(promise);

    // if the number of queued requests matches our limit then
    // wait for one to finish before enqueueing more
    if (queue.length >= maxParallelRequests) {
      await Promise.race(queue);
    }
  }
  // wait for the rest of the calls to finish
  await Promise.all(queue);
}

/**
 * The state that represents the progress in processing throttled requests
 */
export type ThrottledProgress<TData> = {
  /** the number of requests that will be made */
  totalRequests: number;
  /** the errors that came from failed requests */
  errors: Error[];
  /** the responses that came from successful requests */
  values: TData[];
  /** a value between 0 and 100 which represents the percentage of requests that have been completed (whether successfully or not) */
  percentageLoaded: number;
  /** whether the throttle is currently processing requests */
  loading: boolean;
};

function createThrottledProgress<TData>(
  totalRequests: number
): ThrottledProgress<TData> {
  return {
    totalRequests,
    percentageLoaded: 0,
    loading: false,
    errors: [],
    values: [],
  };
}

/**
 * A reducing function which takes the supplied `ThrottledProgress` and applies a new value to it
 */
function updateThrottledProgress<TData>(
  currentProgress: ThrottledProgress<TData>,
  newData: AsyncState<TData>
): ThrottledProgress<TData> {
  const errors = newData.error
    ? [...currentProgress.errors, newData.error]
    : currentProgress.errors;

  const values = newData.value
    ? [...currentProgress.values, newData.value]
    : currentProgress.values;

  const percentageLoaded =
    currentProgress.totalRequests === 0
      ? 0
      : Math.round(
          ((errors.length + values.length) / currentProgress.totalRequests) * 100
        );

  const loading =
    currentProgress.totalRequests === 0
      ? false
      : errors.length + values.length < currentProgress.totalRequests;

  return {
    totalRequests: currentProgress.totalRequests,
    loading,
    percentageLoaded,
    errors,
    values,
  };
}

type ThrottleActions<TValue> =
  | {
      type: "initialise";
      totalRequests: number;
    }
  | {
      type: "requestSuccess";
      value: TValue;
    }
  | {
      type: "requestFailed";
      error: Error;
    };

/**
 * Create a ThrottleRequests and an updater
 */
export function useThrottleRequests<TValue>() {
  function reducer(
    throttledProgressAndState: ThrottledProgress<TValue>,
    action: ThrottleActions<TValue>
  ): ThrottledProgress<TValue> {
    switch (action.type) {
      case "initialise":
        return createThrottledProgress(action.totalRequests);

      case "requestSuccess":
        return updateThrottledProgress(throttledProgressAndState, {
          loading: false,
          value: action.value,
        });

      case "requestFailed":
        return updateThrottledProgress(throttledProgressAndState, {
          loading: false,
          error: action.error,
        });
    }
  }

  const [throttle, dispatch] = useReducer(
    reducer,
    createThrottledProgress<TValue>(/** totalRequests */ 0)
  );

  const updateThrottle = useMemo(() => {
    /**
     * Update the throttle with a successful request
     * @param values from request
     */
    function requestSucceededWithData(value: TValue) {
      return dispatch({
        type: "requestSuccess",
        value,
      });
    }

    /**
     * Update the throttle upon a failed request with an error message
     * @param error error
     */
    function requestFailedWithError(error: Error) {
      return dispatch({
        type: "requestFailed",
        error,
      });
    }

    /**
     * Given an array of requestsToMake and a limit on the number of max parallel requests
     * queue up those requests and start firing them
     * - based upon https://stackoverflow.com/a/48007240/761388
     *
     * @param requestsToMake
     * @param maxParallelRequests the maximum number of requests to make - defaults to 6
     */
    function queueRequests(
      requestsToMake: RequestToMake[],
      maxParallelRequests = 6
    ) {
      dispatch({
        type: "initialise",
        totalRequests: requestsToMake.length,
      });

      return throttleRequests(requestsToMake, maxParallelRequests);
    }

    return {
      queueRequests,
      requestSucceededWithData,
      requestFailedWithError,
    };
  }, [dispatch]);

  return {
    throttle,
    updateThrottle,
  };
}

The useThrottleRequests hook returns 2 properties:

  • throttle - a ThrottledProgress<TData> that contains the following data:

    • totalRequests - the number of requests that will be made
    • errors - the errors that came from failed requests
    • values - the responses that came from successful requests
    • percentageLoaded - a value between 0 and 100 which represents the percentage of requests that have been completed (whether successfully or not)
    • loading - whether the throttle is currently processing requests
  • updateThrottle - an object which exposes 3 functions:

    • queueRequests - the function to which you pass the requests that should be queued and executed in a throttled fashion
    • requestSucceededWithData - the function which is called if a request succeeds to provide the data
    • requestFailedWithError - the function which is called if a request fails to provide the error

That's a lot of words to describe our useThrottleRequests hook. Let's look at what it looks like by migrating our use10_000Requests hook to (no pun intended) use it. Here's a new implementation of App.tsx:

import React, { useState } from "react";
import { useAsync } from "react-use";
import { useThrottleRequests } from "./useThrottleRequests";
import "./App.css";

function use10_000Requests(startedAt: string) {
  const { throttle, updateThrottle } = useThrottleRequests();
  const [progressMessage, setProgressMessage] = useState("not started");

  useAsync(async() => {
      if (!startedAt) return;

      setProgressMessage("preparing");

      const requestsToMake = Array.from(Array(10_000)).map(
        (_, index) => async () => {
          try {
            setProgressMessage(`loading ${index}...`);

            const response = await fetch(
              `/manifest.json?querystringValueToPreventCaching=${startedAt}_request-${index}`
            );
            const json = await response.json();

            updateThrottle.requestSucceededWithData(json);
          } catch (error) {
            console.error(`failed to load ${index}`, error);
            updateThrottle.requestFailedWithError(error);
          }
        }
      );

      await updateThrottle.queueRequests(requestsToMake);

  }, [startedAt, updateThrottle, setProgressMessage]);

  return { throttle, progressMessage };
}

function App() {
  const [startedAt, setStartedAt] = useState("");

  const { progressMessage, throttle } = use10_000Requests(startedAt);

  return (
    <div className="App">
      <header className="App-header">
        <h1>The HTTP request machine</h1>
        <button
          className="App-button"
          onClick={(_) => setStartedAt(new Date().toISOString())}
        >
          Make 10,000 requests
        </button>
        {throttle.loading && <div>{progressMessage}</div>}
        {throttle.values.length > 0 && (
          <div className="App-results">
            {throttle.values.length} requests completed successfully
          </div>
        )}
        {throttle.errors.length > 0 && (
          <div className="App-results">
            {throttle.errors.length} requests errored
          </div>
        )}
      </header>
    </div>
  );
}

export default App;

Looking at the new use10_000Requests hook, there's a few subtle differences to our prior implementation. First of all, we're now exposing the throttle; a ThrottleProgress<TData>. Our updated hook also exposes a progressMessage which is a simple string stored with useState that we update as our throttle runs. In truth the information being surfaced here isn't that interesting. The progressMessage is in place just to illustrate that you could capture some data from your requests as they complete for display purposes; a running total for instance.

So, how does our new hook approach perform?

Very well indeed! Please note that the above GIF has again been edited for brevity. If we look back at the problems we faced with the prior approach, how do we compare?

  1. the browser becoming unresponsive - the browser remains responsive.
  2. failing HTTP requests due to insufficient resources - the browser does not experience failing HTTP requests.
  3. no information displayable to the user around progress - details of progress are displayed to the user throughout.

Tremendous!

What shall we build?

Our current example is definitely contrived. Let's try and apply our useThrottleRequests hook to a more realistic scenario. We're going to build an application which, given a repo on GitHub, lists all the contributors blogs. (You can specify a blog URL on your GitHub profile; many people use this to specify their Twitter profile.)

We can build this thanks to the excellent GitHub REST API. It exposes two endpoints of interest given our goal.

1. List repository contributors

List repository contributors lists contributors to the specified repository at this URL: GET https://api.github.com/repos/{owner}/{repo}/contributors. The response is an array of objects, crucially featuring a url property that points to the user in question's API endpoint:

[
  // ...
  {
    // ...
    "url": "https://api.github.com/users/octocat",
    // ...
  },
  // ...
]
2. Get a user

Get a user is the API that the url property above is referring to. When called it returns an object representing the publicly available information about a user:

{
  // ...
  "name": "The Octocat",
  // ...
  "blog": "https://github.blog",
  // ...
}

Blogging devs v1.0

We're now ready to build our blogging devs app; let's replace the existing App.tsx with:

import React, { useCallback, useMemo, useState } from "react";
import { useAsync } from "react-use";
import { useThrottleRequests } from "./useThrottleRequests";
import "./App.css";

type GitHubUser = { name: string; blog?: string };

function timeout(ms: number) {
  return new Promise((resolve) => setTimeout(resolve, ms));
}

function useContributors(contributorsUrlToLoad: string) {
  const { throttle, updateThrottle } = useThrottleRequests<GitHubUser>();
  const [progressMessage, setProgressMessage] = useState("");

  useAsync(async () => {
    if (!contributorsUrlToLoad) return;

    setProgressMessage("loading contributors");

    // load contributors from GitHub
    const contributorsResponse = await fetch(contributorsUrlToLoad);
    const contributors: { url: string }[] = await contributorsResponse.json();

    setProgressMessage(`loading ${contributors.length} contributors...`);

    // For each entry in result, retrieve the given user from GitHub
    const requestsToMake = contributors.map(({ url }, index) => async () => {
      try {
        setProgressMessage(
          `loading ${index} / ${contributors.length}: ${url}...`
        );

        const response = await fetch(url);
        const json: GitHubUser = await response.json();

        // wait for 1 second before completing the request
        // - makes for better demos
        await timeout(1000);

        updateThrottle.requestSucceededWithData(json);
      } catch (error) {
        console.error(`failed to load ${url}`, error);
        updateThrottle.requestFailedWithError(error);
      }
    });

    await updateThrottle.queueRequests(requestsToMake);

    setProgressMessage("");
  }, [contributorsUrlToLoad, updateThrottle, setProgressMessage]);

  return { throttle, progressMessage };
}

function App() {
  // The owner and repo to query; we're going to default
  // to using DefinitelyTyped as an example repo as it
  // is one of the most contributed to repos on GitHub
  const [owner, setOwner] = useState("DefinitelyTyped");
  const [repo, setRepo] = useState("DefinitelyTyped");
  const handleOwnerChange = useCallback(
    (event: React.ChangeEvent<HTMLInputElement>) =>
      setOwner(event.target.value),
    [setOwner]
  );
  const handleRepoChange = useCallback(
    (event: React.ChangeEvent<HTMLInputElement>) => setRepo(event.target.value),
    [setRepo]
  );

  const contributorsUrl = `https://api.github.com/repos/${owner}/${repo}/contributors`;

  const [contributorsUrlToLoad, setUrlToLoad] = useState("");
  const { progressMessage, throttle } = useContributors(contributorsUrlToLoad);

  const bloggers = useMemo(
    () => throttle.values.filter((contributor) => contributor.blog),
    [throttle]
  );

  return (
    <div className="App">
      <header className="App-header">
        <h1>Blogging devs</h1>

        <p>
          Show me the{" "}
          <a
            className="App-link"
            href={contributorsUrl}
            target="_blank"
            rel="noopener noreferrer"
          >
            contributors for {owner}/{repo}
          </a>{" "}
          who have blogs.
        </p>

        <div className="App-labelinput">
          <label htmlFor="owner">GitHub Owner</label>
          <input
            id="owner"
            type="text"
            value={owner}
            onChange={handleOwnerChange}
          />
          <label htmlFor="repo">GitHub Repo</label>
          <input
            id="repo"
            type="text"
            value={repo}
            onChange={handleRepoChange}
          />
        </div>

        <button
          className="App-button"
          onClick={(e) => setUrlToLoad(contributorsUrl)}
        >
          Load bloggers from GitHub
        </button>

        {progressMessage && (
          <div className="App-progress">{progressMessage}</div>
        )}

        {throttle.percentageLoaded > 0 && (
          <>
            <h3>Behold {bloggers.length} bloggers:</h3>
            <div className="App-results">
              {bloggers.map((blogger) => (
                <div key={blogger.name}>
                  <div>{blogger.name}</div>
                  <a
                    className="App-link"
                    href={blogger.blog}
                    target="_blank"
                    rel="noopener noreferrer"
                  >
                    {blogger.blog}
                  </a>
                </div>
              ))}
            </div>
          </>
        )}

        {throttle.errors.length > 0 && (
          <div className="App-results">
            {throttle.errors.length} requests errored
          </div>
        )}
      </header>
    </div>
  );
}

export default App;

The application gives users the opportunity to enter the organisation and repository of a GitHub project. Then, when the button is clicked, it:

  • loads the contributors
  • for each contributor it loads the individual user (separate HTTP request for each)
  • as it loads it communicates how far through the loading progress it has got
  • as users are loaded, it renders a tile for each user with a listed blog

Just to make the demo a little clearer we've artificially slowed the duration of each request by a second. What does it look like when you put it together? Well like this:

We have built a React Hook which allows us to:

  • gradually load data
  • without blocking the UI of the browser
  • and which provides progress data to keep users informed.

This post was originally published on LogRocket.

Saturday, 31 October 2020

Azure DevOps Node API: The missing episodes

I've been taking a good look at the REST API for Azure DevOps. I'm delighted to say that it's a very full API. However, there's quirks.

I'm writing a tool that interrogates Azure DevOps in order that it can construct release documentation. That release documentation we would like to publish to the project wiki.

To make integration with Azure DevOps even easier, the ADO team have put a good amount of work into client libraries that allow you to code in your language of choice. In my case I'm writing a Node.js tool (using TypeScript) and happily the client lib for Node is written and published with TypeScript too. Tremendous! However, there is a "but" coming....

Wiki got a big ol' "but"

As I've been using the Node client lib, I've found minor quirks. Such as the GitApi.getRefs missing the pagination parts of the API.

Whilst the GitApi was missing some parameters on a method, the WikiApi was missing whole endpoints, such as the Pages - Create Or Update one. The various client libraries are auto-generated which makes contribution a difficult game. The lovely Matt Cooper has alerted the team

These clients are generated from the server-side controllers, and at a glance, I don't understand why those two parameters weren't included. Full transparency, we don't dedicate a lot of cycles here, but I will get it on the team's radar to investigate/improve.

In the meantime, I still had a tool to write.

Handrolled Wiki API

Whilst the Node.js client lib was missing some crucial pieces, there did seem to be a way forward. Using the API directly; not using the client lib to do our HTTP and using axios instead. Happily the types we needed were still available for be leveraged.

Looking at the docs it seemed it ought to be simple:

https://docs.microsoft.com/en-us/rest/api/azure/devops/?view=azure-devops-rest-6.1#assemble-the-request

But when I attempted this I found my requests erroring out with 203 Non-Authoritative Informations. It didn't make sense. I couldn't get a single request to be successful, they all failed. It occurred to me that the answer was hiding in node_modules. I'd managed to make successful requests to the API using the client lib. What was it doing that I wasn't?

The answer ended up being an authorization one-liner:

    const request = await axios({
        url,
        headers: {
            Accept: 'application/json',
            'Content-Type': 'application/json',
            // This!
            Authorization: `Basic ${Buffer.from(`PAT:${adoPersonalAccessToken}`).toString('base64')}`,
            'X-TFS-FedAuthRedirect': 'Suppress',
        },
    });
}

With this in hand everything started to work and I found myself able to write my own clients to fill in the missing pieces from the client lib:

import axios from 'axios';
import { WikiPage, WikiPageCreateOrUpdateParameters, WikiType } from 'azure-devops-node-api/interfaces/WikiInterfaces';
import { IWikiApi } from 'azure-devops-node-api/WikiApi';

async function getWikiPage({
    adoUrl,
    adoProject,
    adoPat,
    wikiId,
    path,
}: {
    adoUrl: string;
    adoProject: string;
    adoPat: string;
    wikiId: string;
    path: string;
}) {
    try {
        const url = `${makeBaseApiUrl({
            adoUrl,
            adoProject,
        })}/wiki/wikis/${wikiId}/pages?${apiVersion}&path=${path}&includeContent=True&recursionLevel=full`;
        const request = await axios({
            url,
            headers: makeHeaders(adoPat),
        });

        const page: WikiPage = request.data;
        return page;
    } catch (error) {
        return undefined;
    }
}

async function createWikiPage({
    adoUrl,
    adoProject,
    adoPat,
    wikiId,
    path,
    data,
}: {
    adoUrl: string;
    adoProject: string;
    adoPat: string;
    wikiId: string;
    path: string;
    data: WikiPageCreateOrUpdateParameters;
}) {
    try {
        const url = `${makeBaseApiUrl({
            adoUrl,
            adoProject,
        })}/wiki/wikis/${wikiId}/pages?${apiVersion}&path=${path}`;

        const request = await axios({
            method: 'PUT',
            url,
            headers: makeHeaders(adoPat),
            data,
        });

        const newPage: WikiPage = request.data;
        return newPage;
    } catch (error) {
        return undefined;
    }
}

const apiVersion = "api-version=6.0";

/**
* Create the headers necessary to ake Azure DevOps happy
* @param adoPat Personal Access Token from ADO
*/
function makeHeaders(adoPat: string) {
    return {
        Accept: 'application/json',
        'Content-Type': 'application/json',
        Authorization: `Basic ${Buffer.from(`PAT:${adoPat}`).toString('base64')}`,
        'X-TFS-FedAuthRedirect': 'Suppress',
    };
}

/**
* eg https://dev.azure.com/{organization}/{project}/_apis
*/
function makeBaseApiUrl({ adoUrl, adoProject }: { adoUrl: string; adoProject: string }) {
    return `${adoUrl}/${adoProject}/_apis`;
}

With this I was able to write code like this:

    let topLevelPage = await getWikiPage({
        adoUrl,
        adoProject,
        adoPat,
        wikiId,
        path: config.wikiTopLevelName,
    });

    if (!topLevelPage)
        topLevelPage = await createWikiPage({
            adoUrl,
            adoProject,
            adoPat,
            wikiId,
            path: config.wikiTopLevelName,
            data: { content: '' },
        });

and the wikis were ours!

Monday, 19 October 2020

The Mysterious Case of Safari and the Empty Download

TL;DR: Safari wants a Content-Type header in responses. Even if the response is Content-Length: 0. Without this, Safari can attempt to trigger an empty download. Don't argue; just go with it; some browsers are strange.

The longer version

Every now and then a mystery presents itself. A puzzle which just doesn't make sense and yet stubbornly continues to exist. I happened upon one of these the other day and to say it was frustrating does it no justice at all.

It all came back to the default iOS and Mac browser; Safari. When our users log into our application, they are redirected to a shared login provider which, upon successful authentication, hands over a cookie containing auth details and redirects back to our application. A middleware in our app reads what it needs from the cookie and then creates a cookie of its own which is to be used throughout the session. As soon as the cookie is set, the page refreshes and the app boots up in an authenticated state.

That's the background. This mechanism had long been working fine with Chrome (which the majority of our users browse with), Edge, Firefox and Internet Explorer. But we started to get reports from Safari users that, once they'd supplied their credentials, they'd not be authenticated and redirected back to our application. Instead they'd be prompted to download an empty document and the redirect would not take place.

As a team we could not fathom why this should be the case; it just didn't make sense. There followed hours of experimentation before Hennie noticed something. It was at the point when the redirect back to our app from the login provider took place. Specifically the initial response that came back which contained our custom cookie and a Refresh: 0 header to trigger a refresh in the browser. There was no content in the response, save for headers. It was Content-Length: 0 all the way.

Hennie noticed that there was no Content-Type set and wondered if that was significant. It didn't seem like it would be a necessary header given there was no content. But Safari reckons not with logic. As an experiment we tried setting the response header to Content-Type: text/html. It worked! No mystery download, no failed redirect (which it turned out was actually a successful redirect which wasn't being surfaced in Safari's network request tab).

It appears that always providing a Content-Type header in your responses is wise if only for the case of Safari. In fact, it's generally unlikely that this won't be set anyway, but it can happen as we have experienced. Hopefully we've suffered so you don't have to.

Friday, 2 October 2020

Autofac 6, integration tests and .NET generic hosting

I blogged a little while ago around to support integration tests using Autofac. This was specific to Autofac but documented a workaround for a long standing issue with ConfigureTestContainer that was introduced into .NET core 3.0 which affects all third-party containers that use ConfigureTestContainer in their tests.

I'll not repeat the contents of the previous post - it all still stands. However, with Autofac 6 the approach documented there will cease to work. This is because the previous approach relied upon ContainerBuilder not being sealed. As of Autofac 6 it is.

Happily the tremendous Alistair Evans came up with an alternative approach which is listed below:

    /// <summary>
    /// Based upon https://github.com/dotnet/AspNetCore.Docs/tree/master/aspnetcore/test/integration-tests/samples/3.x/IntegrationTestsSample
    /// </summary>
    /// <typeparam name="TStartup"></typeparam>
    public class AutofacWebApplicationFactory<TStartup> : WebApplicationFactory<TStartup> where TStartup : class
    {
        protected override IHost CreateHost(IHostBuilder builder)
        {
            builder.UseServiceProviderFactory<ContainerBuilder>(new CustomServiceProviderFactory());
            return base.CreateHost(builder);
        }
    }

    /// <summary>
    /// Based upon https://github.com/dotnet/aspnetcore/issues/14907#issuecomment-620750841 - only necessary because of an issue in ASP.NET Core
    /// </summary>
    public class CustomServiceProviderFactory : IServiceProviderFactory<ContainerBuilder>
    {
        private AutofacServiceProviderFactory _wrapped;
        private IServiceCollection _services;

        public CustomServiceProviderFactory()
        {
            _wrapped = new AutofacServiceProviderFactory();
        }

        public ContainerBuilder CreateBuilder(IServiceCollection services)
        {
            // Store the services for later.
            _services = services;

            return _wrapped.CreateBuilder(services);
        }

        public IServiceProvider CreateServiceProvider(ContainerBuilder containerBuilder)
        {
            var sp = _services.BuildServiceProvider();
#pragma warning disable CS0612 // Type or member is obsolete
            var filters = sp.GetRequiredService<IEnumerable<IStartupConfigureContainerFilter<ContainerBuilder>>>();
#pragma warning restore CS0612 // Type or member is obsolete

            foreach (var filter in filters)
            {
                filter.ConfigureContainer(b => { })(containerBuilder);
            }

            return _wrapped.CreateServiceProvider(containerBuilder);
        }        
    }

Using this in place of the previous approach should allow you continue running your integration tests with Autofac 6. Thanks Alistair!

Concern for third-party containers

Whilst this gets us back up and running, Alistair pointed out that this approach depends upon a deprecated interface. This is the IStartupConfigureContainerFilter which has been marked as Obsolete since mid 2019. What this means is, at some point, this approach will stop working.

The marvellous David Fowler has said that ConfigureTestContainer issue should be resolved in .NET. However it's worth noting that this has been an issue since .NET Core 3 shipped and unfortunately the wonderful Chris Ross has advised that it's not likely to be fixed for .NET 5.

I'm very keen this does get resolved in .NET. Building tests upon an Obsolete attribute doesn't fill me with confidence. I'm a long time user of Autofac and I'd like to continue to be. Here's hoping that's made possible by a fix landing in .NET. If this is something you care about, it may be worth upvoting / commenting on the issue in GitHub so the team are aware of desire around this being resolved.

Friday, 4 September 2020

Why your team needs a newsfeed

I'm part of a team that builds an online platform. I'm often preoccupied by how to narrow the gap between our users and "us" - the people that build the platform. It's important we understand how people use and interact with what we've built. If we don't then we're liable to waste our time and energy building the wrong things. Or the wrong amount of the right things.

On a recent holiday I spent a certain amount of time pondering how to narrow the gap between our user and us. We have lots of things that help us; we use various analytics tools like mixpanel, we've got a mini analytics platform of our own, we have teams notifications that pop up client feedback and so on. They are all great, but they're somewhat disparate; they don't give us a clear insight as to who uses our platform and how they do so. The information is there, but it's tough to grok. It doesn't make for a joined up story.

Reaching around for how to solve this I had an idea: what if our platform had a newsfeed? The kind of thing that social media platforms the likes of Twitter and Facebook have used to great effect; a stream of mini-activities which show how the community interacts with the product. People logging in and browsing around, using features on the platform. If we could see this in near real time we'd be brought closer to our users; we'd have something that would help us have real empathy and understanding. We'd see our product as the stories of users interacting with it.

How do you build a newsfeed?

This was an experiment that seemed worth pursuing. So I decided to build a proof of concept and see what happened. Now I intended to put the "M" into MVP with this; I went in with a number of intentional constraints:

  1. The news feed wouldn't auto update (users have the F5 key for that)
  2. We'd host the newsfeed in our own mini analytics platform (which is already used by the team to understand how people use the platform)
  3. News stories wouldn't be stored anywhere; we'd generate them on the fly by querying various databases / APIs. The cost of this would be that our news stories wouldn't be "persistent"; you wouldn't be able to address them with a URL; there'd be no way to build "like" or "share" functionality.

All of the above constraints are, importantly, reversable decisions. If we want auto update it could be built later. If we want the newsfeed to live somewhere else we could move it. If we wanted news stories to be persisted then we could do that.

Implementation

With these constraints in mind, I turned my attention to the implementation. I built a NewsFeedService that would be queried for news stories. The interface I decided to build looked like this:

NewsFeedService.getNewsFeed(from: Date, to: Date): NewsFeed

type NewsFeed {
    startedAt: Date;
    ended at: Date;
    stories: NewsStory[];
}

type NewsStory {
    /** When the story happened */
    happenedAt: Date;
    /** A code that represents the type of story this is; eg USER_SESSION */
    storyCode: string
    /** The story details in markdown format */
    story: string;
}

Each query to NewsFeedService.getNewsFeed would query various databases / APIs related to our product, looking for interesting events. Whether it be users logging in, users performing some kind of action, whatever. For each interested event a news story like this would be produced:

Jane Smith logged in at 10:03am for 25 minutes. They placed an order worth £3,000.

Now the killer feature here is Markdown. Our stories are written in Markdown. Why is Markdown cool? Well to quote the creators of Markdown:

Markdown allows you to write using an easy-to-read, easy-to-write plain text format, then convert it to structurally valid XHTML (or HTML).

This crucially includes the ability to include links. This was significant because I want us to be able to be able to click on pieces of information in the stories and be taken to the relevant place in the platform to see more details. Just as you see status updates on, for example, Twitter which lead you on to more details:

Again consider this example news story:

Jane Smith logged in at 10:03am for 25 minutes. They placed an order worth £3,000.

Consider that story but without a link. It's not the same is it? A newsfeed without links would be missing a trick. Markdown gives us links. And happily due to my extensive work down the open source mines, I speak it like a native.

The first consumer of the newsfeed was to be our own mini analytics platform, which is a React app. Converting the markdown stories to React is a solved problem thanks to the wonderful react-markdown. You can simply sling Markdown at it and out comes HTML. Et voilà a news feed!

What's next?

So that's it! We've built a (primitive) news feed. We can now see in real time how are users are getting on. We're closer to them, we understand them better as a consequence. If we want to take it further there's a number of things we could do:

  1. We could make the feed auto-update
  2. We could push news stories to other destinations. Markdown is a gloriously portable format which can be used in a variety of environments. For instance the likes of Slack and Teams accept it and apps like these are generally open on people's desktops and phones all the time anyway. Another way to narrow the gap between us and and our users.

It's very exciting!

Sunday, 9 August 2020

Devcontainers AKA performance in a secure sandbox

Many corporate machines arrive in engineers hands with a preponderance of pre-installed background tools; from virus checkers to backup utilities to port blockers; the list is long.

The reason that these tools are installed is generally noble. However, the implementation can often be problematic. The tools may be set up in such a way as they impact and interfere with one another. Really powerful machines with 8 CPUs and hardy SSDs can be slowed to a crawl. Put simply: the good people responsible for ensuring security are rarely encouraged to incentivise performance alongside it. And so don't.

The unfortunate consequence of considering the role of security without regard to performance is this: sluggish computers. The further consequence (and this is the one I want you to think long and hard about) is low developer productivity. And that sucks. It impacts what an organisation is able to do, how fast an organisation is able to move. Put simply: it can be the difference between success and failure.

The most secure computer is off. But you won't ship much with it. Encouraging your organisation to consider tackling security with performance in mind is worthwhile. It's a long game though. In the meantime what can we do?

"Hide from the virus checkers* in a devcontainer"

Devcontainers, the infrastructure as code equivalent for developing software, have an underappreciated quality: unlocking your machine's performance.

Devcontainers are isolated secure sandboxes in which you can build software. To quote the docs:

A devcontainer.json file in your project tells VS Code how to access (or create) a development container with a well-defined tool and runtime stack. This container can be used to run an application or to sandbox tools, libraries, or runtimes needed for working with a codebase.

Workspace files are mounted from the local file system or copied or cloned into the container.

We're going to set up a devcontainer to code an ASP.NET Core application with a JavaScript (well TypeScript) front end. If there's one thing that's sure to catch a virus checkers beady eye, it's node_modules. node_modules contains more files than a black hole has mass. Consider a project with 5,000 source files. One trusty yarn later and the folder now has a tidy 250,000 files. The virus checker is now really sitting up and taking notice.

Our project has a git commit hook set up with Husky that formats our TypeScript files with Prettier. Every commit the files are formatted to align with the project standard. With all the virus checkers in place a git commit takes around 45 seconds. Inside a devcontainer we can drop this to 5 seconds. That's nine times faster. I'll repeat that: that's nine times faster!

The "cloned into a container" above is key to what we're going to do. We're not going to mount our local file system into the devcontainer. Oh no. We're going to build a devcontainer with ASP.NET CORE and JavaScript in. Then, inside there, we're going to clone our repo. Then we can develop, build and debug all inside the container. It will feel like we're working on our own machine because VS Code does such a damn fine job. In reality, we're connecting to another computer (a Linux computer to boot) that is running in isolation to our own. In our case that machine is sharing our hardware; but that's just an implementation detail. It could be anywhere (and in the future may well be).

Make me a devcontainer...

Enough talk... We're going to need a .devcontainer/devcontainer.json:

{
  "name": "my devcontainer",
  "dockerComposeFile": "../docker-compose.devcontainer.yml",
  "service": "my-devcontainer",
  "workspaceFolder": "/workspace",

  // Set *default* container specific settings.json values on container create.
  "settings": {
    "terminal.integrated.shell.linux": "/bin/zsh"
  },

  // Add the IDs of extensions you want installed when the container is created.
  "extensions": [
    "ms-dotnettools.csharp",
    "dbaeumer.vscode-eslint",
    "esbenp.prettier-vscode",
    "ms-mssql.mssql",
    "eamodio.gitlens",
    "ms-azuretools.vscode-docker",
    "k--kato.docomment",
    "Leopotam.csharpfixformat"
  ],

  // Use 'postCreateCommand' to clone the repo into the workspace folder when the devcontainer starts
  // and copy in the .env file
  "postCreateCommand": "git clone [email protected]:my-org/my-repo.git . && cp /.env /workspace/.env"

  // "remoteUser": "vscode"
}

Now the docker-compose.devcontainer.yml which lives in the root of the project. It provisions a SQL Server container (using the official image) and our devcontainer:

version: "3.7"
services:
  my-devcontainer:
    image: my-devcontainer
    build: 
      context: .
      dockerfile: Dockerfile.devcontainer
    command: /bin/zsh -c "while sleep 1000; do :; done"
    volumes:
      # mount .zshrc from home - make sure it doesn't contain Windows line endings
      - ~/.zshrc:/root/.zshrc

    # user: vscode
    ports:
      - "5000:5000"
      - "8080:8080"
    environment:
      - CONNECTIONSTRINGS__MYDATABASECONNECTION
    depends_on:
      - db
  db:
    image: mcr.microsoft.com/mssql/server:2019-latest
    privileged: true
    ports:
      - 1433:1433
    environment:
      SA_PASSWORD: "Your_password123"
      ACCEPT_EULA: "Y"         

The devcontainer will be built with the Dockerfile.devcontainer in the root of our repo. It relies upon your SSH keys and a .env file being available to be copied in:


#-----------------------------------------------------------------------------------------------------------
# Based upon: https://github.com/microsoft/vscode-dev-containers/tree/master/containers/dotnetcore
#-----------------------------------------------------------------------------------------------------------
ARG VARIANT="3.1-bionic"
FROM mcr.microsoft.com/dotnet/core/sdk:${VARIANT}

# Because MITM certificates
COPY ./docker/certs/. /usr/local/share/ca-certificates/
ENV NODE_EXTRA_CA_CERTS=/usr/local/share/ca-certificates/mitm.pem
RUN update-ca-certificates 

# This Dockerfile adds a non-root user with sudo access. Use the "remoteUser"
# property in devcontainer.json to use it. On Linux, the container user's GID/UIDs
# will be updated to match your local UID/GID (when using the dockerFile property).
# See https://aka.ms/vscode-remote/containers/non-root-user for details.
ARG USERNAME=vscode
ARG USER_UID=1000
ARG USER_GID=$USER_UID

# Options for common package install script
ARG INSTALL_ZSH="true"
ARG UPGRADE_PACKAGES="true"
ARG COMMON_SCRIPT_SOURCE="https://raw.githubusercontent.com/microsoft/vscode-dev-containers/master/script-library/common-debian.sh"
ARG COMMON_SCRIPT_SHA="dev-mode"

# Settings for installing Node.js.
ARG INSTALL_NODE="true"
ARG NODE_SCRIPT_SOURCE="https://raw.githubusercontent.com/microsoft/vscode-dev-containers/master/script-library/node-debian.sh"
ARG NODE_SCRIPT_SHA="dev-mode"

# ARG NODE_VERSION="lts/*"
ARG NODE_VERSION="14"
ENV NVM_DIR=/usr/local/share/nvm

# Have nvm create a "current" symlink and add to path to work around https://github.com/microsoft/vscode-remote-release/issues/3224
ENV NVM_SYMLINK_CURRENT=true
ENV PATH=${NVM_DIR}/current/bin:${PATH}

# Configure apt and install packages
RUN apt-get update \
    && export DEBIAN_FRONTEND=noninteractive \
    #
    # Verify git, common tools / libs installed, add/modify non-root user, optionally install zsh
    && apt-get -y install --no-install-recommends curl ca-certificates 2>&1 \
    && curl -sSL ${COMMON_SCRIPT_SOURCE} -o /tmp/common-setup.sh \
    && ([ "${COMMON_SCRIPT_SHA}" = "dev-mode" ] || (echo "${COMMON_SCRIPT_SHA} */tmp/common-setup.sh" | sha256sum -c -)) \
    && /bin/bash /tmp/common-setup.sh "${INSTALL_ZSH}" "${USERNAME}" "${USER_UID}" "${USER_GID}" "${UPGRADE_PACKAGES}" \
    #
    # Install Node.js
    && curl -sSL ${NODE_SCRIPT_SOURCE} -o /tmp/node-setup.sh \
    && ([ "${NODE_SCRIPT_SHA}" = "dev-mode" ] || (echo "${COMMON_SCRIPT_SHA} */tmp/node-setup.sh" | sha256sum -c -)) \
    && /bin/bash /tmp/node-setup.sh "${NVM_DIR}" "${NODE_VERSION}" "${USERNAME}" \
    #
    # Clean up
    && apt-get autoremove -y \
    && apt-get clean -y \
    && rm -f /tmp/common-setup.sh /tmp/node-setup.sh \
    && rm -rf /var/lib/apt/lists/* \
    #
    # Workspace
    && mkdir workspace \
    && chown -R ${NONROOT_USER}:root workspace


# Install Vim
RUN apt-get update && apt-get install -y \
    vim \
    && rm -rf /var/lib/apt/lists/*

# Set up a timezone in the devcontainer - necessary for anything timezone dependent
ENV TZ=Europe/London
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone \
 && apt-get update \
 && apt-get install --no-install-recommends -y \
    apt-utils \
    tzdata  \
 && apt-get autoremove -y \
 && apt-get clean -y \
 && rm -rf /var/lib/apt/lists/* 

ENV DOTNET_RUNNING_IN_CONTAINER=true 

# Copy across SSH keys so you can git clone
RUN mkdir /root/.ssh
RUN chmod 700 /root/.ssh

COPY .ssh/id_rsa /root/.ssh
RUN chmod 600 /root/.ssh/id_rsa

COPY .ssh/id_rsa.pub /root/.ssh
RUN chmod 644 /root/.ssh/id_rsa.pub

COPY .ssh/known_hosts /root/.ssh
RUN chmod 644 /root/.ssh/known_hosts  

# Disable initial git clone prompt
RUN echo "StrictHostKeyChecking no" >> /etc/ssh/ssh_config

# Copy across .env file so you can customise environment variables
# This will be copied into the root of the repo post git clone
COPY .env /.env
RUN chmod 644 /.env  

# Install dotnet entity framework tools
RUN dotnet tool install dotnet-ef --tool-path /usr/local/bin --version 3.1.2 

With this devcontainer you're good to go for an ASP.NET Core / JavaScript developer setup that is blazing fast! Remember to fire up Docker and give it goodly access to the resources of your host machine. All the CPUs, lots of memory and all the performance that there ought to be.

* "virus checkers" is a euphemism here for all the background tools that may be running. It was that or calling them "we are legion"

Saturday, 11 July 2020

Devcontainers and SSL interception

Devcontainers are cool. They are the infrastructure as code equivalent for developing software.

Imagine your new starter joins the team, you'd like them to be contributing code on day 1. But if the first thing that happens is you hand them a sheaf of paper upon which are the instructions for how to get their machines set up for development, well, maybe it's going to be a while. But if your project has a devcontainer then you're off to the races. One trusty git clone, fire up VS Code and they can get going.

That's the dream right?

I've recently been doing some work getting a project I work on set up with a devcontainer. As I've worked on that I've become aware of some of the hurdles that might hamper your adoption of devcontainers in a corporate environment.

Certificates: I'm starting with the man in the middle

It is a common practice in company networks to perform SSL interception. Not SSL inception; that'd be more fun.

SSL interception is the practice of installing a "man-in-the-middle" (MITM) CA certificate on users machines. When SSL traffic takes place from a users machine, it goes through a proxy. That proxy performs the SSL on behalf of that user and, if it's happy, supplies another certificate back to the users machine which satisfies the MITM CA certificate. So rather than seeing, for example, Google's certificate from https://google.com you'd see the one resulting from the SSL interception. You can read more here.

Now this is a little known and less understood practice. I barely understand it myself. Certificates are hard. Even having read the above you may be none the wiser about why this is relevant. Let's get to the broken stuff.

"Devcontainers don't work at work!"

So, you're ready to get going with your first devcontainer. You fire up the vscode-dev-containers repo and find the container that's going to work for you. Copy pasta the .devcontainer into your repo, install the Remote Development extension into VS Code and enter the Remote-Containers: Reopen Folder in Container. Here comes the future!

But when it comes to performing SSL inside the devcontainer, trouble awaits. Here's what a yarn install results in:

yarn install v1.22.4
[1/4] Resolving packages...
[2/4] Fetching packages...
error An unexpected error occurred: "https://registry.yarnpkg.com/@octokit/core/-/core-2.5.0.tgz: self signed certificate in certificate chain".

Oh no!

Gosh but it's okay - you're just bumping on the SSL interception. Why though? Well it's like this: when you fire up your devcontainer it builds a new Docker container. It's as well to imagine the container as a virtual operating system. So what's the difference between this operating system and the one our machine is running? Well a number of things, but crucially our host operating system has the MITM CA certificate installed. So when we SSL, we have the certificate that will match up with what the proxy sends back to us certificate-wise. And inside our trusty devcontainer we don't have that. Hence the sadness.

Devcontainer + MITM cert = working

We need to do two things to get this working:

  1. Acquire the requisite CA certificate(s) from your friendly neighbourhood networking team. Place them in a certs folder inside your repo, in the .devcontainer folder.
  2. Add the following lines to your .devcontainer/Dockerfile, just after the initial FROM statement:
# Because MITM certificates
COPY certs/. /usr/local/share/ca-certificates/
ENV NODE_EXTRA_CA_CERTS=/usr/local/share/ca-certificates/mitm.pem
RUN update-ca-certificates

Which does the following:

  • Copies the certs into the devcontainer
  • This is a Node example and so we set an environment variable called NODE_EXTRA_CA_CERTS which points to the path of your MITM CA certificate file inside your devcontainer.
  • updates the directory /etc/ssl/certs to hold SSL certificates and generates ca-certificates.crt

With these in place then you should be able to build your devcontainer with no SSL trauma. Enjoy!

Sunday, 21 June 2020

Task.WhenAll / Select is a footgun 👟🔫

This post differs from my typical fayre. Most often I write "here's how to do a thing". This is not that. It's more "don't do this thing I did". And maybe also, "how can we avoid a situation like this happening again in future?". On this topic I very much don't have all the answers - but by putting my thoughts down maybe I'll learn and maybe others will educate me. I would love that!

Doing things that don't scale

The platform that I work on once had zero users. We used to beg people to log in and see what we had built. Those days are (happily) but a memory. We're getting popular.

As our platform has grown in popularity it has revealed some bad choices we made. Approaches that look fine on the surface (and that work just dandy when you have no users) may start to cause problems as your number of users grows.

I wanted to draw attention to one approach in particular that impacted us severely. In this case "impacted us severely" is a euphemism for "brought the site down and caused a critical incident".

You don't want this to happen to you. Trust me. So, what follows is a cautionary tale. The purpose of which is simply this: reader, do you have code of this ilk in your codebase? If you do: out, damn'd spot! out, I say!

So cool, so terrible

I love LINQ. I love a declarative / functional style of coding. It appeals to me on some gut level. I find it tremendously readable. Read any C# of mine and the odds are pretty good that you'll find some LINQ in the mix.

Imagine this scenario: you have a collection of user ids. You want to load the details of each user represented by their id from an API. You want to bag up all of those users into some kind of collection and send it back to the calling code.

Reading that, if you're like me, you're imagining some kind of map operation which loads the user details for each user id. Something like this:

var users = userIds.Select(userId => GetUserDetails(userId)).ToArray(); // users is User[]

Lovely. But you'll note that I'm loading users from an API. Oftentimes, APIs are asynchronous. Certainly, in my case they were. So rather than calling a GetUserDetails function I found myself calling a GetUserDetailsAsync function, behind which an HTTP request is being sent and, later, a response is being returned.

So how do we deal with this? Task.WhenAll my friends!

var userTasks = userIds.Select(userId => GetUserDetailsAsync(userId));
var users = await Task.WhenAll(tasks); // users is User[]

It worked great! Right up until to the point where it didn't. These sorts of shenanigans were fine when we had a minimal number of users... But there came a point where problems arose. It got to the point where that simple looking mapping operation became a cause of many, many, many HTTP requests being fired concurrently. Then bad things started to happen. Not only did we realise we were launching a denial of service attack on the API we were consuming, we were bringing our own application to collapse.

Not a proud day.

What is the problem?

Through log analysis, code reading and speculation, (with the help of the invaluable Robski) we came to realise that the cause of our woes was the Task.WhenAll / Select combination. Exercising that codepath was a surefire way to bring the application to its knees.

As I read around on the topic I happened upon Mark Heath's excellent list of Async antipatterns. Number #6 on the list is "Excessive parallelization". It describes a nearly identical scenario to my own:

Now, this does "work", but what if there were 10,000 orders? We've flooded the thread pool with thousands of tasks, potentially preventing other useful work from completing. If ProcessOrderAsync makes downstream calls to another service like a database or a microservice, we'll potentially overload that with too high a volume of calls.

We're definitely overloading the API we're consuming with too high a volume of calls. I have to admit that I'm less clear on the direct reason that a Task.WhenAll / Select combination could prove fatal to our application. Mark suggests this approach will flood the thread pool with tasks. As I read around on async and await it's repeated again and again that a Task is not the same thing as a Thread. I have to hold my hands up here and say that I don't understand the implementation of async / await in C# well enough. These docs are helpful but I still don't think the penny has fully dropped for me yet. I will continue to read.

One thing we learned as we debugged the production k8s pod was that, prior to its collapse, our app appeared to be opening up 1 million connections to the API we were consuming. Which seemed a bit much. Worthy of investigation. It's worth saying that we're not certain this is exactly what is happening; we have less instrumentation in place than we'd like. But some fancy wc grepping on Robski's behalf suggested this was the case.

What will we change in future?

A learning that came out of this for us was this: we need more metrics exposed. We don't understand our application's behaviour under load as well as we'd like. So we're planning to do some work with App Metrics and Grafana so we've a better idea of how our application performs. If you want to improve something, first measure it.

Another fly in the ointment was that we were unable to reproduce the issue when running locally. It's worth saying here that I develop on a Windows machine and, when deployed, our application runs in a (Linux) Docker container. So there's a difference and a distance between our development experience and our running one.

I'm planning to migrate to developing in a devcontainer where that's possible. That should narrow the gap between our production experience and our development one. Reducing the difference between the two is always useful as it means you're less likely to get different behaviour (ie "problems") in production as compared to development. I'm curious as to whether I'll be able to replicate that behaviour in a devcontainer.

What did we do right now?

To solve the immediate issue we were able to pivot away to a completely different approach. We moved aggregation from our ASP.NET Core web application to our TypeScript / React client with a (pretty sweet) custom hook. The topic for a subsequent blog post.

Moving to a different approach solved my immediate issue. But it left me puzzling. What was actually going wrong? Is it thread pool exhaustion? Is it something else? So many possibilities!

If anyone has any insights they'd like to share that would be incredible! I've also asked a question on Stack Overflow which has kindly had answers from generous souls. James Skimming's answer lead me to Steve Gordon's excellent post on connection pooling which I'm still absorbing and seems like it could be relevant.

Thursday, 21 May 2020

Autofac, WebApplicationFactory and integration tests

Updated 2nd Oct 2020: for an approach that works with Autofac 6 see this post.


This is one of those occasions where I'm not writing up my own work so much as my discovery after in depth googling.

Integration tests with ASP.NET Core are the best. They spin up an in memory version of your application and let you fire requests at it. They've gone through a number of iterations since ASP.NET Core has been around. You may also be familiar with the TestServer approach of earlier versions. For some time, the advised approach has been using WebApplicationFactory.

What makes this approach particularly useful / powerful is that you can swap out dependencies of your running app with fakes / stubs etc. Just like unit tests! But potentially more useful because they run your whole app and hence give you a greater degree of confidence. What does this mean? Well, imagine you changed a piece of middleware in your application; this could potentially break functionality. Unit tests would probably not reveal this. Integration tests would.

There is a fly in the ointment. A hair in the gazpacho. ASP.NET Core ships with dependency injection in the box. It has its own Inversion of Control container which is perfectly fine. However, many people are accustomed to using other IOC containers such as Autofac.

What's the problem? Well, swapping out dependencies registered using ASP.NET Core's IOC requires using a hook called ConfigureTestServices. There's an equivalent hook for swapping out services registered using a custom IOC container: ConfigureTestContainer. Unfortunately, there is a bug in ASP.NET Core as of version 3.0: When using GenericHost, in tests ConfigureTestContainer is not executed

This means you cannot swap out dependencies that have been registered with Autofac and the like. According to the tremendous David Fowler of the ASP.NET team, this will hopefully be resolved.

In the meantime, there's a workaround thanks to various commenters on the thread. Instead of using WebApplicationFactory directly, subclass it and create a custom AutofacWebApplicationFactory (the name is not important). This custom class overrides the behavior of ConfigureServices and CreateHost with a CustomServiceProviderFactory:

namespace My.Web.Tests.Helpers {
    /// <summary>
    /// Based upon https://github.com/dotnet/AspNetCore.Docs/tree/master/aspnetcore/test/integration-tests/samples/3.x/IntegrationTestsSample
    /// </summary>
    /// <typeparam name="TStartup"></typeparam>
    public class AutofacWebApplicationFactory<TStartup> : WebApplicationFactory<TStartup> where TStartup : class {
        protected override void ConfigureWebHost(IWebHostBuilder builder) {
            builder.ConfigureServices(services => {
                    services.AddSingleton<IAuthorizationHandler>(new PassThroughPermissionedRolesHandler());
                })
                .ConfigureTestServices(services => {
                }).ConfigureTestContainer<Autofac.ContainerBuilder>(builder => {
                    // called after Startup.ConfigureContainer
                });
        }

        protected override IHost CreateHost(IHostBuilder builder) {
            builder.UseServiceProviderFactory(new CustomServiceProviderFactory());
            return base.CreateHost(builder);
        }
    }

    /// <summary>
    /// Based upon https://github.com/dotnet/aspnetcore/issues/14907#issuecomment-620750841 - only necessary because of an issue in ASP.NET Core
    /// </summary>
    public class CustomServiceProviderFactory : IServiceProviderFactory<CustomContainerBuilder> {
        public CustomContainerBuilder CreateBuilder(IServiceCollection services) => new CustomContainerBuilder(services);

        public IServiceProvider CreateServiceProvider(CustomContainerBuilder containerBuilder) =>
        new AutofacServiceProvider(containerBuilder.CustomBuild());
    }

    public class CustomContainerBuilder : Autofac.ContainerBuilder {
        private readonly IServiceCollection services;

        public CustomContainerBuilder(IServiceCollection services) {
            this.services = services;
            this.Populate(services);
        }

        public Autofac.IContainer CustomBuild() {
            var sp = this.services.BuildServiceProvider();
#pragma warning disable CS0612 // Type or member is obsolete
            var filters = sp.GetRequiredService<IEnumerable<IStartupConfigureContainerFilter<Autofac.ContainerBuilder>>>();
#pragma warning restore CS0612 // Type or member is obsolete

            foreach (var filter in filters) {
                filter.ConfigureContainer(b => { }) (this);
            }

            return this.Build();
        }
    }
}

I'm going to level with you; I don't understand all of this code. I'm not au fait with the inner workings of ASP.NET Core or Autofac but I can tell you what this allows. With this custom WebApplicationFactory in play you get ConfigureTestContainer back in the mix! You get to write code like this:

using System;
using System.Net;
using System.Net.Http.Headers;
using System.Threading.Tasks;
using FakeItEasy;
using FluentAssertions;
using Microsoft.AspNetCore.TestHost;
using Microsoft.Extensions.DependencyInjection;
using Xunit;
using Microsoft.Extensions.Options;
using Autofac;
using System.Net.Http;
using Newtonsoft.Json;

namespace My.Web.Tests.Controllers
{
    public class MyControllerTests : IClassFixture<AutofacWebApplicationFactory<My.Web.Startup>> {
        private readonly AutofacWebApplicationFactory<My.Web.Startup> _factory;

        public MyControllerTests(
            AutofacWebApplicationFactory<My.Web.Startup> factory
        ) {
            _factory = factory;
        }

        [Fact]
        public async Task My() {
            var fakeSomethingService = A.Fake<IMySomethingService>();
            var fakeConfig = Options.Create(new MyConfiguration {
                SomeConfig = "Important thing",
                OtherConfigMaybeAnEmailAddress = "[email protected]"
            });

            A.CallTo(() => fakeSomethingService.DoSomething(A<string>.Ignored))
                .Returns(Task.FromResult(true));

            void ConfigureTestServices(IServiceCollection services) {
                services.AddSingleton(fakeConfig);
            }

            void ConfigureTestContainer(ContainerBuilder builder) {
                builder.RegisterInstance(fakeSomethingService);
            }

            var client = _factory
                .WithWebHostBuilder(builder => {
                    builder.ConfigureTestServices(ConfigureTestServices);
                    builder.ConfigureTestContainer<Autofac.ContainerBuilder>(ConfigureTestContainer);
                })
                .CreateClient();

            // Act
            var request = StringContent("{\"sommat\":\"to see\"}");
            request.Headers.ContentType = MediaTypeHeaderValue.Parse("application/json");
            var response = await client.PostAsync("/something/submit", request);

            // Assert
            response.StatusCode.Should().Be(HttpStatusCode.OK);

            A.CallTo(() => fakeSomethingService.DoSomething(A<string>.Ignored))
                .MustHaveHappened();
        }

    }
}

Sunday, 10 May 2020

From react-window to react-virtual

The tremendous Tanner Linsley recently released react-virtual. react-virtual provides "hooks for virtualizing scrollable elements in React".

I was already using the (also excellent) react-window for this purpose. react-window does the virtualising job and does it very well indeed However, I was both intrigued by the lure of the new shiny thing. I've also never been the biggest fan of react-window's API. So I tried switching over from react-window to react-virtual as an experiment. To my delight, the experiment went so well I didn't look back!

What did I get out of the switch?

  • Simpler code / nicer developer ergonomics. The API for react-virtual allowed me to simplify my code and lose a layer of components.
  • TypeScript support in the box
  • Improved perceived performance. I didn't run any specific tests to quantify this, but I can say that the same functionality now feels snappier.

I tweeted my delight at this and Tanner asked if there was commit diff I could share. I couldn't as it's a private codebase, but I thought it could form the basis of a blogpost.

In case you hadn't guessed, this is that blog post...

Make that change

So what does the change look like? Well first remove react-window from your project:

yarn remove react-window @types/react-window

Add the dependency to react-virtual:

yarn add react-virtual

Change your imports from:

import { FixedSizeList, ListChildComponentProps } from 'react-window';

to:

import { useVirtual } from 'react-virtual';

Change your component code from:

type ImportantDataListProps = {
    classes: ReturnType<typeof useStyles>;
    importants: ImportantData[];
};

const ImportantDataList: React.FC<ImportantDataListProps> = React.memo(props => (
    <FixedSizeList
        height={400}
        width={'100%'}
        itemSize={80}
        itemCount={props.importants.length}
        itemData={props}
    >
        {RenderRow}
    </FixedSizeList>
));

type ListItemProps = {
    classes: ReturnType<typeof useStyles>;
    importants: ImportantData[];
};

function RenderRow(props: ListChildComponentProps) {
    const { index, style } = props;
    const { importants, classes } = props.data as ListItemProps;
    const important = importants[index];

    return (
        <ListItem button style={style} key={index}>
            <ImportantThing classes={classes} important={important} />
        </ListItem>
    );
}

Of the above you can delete the ListItemProps type and the associate RenderRow function. You won't need them again! There's no longer a need to pass down data to the child element and then extract it for usage; it all comes down into a single simpler component.

Replace the ImportantDataList component with this:

const ImportantDataList: React.FC<ImportantDataListProps> = React.memo(props => {
    const parentRef = React.useRef<HTMLDivElement>(null);

    const rowVirtualizer = useVirtual({
        size: props.importants.length,
        parentRef,
        estimateSize: React.useCallback(() => 80, []), // This is just a best guess
        overscan: 5
    });

    return (
            <div
                ref={parentRef}
                style={{
                    width: `100%`,
                    height: `500px`,
                    overflow: 'auto'
                }}
            >
                <div
                    style={{
                        height: `${rowVirtualizer.totalSize}px`,
                        width: '100%',
                        position: 'relative'
                    }}
                >
                    {rowVirtualizer.virtualItems.map(virtualRow => (
                        <div
                            key={virtualRow.index}
                            ref={virtualRow.measureRef}
                            className={props.classes.hoverRow}
                            style={{
                                position: 'absolute',
                                top: 0,
                                left: 0,
                                width: '100%',
                                height: `${virtualRow.size}px`,
                                transform: `translateY(${virtualRow.start}px)`
                            }}
                        >
                            <ImportantThing
                                classes={props.classes}
                                important={props.importants[virtualRow.index]}
                            />
                        </div>
                    ))}
                </div>
            </div>
    );
});

And you are done! Thanks Tanner for this tremendous library!

Saturday, 4 April 2020

Up to the clouds!

This last four months has been quite the departure for me. Most typically I find myself building applications; for this last period of time I've been taking the platform that I work on, and been migrating it from running on our on premise servers to running in the cloud.

This turned out to be much more difficult than I'd expected and for reasons that often surprised me. We knew where we wanted to get to, but not all of what we'd need to do to get there. So many things you can only learn by doing. Whilst these experiences are still fresh in my mind I wanted to document some of the challenges we faced.

The mission

At the start of January, the team decided to make a concerted effort to take our humble ASP.NET Core application and migrate it to the cloud. We sat down with some friends from the DevOps team who are part of our organisation. We're fortunate in that these marvellous people are very talented engineers indeed. It was going to be a collaboration between our two teams of budding cloudmongers that would make this happen.

Now our application is young. It is not much more than a year old. However it is growing fast. And as we did the migration from on premise to the cloud, that wasn't going to stop. Development of the application was to continue as is, shipping new versions daily. Without impeding that, we were to try and get the application migrated to the cloud.

I would liken it to boarding a speeding train, fighting your way to the front, taking the driver hostage and then diverting the train onto a different track. It was challenging. Really, really challenging.

So many things had to change for us to get from on premise servers to the cloud, all the while keeping our application a going (and shipping) concern. Let's go through them one by one.

Kubernetes and Docker

Our application was built using ASP.NET Core. A technology that is entirely cloud friendly (that's one of the reasons we picked it). We were running on a collection of hand installed, hand configured Windows servers. That had to change. We wanted to move our application to run on Kubernetes; so we didn't have to manually configure servers. Rather k8s would manage the provisioning and deployment of containers running our application. Worth saying now: I knew nothing about Kubernetes. Or nearly nothing. I learned a bunch along the way, but, as I've said, this was a collaboration between our team and the mighty site reliability engineers of the DevOps team. They knew a lot about this k8s stuff and moreoften than not, our team stood back and let them work their magic.

In order that we could migrate to running in k8s, we first needed to containerise our application. We needed a Dockerfile. There followed a good amount of experimentation as we worked out how to build ourselves images. There's an art to building an optimal Docker image.

So that we can cover a lot of ground, this post will remain relatively high level. So here's a number of things that we encountered along the way that are worth considering:

  • Multi-stage builds were an absolute necessity for us. We'd build the front end of our app (React / TypeScript) using one stage with a Node base image. Then we'd build our app using a .NET Core SDK base image. Finally, we'd use a ASP.Net image to run the app; copying in the output of previous stages.
  • Our application accesses various SQL Server databases. We struggled to get our application to connect to them. The issue related to the SSL configuration of our runner image. The fix was simple but frustrating; use a -bionic image as it has the configuration you need. We found that gem here.
  • Tests. Automated tests. We want to run them in our build; but how? Once more multi-stage builds to the rescue. We'd build our application, then in a separate stage we'd run the tests; copying in the app from the build stage. If the tests failed, the build failed. If they passed then the intermediate stage containing the tests would be discarded by Docker. No unnecessary bloat of the image; all that testing goodness still; now in containerised form!

Jenkins

Our on premise world used TeamCity for our continuous integration needs and Octopus for deployment. We liked these tools well enough; particularly Octopus. However, the DevOps team were very much of the mind that we should be use Jenkins instead. And Pipeline. It was here that we initially struggled. To quote the docs:

Jenkins Pipeline (or simply "Pipeline" with a capital "P") is a suite of plugins which supports implementing and integrating continuous delivery pipelines into Jenkins.

Whilst continuous delivery is super cool, and is something our team was interested in, we weren't ready for it yet. We didn't yet have the kind of automated testing in place that gave us the confidence that we'd need to move to it. One day, but not today. For now there was still some manual testing done on each release, prior to shipping. Octopus suited us very well here as it allowed us to deploy, on demand, a build of our choice to a given environment. So the question was: what to do? Fortunately the immensely talented Aby Egea came up with a mechanism that supported that very notion. A pipeline that would, optionally, deploy our build to a specified environment. So we were good!

One thing we got to really appreciate about Jenkins was that the build is scripted with a Jenkinsfile. This was in contrast to our TeamCity world where it was all manually configured. Configuration as code is truly a wonderful thing as your build pipeline becomes part of your codebase; open for everyone to see and understand. If anyone wants to change the build pipeline it has to get code reviewed like everything else. It was as code in our Jenkinsfile that the deployment mechanism lived.

Vault

Another thing that we used Octopus for was secrets. Applications run on configuration; these are settings that drive the behaviour of your application. A subset of configuration is "secrets". Secrets are configuration that can't be stored in source code; they would represent a risk if they did. For instance a database connection string. We'd been merrily using Octopus for this; as Octopus deploys an application to a server it enriches the appsettings.json file with any required secrets.

Without Octopus in the mix, how were we to handle our secrets? The answer is with Hashicorp Vault. We'd store our secrets in there and, thanks to clever work by Robski of the DevOps team, when our container was brought up by Kubernetes, it would mount into the filesystem an appsettings.Vault.json file which we read thanks to our trusty friend .AddJsonFile with optional: true. (As the file didn't exist in our development environment.)

Hey presto! Safe secrets in k8s.

Networking

Our on premise servers sat on the company network. They could see everything that there was to see. All the other servers around them on the network, bleeping and blooping. The opposite was true in AWS. There was nothing to see. Nothing to access. As it should be. It's safer that way should a machine become compromised. For each database and each API our application depended upon, we needed to specifically whitelist access.

Kerberos

There's always a fly in the ointment. A nasty surprise on a dark night. Ours was realising that our application depended upon an API that was secured using Windows Authentication. Our application was accessing it by running under a service account which had been permissioned to access it. However, in AWS, our application wasn't running as under a service account on the company network. Disappointingly, in the short term the API was not going to support an alternate authentication mechanism.

What to do? Honestly it wasn't looking good. We were considering proxying through one of our Windows servers just to get access to that API. I was tremendously disappointed. At this point our hero arrived; one JMac hacked together a Kerberos sidecar approach one weekend. You can see a similar approach here. This got us to a point that allowed us to access the API we needed to.

I'm kind of amazed that there isn't better documentation out there around have a Kerberos sidecar in a k8s setup. Tragically Windows Authentication is a widely used authentication mechanism. That being the case, having good docs to show how you can get a Kerberos sidecar in place would likely greatly advance the ability of enterprises to migrate to the cloud. The best docs I've found are here. It is super hard though. So hard!

Hangfire

We were using Hosted Services to perform background task running in our app. The nature of our background tasks meant that it was important to only run a single instance of a background task at a time. Or bad things would happen. This was going to become a problem since we had ambitions to be able to horizontally scale our application; to add new pods as running our app as demand determined.

So we started to use Hangfire to perform task running in our app. With Hangfire, when a job is picked up it gets locked so other servers can't pick it up. That's what we need.

Hangfire is pretty awesome. However it turns out that there's quirks when you move to a containerised environment. We have a number of recurring jobs that are scheduled to run at certain dates and times. In order that Hangfire can ascertain what time it is, it needs a timezone. It turns out that timezones on Windows != timezones in Docker / Linux.

This was a problem because, as we limbered up for the great migration, we were trying to run our cloud implementation side by side with our on premise one. And Windows picked a fight with Linux over timezones. You can see others bumping into this condition here. We learned this the hard way; jobs mysteriously stopping due to timezone related errors. Windows Hangfire not able to recognise Linux Hangfire timezones and vica versa.

The TL;DR is that we had to do a hard switch with Hangfire; it couldn't run side by side. Not the end of the world, but surprising.

Azure Active Directory Single Sign-On

Historically our application had used two modes of authentication; Windows Authentication and cookies. Windows Authentication doesn't generally play nicely with Docker. It's doable, but it's not the hill you want to die on. So we didn't; we swapped out Windows Authentication for Azure AD SSO and didn't look back.

We also made some changes so our app would support cookies auth alongside Azure AD auth; I've written about this previously.

Do the right thing and tell people about it

We're there now; we've made the move. It was a difficult journey but one worth making; it sets up our platform for where we want to take it in the future. Having infrastructure as code makes all kinds of approaches possible that weren't before. Here's some things we're hoping to get out of the move:

  • blue green deployments - shipping without taking down our platform
  • provision environments on demand - currently we have a highly contended situation when it comes to test environments. With k8s and AWS we can look at spinning up environments as we need them and throwing them away also
  • autoscaling for need - we can start to look at spinning up new containers in times of high load and removing excessive containers in times of low load

We've also become more efficient as a team. We are no longer maintaining servers, renewing certificates, installing software, RDPing onto boxes. All that time and effort we can plough back into making awesome experiences for our users.

There's a long list of other benefits and it's very exciting indeed! It's not enough for us to have done this though. It's important that we tell the story of what we've done and how and why we've done it. That way people have empathy for the work. Also they can start to think about how they could start to reap similar benefits themselves. By talking to others about the road we've travelled, we can save them time and help them to travel a similar road. This is good for them and it's good for us; it helps our relationships and it helps us all to move forwards together.

A rising tide lifts all boats. By telling others about our journey, we raise the water level. Up to the clouds!