bicep build
and it would run the linter as part of the build process. This was a little confusing as it was not obvious that the linter was running.
As of version 0.21.1 there is a dedicated bicep lint
command. This is a nice step forwards; it allows you to explicitly lint your your code, rather than have it happen as a side effect of build. And it is useful if you want to run the linter as part of a CI/CD pipeline. What's more the bicep lint
command is now available in the Azure CLI as well. You can run az bicep lint
to lint your bicep files.
In this post we'll look at how to run lint Bicep in Azure Pipelines and GitHub Actions, and surface the output in the UI.
The general approach is the same for both Azure Pipelines and GitHub Actions. One way or another, we'll run the Bicep lint
command to lint our Bicep files and capture the output. As yet, there is no option to export the results of the lint command as a file. This may come, and there is a discussion about it. However, there is a way to achieve our goal, which came out in discussion with Anthony Martin of the Bicep team. We can write the output of the lint
command to a file like so:
bicep lint main.bicep --diagnostics-format sarif > lint.sarif
This will write the output of the lint
command to a file called lint.sarif
. This is a SARIF file. SARIF stands for Static Analysis Results Interchange Format. It's a standard for representing the results of static analysis tools. It's a JSON file, easy to parse and has integrations with GitHub Actions / Azure Pipelines. An example SARIF output is below:
{
"$schema": "https://schemastore.azurewebsites.net/schemas/json/sarif-2.1.0-rtm.6.json",
"version": "2.1.0",
"runs": [
{
"tool": {
"driver": {
"name": "bicep"
}
},
"results": [
{
"ruleId": "no-unused-vars",
"message": {
"text": "Variable \"unusedVar\" is declared but never used. [https://aka.ms/bicep/linter/no-unused-vars]"
},
"locations": [
{
"physicalLocation": {
"artifactLocation": {
"uri": "file:///home/runner/work/blog.johnnyreilly.com/blog.johnnyreilly.com/./infra/main.bicep"
},
"region": {
"startLine": 19,
"charOffset": 5
}
}
}
]
}
],
"columnKind": "utf16CodeUnits"
}
]
}
In the example above we directly used the bicep lint
command. An alternative approach is to use the Azure CLI like so:
az bicep lint --file main.bicep --diagnostics-format sarif > lint.sarif
This is a little more verbose, but it does mean that you don't need to install the Bicep CLI on your build agent. You can use the Azure CLI instead; and this is already a first class citizen of Azure Pipelines and GitHub Actions, with dedicated support. (That said, there is a slight issue with this approach at the time of writing, which we'll come to later.)
However we generate it, we can take the SARIF file and use it to surface the results of the linting process in Azure Pipelines and GitHub Actions.
We'll start off by looking at how to lint Bicep in GitHub Actions with the Azure CLI. But before we do that, we need something to lint. Within your main.bicep
file you should have something like this:
var unusedVar = 1 // unused variable
As it suggests, this is an unused variable. We're just using this to demonstrate the linting process. And alongside that, we want a bicepconfig.json
file that looks like this:
{
"analyzers": {
"core": {
"rules": {
"no-unused-vars": {
"level": "warning"
}
}
}
}
}
This explicitly enables the no-unused-vars
rule, and sets it to warning
level. This means that the linter will warn us about unused variables, but it won't fail the build. We could set it to error
level, and then the build would fail if there were any unused variables. We'll come back to this later.
In a GitHub workflow in your repository you should have steps like these:
- name: Lint Bicep
uses: azure/CLI@v1
with:
inlineScript: |
az bicep install
az bicep lint --file ./infra/main.bicep --diagnostics-format sarif > bicep.sarif
- name: Upload SARIF
if: (success() || failure())
uses: github/codeql-action/upload-sarif@v3
with:
category: bicep
sarif_file: bicep.sarif
The above:
lint
command and writes the results to a file called bicep.sarif
The upload-sarif
action is provided by GitHub. It allows surfacing the results of static analysis tools in GitHub. Doing this will show the results of the linting process in the GitHub UI, and it will also show them in the GitHub Security / Code Scanning UI, like so:
We can also lint Bicep in GitHub Actions with the Bicep CLI. At the time of writing, there's a reason you might want to favour this approach over the Azure CLI approach. I won't drill into it in depth, but at present if az bicep lint
fails / returns a non-0 exit code then no output is produced. We could make this happen by updating the bicepconfig.json
to dial up the no-unused-vars
rule to error
level, like so:
{
"analyzers": {
"core": {
"rules": {
"no-unused-vars": {
"level": "error"
}
}
}
}
}
Now an unused variable will cause the build to fail. But the output of the lint
command will not be surfaced in the GitHub UI. This is because the Azure CLI is not surfacing the output of the lint
command when it fails.
The issue with the Azure CLI will hopefully be remedied; you can track that here. For now you can use the Bicep CLI directly, where this isn't an issue. We'll do that now.
I'm basing this approach on Anthony Martin's example of Bicep linting with GitHub Actions:
- name: Setup Bicep
uses: anthony-c-martin/setup-bicep@v0.3
- name: Lint Bicep
run: |
bicep lint ./infra/main.bicep --diagnostics-format sarif > bicep.sarif
- name: Upload SARIF
if: always()
uses: github/codeql-action/upload-sarif@v3
with:
category: bicep
sarif_file: bicep.sarif
The above does the same as the Azure CLI approach, but it uses the Bicep CLI directly. The setup-bicep
action is provided by Anthony Martin and installs the Bicep CLI on your build agent.
As I say, right now you may want to favour this approach over the Azure CLI approach to cover both surfacing warnings and errors. But hopefully that will change soon.
We've now seen how to lint Bicep in GitHub Actions with both the Azure CLI and the Bicep CLI. We can do the same in Azure Pipelines; but only the Azure CLI approach (as I'm not sure if there's a setup-bicep
equivalent for Azure Pipelines).
How you want to surface the results in Azure Pipelines is up to you. You could surface into the "Scans" portion of Azure Pipelines UI or into "Tests". I'll show you how to do both.
To surface the results in the scans part of Azure Pipelines you need to publish the SARIF file as a build artifact. You can do that like so:
jobs:
- job: LintInfra
displayName: Lint Infra
dependsOn: []
pool:
vmImage: 'ubuntu-latest'
steps:
- task: AzureCLI@2
displayName: Lint main.bicep
inputs:
azureSubscription: service-connection-with-access-to-registry # you may not need this
scriptType: bash
scriptLocation: inlineScript
inlineScript: |
az bicep install
az bicep lint --file infra/main.bicep --diagnostics-format sarif > $(System.DefaultWorkingDirectory)/bicep.sarif
- task: PublishBuildArtifacts@1
condition: always()
inputs:
pathToPublish: $(System.DefaultWorkingDirectory)/bicep.sarif
artifactName: CodeAnalysisLogs # required to show up in the scans tab
The above is essentially the same as the GitHub Actions example, but it uses the Azure CLI instead of the Bicep CLI. The PublishBuildArtifacts
task is provided by Azure Pipelines. It allows you to publish build artifacts, which will show up in the Scans part of Azure Pipelines. You can see the results of the linting process in Scans, like so:
You'll notice that the path above has a file:///home/vsts/work/1/s
prefix before the bicep path report of infra/main.bicep
. This is unfortunate and breaks "clickability". You cannot click on this and be taken to the file. It's possible to remedy this behaviour by doing a little find and replace magic on the SARIF file. You don't need to do this, but it does add to the developer experience.
Below is the same portion of the Azure Pipelines yaml file but with some additional bash that will use sed
to replace all instances of the file:///home/vsts/work/1/s
prefix with an empty string:
jobs:
- job: LintInfra
displayName: Lint Infra
dependsOn: []
pool:
vmImage: 'ubuntu-latest'
steps:
- task: AzureCLI@2
displayName: Lint main.bicep
inputs:
azureSubscription: service-connection-with-access-to-registry # you may not need this
scriptType: bash
scriptLocation: inlineScript
inlineScript: |
az bicep install
az bicep lint --file infra/main.bicep --diagnostics-format sarif > $(System.DefaultWorkingDirectory)/bicep.sarif
STRING_TO_REPLACE='file://$(Build.SourcesDirectory)/'
echo "##[group]Bicep linting results before $STRING_TO_REPLACE replace:"
cat $(System.DefaultWorkingDirectory)/bicep.sarif
echo "##[endgroup]"
sed -i "s|$STRING_TO_REPLACE||g" $(System.DefaultWorkingDirectory)/bicep.sarif
echo "##[group]Bicep linting results after $STRING_TO_REPLACE replace:"
cat $(System.DefaultWorkingDirectory)/bicep.sarif
echo "##[endgroup]"
- task: PublishBuildArtifacts@1
condition: always()
inputs:
pathToPublish: $(System.DefaultWorkingDirectory)/bicep.sarif
artifactName: CodeAnalysisLogs # required to show up in the scans tab
And hey presto! Clickability restored.
You can also surface the results in the tests part of Azure Pipelines. To do that we're going to borrow a suggestion from Anthony Martin and use the sarif-junit
package. This package allows us to convert a SARIF file to a JUnit file. JUnit is a standard for representing test results and can be used with the PublishTestResults
task.
jobs:
- job: LintInfra
displayName: Lint Infra
dependsOn: []
pool:
vmImage: 'ubuntu-latest'
steps:
- task: NodeTool@0
displayName: 'Install Node.js'
inputs:
versionSpec: '18.x'
- task: AzureCLI@2
displayName: Lint main.bicep
inputs:
azureSubscription: service-connection-with-access-to-registry # you may not need this
scriptType: bash
scriptLocation: inlineScript
inlineScript: |
az bicep install
az bicep lint --file infra/main.bicep --diagnostics-format sarif > $(System.DefaultWorkingDirectory)/bicep.sarif
npx -y sarif-junit -i $(System.DefaultWorkingDirectory)/bicep.sarif -o $(System.DefaultWorkingDirectory)/bicep.xml
- task: PublishTestResults@2
condition: always()
inputs:
testResultsFormat: 'JUnit'
testResultsFiles: '$(System.DefaultWorkingDirectory)/bicep.xml'
So the above is the same approach again but requires Node.js to be installed, and the results end up in the Tests part of Azure Pipelines. You can see the results of the linting process in Tests, like so:
That's it! We've seen how to lint Bicep in Azure Pipelines and GitHub Actions. We've seen how to surface the results in the scans and tests parts of Azure Pipelines and in GitHub. We've seen how to do it with the Azure CLI and the Bicep CLI. And we've seen how to do it with warnings and errors. Hopefully this will help you to ensure that your Bicep files conform to best practices.
Many thanks to Anthony Martin for his help, which laid the groundwork for much of what we explored in this post.
]]>I wanted to migrate these plugins to Docusaurus 3. This post is about how I did that - and if you've got a rehype plugin it could probably provide some guidance on the changes you'd need to make.
The Docusaurus team put out a blog post on preparing for the Docusaurus 3 migration. Part of that post mentions MDX plugins:
All the official packages (Unified, Remark, Rehype...) in the MDX ecosystem are now ES Modules only and do not support CommonJS anymore.
This affects how you write your plugins. It also has a bearing on how you import your plugins, given that the Docusaurus configuration file itself is still CommonJS. The post adds:
If you created custom Remark or Rehype plugins, you may need to refactor those, or eventually rewrite them completely, due to how the new AST is structured.
This turned out to be the case for me. I had to rewrite my plugins completely. I'll go through each of them in turn.
fetchpriority
pluginThe fetchpriority
plugin is a rehype plugin that I wrote to improve the Core Web Vitals of my blog. It does this by making the first image on a page eager loaded with fetchpriority="high"
and lazy loading all other images. The Docusaurus 2 / MDX 1 code looked like this:
// @ts-check
const visit = require('unist-util-visit');
/**
* Create a rehype plugin that will make the first image eager loaded with fetchpriority="high" and lazy load all other images
* @returns rehype plugin that will make the first image eager loaded with fetchpriority="high" and lazy load all other images
*/
function imageFetchPriorityRehypePlugin() {
/** @type {Map<string, string>} */ const files = new Map();
/** @type {import('unified').Transformer} */
return (tree, vfile) => {
visit(tree, ['element', 'jsx'], (node) => {
if (node.type === 'element' && node['tagName'] === 'img') {
// handles nodes like this:
// {
// type: 'element',
// tagName: 'img',
// properties: {
// src: 'https://some.website.com/cat.gif',
// alt: null
// },
// ...
// }
const key = `img|${vfile.history[0]}`;
const imageAlreadyProcessed = files.get(key);
const fetchpriorityThisImage =
!imageAlreadyProcessed ||
imageAlreadyProcessed === node['properties']['src'];
if (!imageAlreadyProcessed) {
files.set(key, node['properties']['src']);
}
if (fetchpriorityThisImage) {
node['properties'].fetchpriority = 'high';
node['properties'].loading = 'eager';
} else {
node['properties'].loading = 'lazy';
}
} else if (node.type === 'jsx' && node['value']?.includes('<img ')) {
// handles nodes like this:
// {
// type: 'jsx',
// value: '<img src={require("!/workspaces/blog.johnnyreilly.com/blog-website/node_modules/url-loader/dist/cjs.js?limit=10000&name=assets/images/[name]-[hash].[ext]&fallback=/workspaces/blog.johnnyreilly.com/blog-website/node_modules/file-loader/dist/cjs.js!./bower-with-the-long-paths.png").default} width="640" height="497" />'
// }
// if (!vfile.history[0].includes('blog/2023-01-15')) return;
const key = `jsx|${vfile.history[0]}`;
const imageAlreadyProcessed = files.get(key);
const fetchpriorityThisImage =
!imageAlreadyProcessed || imageAlreadyProcessed === node['value'];
if (!imageAlreadyProcessed) {
files.set(key, node['value']);
}
if (fetchpriorityThisImage) {
node['value'] = node['value'].replace(
/<img /g,
'<img loading="eager" fetchpriority="high" ',
);
} else {
node['value'] = node['value'].replace(
/<img /g,
'<img loading="lazy" ',
);
}
}
});
};
}
module.exports = imageFetchPriorityRehypePlugin;
The new plugin looks like this:
// @ts-check
import { visit } from 'unist-util-visit';
/**
* Create a rehype plugin that will make the first image eager loaded with fetchpriority="high" and lazy load all other images
* @returns rehype plugin that will make the first image eager loaded with fetchpriority="high" and lazy load all other images
*/
export default function imageFetchPriorityRehypePlugin() {
/** @type {Map<string, string>} */ const files = new Map();
/** @type {import('unified').Transformer} */
return (tree, vfile) => {
visit(tree, ['mdxJsxTextElement'], (node) => {
if (node.type === 'mdxJsxTextElement' && node['name'] === 'img') {
// handles nodes like this:
// {
// type: 'mdxJsxTextElement',
// name: 'img',
// attributes: [
// {
// type: 'mdxJsxAttribute',
// name: 'alt',
// value: 'title image reading "Azure Container Apps, Bicep, managed certificates and custom domains" with the Azure Container App logos'
// },
// {
// type: 'mdxJsxAttribute',
// name: 'src',
// value: {
// type: 'mdxJsxAttributeValueExpression',
// value: 'require("!/home/john/code/github/blog.johnnyreilly.com/blog-website/node_modules/url-loader/dist/cjs.js?limit=10000&name=assets/images/[name]-[contenthash].[ext]&fallback=/home/john/code/github/blog.johnnyreilly.com/blog-website/node_modules/file-loader/dist/cjs.js!./screenshot-azure-portal-bring-your-own-certificates.webp").default',
// data: [Object]
// }
// },
// { type: 'mdxJsxAttribute', name: 'width', value: '800' },
// { type: 'mdxJsxAttribute', name: 'height', value: '450' }
// ],
// children: []
// }
const srcIndex = node['attributes'].findIndex(
(attr) => attr.name === 'src',
);
const requireString = node['attributes'][srcIndex].value.value;
const key = `jsx|${vfile.history[0]}`;
const imageAlreadyProcessed = files.get(key);
const fetchpriorityThisImage =
!imageAlreadyProcessed || imageAlreadyProcessed === requireString;
if (!imageAlreadyProcessed) {
files.set(key, requireString);
}
// expect to be -1
const loadingIndex = node['attributes'].findIndex(
(attr) => attr.name === 'loading',
);
if (fetchpriorityThisImage) {
// expect to be -1
const fetchpriorityIndex = node['attributes'].findIndex(
(attr) => attr.name === 'fetchpriority',
);
if (loadingIndex > -1) {
node['attributes'][loadingIndex].value = 'eager';
} else {
node['attributes'].push({
type: 'mdxJsxAttribute',
name: 'loading',
value: 'eager',
});
}
if (fetchpriorityIndex > -1) {
node['attributes'][fetchpriorityIndex].value = 'high';
} else {
node['attributes'].push({
type: 'mdxJsxAttribute',
name: 'fetchpriority',
value: 'high',
});
}
} else {
if (loadingIndex > -1) {
node['attributes'][loadingIndex].value = 'lazy';
} else {
node['attributes'].push({
type: 'mdxJsxAttribute',
name: 'loading',
value: 'lazy',
});
}
}
}
});
};
}
What's different? Well, a number of things; let's go through them.
You'll note the old plugin has the name image-fetch-priority-rehype-plugin.js
and the new plugin has the name image-fetch-priority-rehype-plugin.mjs
. This is because the new plugin is an ES Module and the old plugin is CommonJS.
Further to that, the old plugin used module.exports = imageFetchPriorityRehypePlugin
to expose functionality and the new plugin uses export default imageFetchPriorityRehypePlugin
.
The abstract syntax tree (AST) is different. MDX 1 and MDX 3 make different ASTs and we must migrate to the new one. Interestingly, it seems to be slightly simpler in some ways. MDX 1 surfaced both element
/ img
nodes and jsx
nodes. By contrast, MDX 3 appears to surface just mdxJsxTextElement
which are similar to MDX 1's jsx
nodes, but come with their own AST representation of expression based attributes in the data
property.
The logic of the new plugin is similar to the old plugin, but the code is different to cater for the different AST.
And that's it - we have a new fetchpriority
plugin that works with Docusaurus 3 and MDX 3!
cloudinary
pluginFirstly, let's remind ourselves what the cloudinary
plugin does. It takes an image URL and transforms it into a Cloudinary URL. So like this:
-https://my.website.com/cat.gif
+https://res.cloudinary.com/demo/image/fetch/https://my.website.com/cat.gif
And at runtime, Cloudinary's Fetch mechanism will handle transforming the image into a format that is optimised for the browser that is requesting it.
It turns out that the fetchpriority
plugin is a much more straightforward migration than the cloudinary
plugin. And the reason for that is related to the aforementioned AST changes. Let's start with the old plugin:
//@ts-check
const visit = require('unist-util-visit');
/**
* Create a rehype plugin that will replace image URLs with Cloudinary URLs
* @param {*} options cloudName your Cloudinary’s cloud name eg demo, baseUrl the base URL of your website eg https://johnnyreilly.com - should not include a trailing slash, will likely be the same as the config.url in your docusaurus.config.js
* @returns rehype plugin that will replace image URLs with Cloudinary URLs
*/
function imageCloudinaryRehypePlugin(
/** @type {{ cloudName: string; baseUrl: string }} */ options,
) {
const { cloudName, baseUrl } = options;
const srcRegex = / src={(.*)}/;
/** @type {import('unified').Plugin<[], import('hast').Root>} */
return (tree) => {
visit(tree, ['element', 'jsx'], (node) => {
if (node.type === 'element' && node['tagName'] === 'img') {
// handles nodes like this:
// {
// type: 'element',
// tagName: 'img',
// properties: {
// src: 'https://some.website.com/cat.gif',
// alt: null
// },
// ...
// }
const url = node['properties'].src;
node[
'properties'
].src = `https://res.cloudinary.com/${cloudName}/image/fetch/${url}`;
} else if (node.type === 'jsx' && node['value']?.includes('<img ')) {
// handles nodes like this:
// {
// type: 'jsx',
// value: '<img src={require("!/workspaces/blog.johnnyreilly.com/blog-website/node_modules/url-loader/dist/cjs.js?limit=10000&name=assets/images/[name]-[hash].[ext]&fallback=/workspaces/blog.johnnyreilly.com/blog-website/node_modules/file-loader/dist/cjs.js!./bower-with-the-long-paths.png").default} width="640" height="497" />'
// }
const match = node['value'].match(srcRegex);
if (match) {
const urlOrRequire = match[1];
node['value'] = node['value'].replace(
srcRegex,
` src={${`\`https://res.cloudinary.com/${cloudName}/image/fetch/${baseUrl}\$\{${urlOrRequire}\}\``}}`,
);
}
}
});
};
}
module.exports = imageCloudinaryRehypePlugin;
The old plugin had two kinds of nodes it had to deal with, element
and jsx
. The new plugin will have to deal with just one kind of node, mdxJsxTextElement
. (Just the same as with the fetchpriority
plugin.)
Now you may have noticed that the JSX node in the old plugin has a slightly more complex src
attribute:
<img src={require("!/workspaces/blog.johnnyreilly.com/blog-website/node_modules/url-loader/dist/cjs.js?limit=10000&name=assets/images/[name]-[hash].[ext]&fallback=/workspaces/blog.johnnyreilly.com/blog-website/node_modules/file-loader/dist/cjs.js!./bower-with-the-long-paths.png").default} width="640" height="497" />`
That src
attribute is a JavaScript expression. It's not a string. It's a JavaScript expression that will be evaluated later by webpack, and will return the path to the image in the final (webpack-based) Docusaurus build.
So transformation into a Cloudinary URL for JSX nodes is a little tougher. In the MDX 1 plugin, we needed to wrap the require
expression in backticks and prefix it with https://res.cloudinary.com/${cloudName}/image/fetch/${baseUrl}
where ${baseUrl}
is the base URL of our website. We also need to prefix the expression with a $
to indicate that it's a JavaScript expression. Tough to read but it works.
Rereading that paragraph, I realise it's hard to understand. Perhaps easier to see it in action. Here's what we want our plugin to do to the JSX node above:
-require("!/home/john/code/github/blog.johnnyreilly.com/blog-website/node_modules/url-loader/dist/cjs.js?limit=10000&name=assets/images/[name]-[contenthash].[ext]&fallback=/home/john/code/github/blog.johnnyreilly.com/blog-website/node_modules/file-loader/dist/cjs.js!./screenshot-azure-portal-bring-your-own-certificates.webp").default
+`https://res.cloudinary.com/demo/image/fetch/f_auto,q_auto,w_auto,dpr_auto/https://johnnyreilly.com${require("!/home/john/code/github/blog.johnnyreilly.com/blog-website/node_modules/url-loader/dist/cjs.js?limit=10000&name=assets/images/[name]-[contenthash].[ext]&fallback=/home/john/code/github/blog.johnnyreilly.com/blog-website/node_modules/file-loader/dist/cjs.js!./screenshot-azure-portal-bring-your-own-certificates.webp").default}`
It turns out it's even tougher doing this with MDX 3 as compared to MDX 1. This is because MDX 3's AST includes all kinds of metadata around the mdxJsxAttributeValueExpression
:
{
type: 'mdxJsxAttribute',
name: 'src',
value: {
type: 'mdxJsxAttributeValueExpression',
value: 'require("!/home/john/code/github/blog.johnnyreilly.com/blog-website/node_modules/url-loader/dist/cjs.js?limit=10000&name=assets/images/[name]-[contenthash].[ext]&fallback=/home/john/code/github/blog.johnnyreilly.com/blog-website/node_modules/file-loader/dist/cjs.js!./screenshot-azure-portal-bring-your-own-certificates.webp").default',
data: [Object] // <--- There's a lot of metadata in here!
}
},
The data
object above is a full on AST representation of the require
expression. And to make a plugin that works with MDX 3, we need to use that AST representation to build up the new src
attribute. This involves some string manipulation and some AST traversal. It's not pretty but it works.
Here's the new plugin:
//@ts-check
import { visit } from 'unist-util-visit';
import * as acorn from 'acorn';
import { mdxJsx } from 'micromark-extension-mdx-jsx';
import { fromMarkdown } from 'mdast-util-from-markdown';
import { mdxJsxFromMarkdown, mdxJsxToMarkdown } from 'mdast-util-mdx-jsx';
import { toMarkdown } from 'mdast-util-to-markdown';
/**
* @typedef {object} Params a label and an href
* @property {string} cloudName your Cloudinary’s cloud name eg demo
* @property {string} baseUrl the base URL of your website eg https://johnnyreilly.com - should not include a trailing slash, will likely be the same as the config.url in your docusaurus.config.js
*/
/**
* Create a rehype plugin that will replace image URLs with Cloudinary URLs
* @param {Params} params
* @returns rehype plugin that will replace image URLs with Cloudinary URLs
*/
export default function imageCloudinaryRehypePlugin({ cloudName, baseUrl }) {
const imageCloudinaryRehypeVisitor = imageCloudinaryRehypeVisitor({
cloudName,
baseUrl,
});
return (tree) => {
// eslint-disable-next-line @typescript-eslint/no-unsafe-call
visit(tree, ['mdxJsxTextElement'], imageCloudinaryRehypeVisitor);
};
}
/**
* Create a rehype visitor that will replace image URLs with Cloudinary URLs - exposed for testing purposes
* @param {Params} params
* @returns rehype plugin that will replace image URLs with Cloudinary URLs
*/
export function imageCloudinaryRehypeVisitor({ cloudName, baseUrl }) {
const srcRegex = / src=\{(.*)\}/;
return function imageCloudinaryRehypeVisitor(node) {
const imgWithAttributes = node;
if (
imgWithAttributes.type === 'mdxJsxTextElement' &&
imgWithAttributes.name === 'img'
) {
// handles nodes like this:
// {
// type: 'mdxJsxTextElement',
// name: 'img',
// attributes: [
// {
// type: 'mdxJsxAttribute',
// name: 'alt',
// value: 'title image reading "Azure Container Apps, Bicep, managed certificates and custom domains" with the Azure Container App logos'
// },
// {
// type: 'mdxJsxAttribute',
// name: 'src',
// value: {
// type: 'mdxJsxAttributeValueExpression',
// value: 'require("!/home/john/code/github/blog.johnnyreilly.com/blog-website/node_modules/url-loader/dist/cjs.js?limit=10000&name=assets/images/[name]-[contenthash].[ext]&fallback=/home/john/code/github/blog.johnnyreilly.com/blog-website/node_modules/file-loader/dist/cjs.js!./screenshot-azure-portal-bring-your-own-certificates.webp").default',
// data: [Object]
// }
// },
// { type: 'mdxJsxAttribute', name: 'width', value: '800' },
// { type: 'mdxJsxAttribute', name: 'height', value: '450' }
// ],
// children: []
// }
const srcIndex = imgWithAttributes.attributes.findIndex(
(attr) => attr.name === 'src',
);
const requireAttribute = imgWithAttributes.attributes[srcIndex].value;
if (typeof requireAttribute !== 'string') {
const asMarkdown = toMarkdown(imgWithAttributes, {
extensions: [mdxJsxToMarkdown()],
});
// <img
// alt="screenshot of typescript playground saying 'ComponentThatReturnsANumber' cannot be used as a JSX component. Its return type 'number' is not a valid JSX element.(2786)"
// src={require("!/home/john/code/github/blog.johnnyreilly.com/blog-website/node_modules/url-loader/dist/cjs.js?limit=10000&name=assets/images/[name]-[contenthash].[ext]&fallback=/home/john/code/github/blog.johnnyreilly.com/blog-website/node_modules/file-loader/dist/cjs.js!./screenshot-typescript-playground.png").default}
// width="690" height="298" />
const match = asMarkdown.match(srcRegex);
if (match) {
const urlOrRequire = match[1];
const cloudinaryRequireString = `\`https://res.cloudinary.com/${cloudName}/image/fetch/f_auto,q_auto,w_auto,dpr_auto/${baseUrl}\$\{${urlOrRequire}\}\``;
const newMarkdown = asMarkdown.replace(
srcRegex,
` src={${cloudinaryRequireString}}`,
);
// <img
// alt="screenshot of typescript playground saying 'ComponentThatReturnsANumber' cannot be used as a JSX component. Its return type 'number' is not a valid JSX element.(2786)"
// src={`https://res.cloudinary.com/priou/image/fetch/f_auto,q_auto,w_auto,dpr_auto/https://johnnyreilly.com${require("!/home/john/code/github/blog.johnnyreilly.com/blog-website/node_modules/url-loader/dist/cjs.js?limit=10000&name=assets/images/[name]-[contenthash].[ext]&fallback=/home/john/code/github/blog.johnnyreilly.com/blog-website/node_modules/file-loader/dist/cjs.js!./screenshot-typescript-playground.png").default}`}
// width="690" height="298" />
const tree = fromMarkdown(newMarkdown, {
extensions: [mdxJsx({ acorn, addResult: true })],
mdastExtensions: [mdxJsxFromMarkdown()],
});
const newSrcAttributeIndex = tree.children[0]['attributes'].findIndex(
(attr) => attr.name === 'src',
);
if (newSrcAttributeIndex !== -1) {
imgWithAttributes.attributes[srcIndex] =
tree.children[0]['attributes'][newSrcAttributeIndex];
}
}
}
}
};
}
Much is happening here. Let's go through it.
This amounts to the same changes as the fetchpriority
plugin. The old plugin has the name cloudinary-rehype-plugin.js
and the new plugin has the name cloudinary-rehype-plugin.mjs
. This is because the new plugin is an ES Module and the old plugin is CommonJS. Related to this, the old plugin used module.exports = imageCloudinaryRehypePlugin
to expose functionality and the new plugin uses export default imageCloudinaryRehypePlugin
.
We're dealing with a different AST and just need to tackle the mdxJsxTextElement
which are similar to MDX 1's jsx
nodes, but come with their own AST representation of expression based attributes in the data
property.
The hardest part of this (and it is hard / confusing) is dealing with the require
expression in the src
attribute. What we do is:
mdxJsxTextElement
to back to markdown - this is the full img
element in its AST formrequire
expression in the src
attribute of the markdownrequire
expression to a Cloudinary URL using the same mechanism as with the MDX 1 pluginmdxJsxTextElement
using a technique adapted from mdast-util-mdx-jsx
src
attribute with the new src
attribute including the updated require
expression AST in the mdxJsxAttributeValueExpression
attributes data
property.If you were to compare the MDX 1 plugin with the MDX 3 plugin, 2 and 3 from the above points are the same. Points 1, 4 and 5 are new.
With this in place we have a new plugin that works with Docusaurus 3 and MDX 3!
rehype-cloudinary-docusaurus@2
You may recall that I published an npm package named rehype-cloudinary-docusaurus
which packages up the plugin to make it easy for people to use. I've updated that package to use the new plugin and it is available now. You can see the pull request here. The new version is 3.0.0
.
<meta
name="description"
content="Use the TypeScript Azure Open AI SDK to generate article metadata."
/>
Descriptions are important for search engine optimisation (SEO) and for accessibility. You can read up more on the topic here. I wanted to have descriptions for all my blog posts. But writing around 230 descriptions for my existing posts was not something I wanted to do manually. I wanted to automate it.
I've been using Azure Open AI for a while now, and I've been using the TypeScript SDK in the @azure/openai
package to interact with it. What I wanted to do, was to use the SDK to generate descriptions for my blog posts based on the content. Since my blog is powered by Docusaurus and each post is a Markdown file, I had easy access to a individual files that could be summarised.
The plan was, to build a script to do the following:
I wanted to use Bun for this as it supports TypeScript by default. Using Node.js would equally be possible; but it wouldn't have been so easy to use TypeScript.
I started off by creating a new Bun project:
mkdir open-ai-description
cd open-ai-description
bun init
Then adding the various packages we needed, including the Azure Open AI one:
bun add @azure/openai @azure/identity
bun add --dev bun-types typescript
I then created an index.ts
file and added the following code:
import fs from 'fs';
import path from 'path';
import { produceSummary } from './summarizer';
const docusaurusDirectory = '../blog-website';
async function getBlogDirsOrderedDescending() {
const rootBlogPath = path.resolve(docusaurusDirectory, 'blog');
const blogDirs = (await fs.promises.readdir(rootBlogPath))
.filter((file) => fs.statSync(path.join(rootBlogPath, file)).isDirectory())
.map((file) => path.join(rootBlogPath, file));
blogDirs.sort().reverse();
return blogDirs;
}
interface BlogPost {
path: string;
content: string;
}
interface BlogPostWithDescription extends BlogPost {
description: string;
}
async function generatePostsWithDescription() {
const blogDirs = await getBlogDirsOrderedDescending();
const postsWithoutDescription: BlogPost[] = [];
for (const blogDir of blogDirs) {
console.log(`Processing ${blogDir}`);
const indexMdPath = path.join(blogDir, 'index.md');
const blogPostContent = await fs.promises.readFile(indexMdPath, 'utf-8');
const frontMatter = blogPostContent.split('---')[1];
const hasDescription = frontMatter.includes('\ndescription: ');
if (!hasDescription) {
postsWithoutDescription.push({
path: indexMdPath,
content: blogPostContent,
});
}
}
console.log(
`Found ${postsWithoutDescription.length} posts without description`,
);
const postsWithDescription: BlogPostWithDescription[] = [];
for (const post of postsWithoutDescription) {
const [, frontmatter, article] = post.content.split('---');
console.log(
`** Generating description for ${post.path
.replace('/index.md', '')
.split('/')
.pop()}`,
);
const description = await produceSummary(article);
if (description) {
postsWithDescription.push({ ...post, description });
console.log(`** description: ${description}`);
await fs.promises.writeFile(
post.path,
`---${frontmatter}description: '${description.replaceAll("'", '')}'
---${article}`,
);
} else {
console.log(`** no description generated`);
}
}
return postsWithDescription;
}
async function main() {
const startedAt = new Date();
const postsWithDescription: BlogPostWithDescription[] =
await generatePostsWithDescription();
const finishedAt = new Date();
const duration = (finishedAt.getTime() - startedAt.getTime()) / 1000;
console.log(
`${postsWithDescription.length} summaries generated in ${duration} seconds`,
);
}
await main();
The code above does the following:
blog
directory with an index.md
underneath so it's pretty easyproduceSummary
function - more on that in a momentSo far, so text wrangling. Let's look at the produceSummary
function in the summarizer.ts
file:
import { OpenAIClient } from '@azure/openai';
import { AzureCliCredential } from '@azure/identity';
// Make sure you have az login'd and have the correct subscription selected
// debug with:
// bun --inspect-brk open-ai-description/index.ts
// or:
// cd open-ai-description
// bun --inspect-brk index.ts
interface OpenAiClientAndDeploymentName {
openAIClient: OpenAIClient;
deploymentName: string;
}
let openAiClientAndDeploymentName: OpenAiClientAndDeploymentName | undefined;
function getClientAndDeploymentName(): OpenAiClientAndDeploymentName {
if (openAiClientAndDeploymentName) {
return openAiClientAndDeploymentName;
}
// You will need to set these environment variables or edit the following values
const endpoint = process.env['ENDPOINT'];
if (!endpoint) {
throw new Error(
'Missing ENDPOINT environment variable with a value like https://<resource name>.openai.azure.com',
);
}
// This will use the Azure CLI's currently logged in user;
// make sure you've done `az login` and have the correct subscription selected
const credential = new AzureCliCredential();
const openAIClient = new OpenAIClient(endpoint, credential);
const deploymentName = 'OpenAi-gpt-35-turbo';
openAiClientAndDeploymentName = { openAIClient, deploymentName };
return openAiClientAndDeploymentName;
}
export async function produceSummary(article: string): Promise<string> {
const { openAIClient, deploymentName } = getClientAndDeploymentName();
const minChars = 120;
const maxChars = 156;
const messages = [
{
role: 'system',
content: `You are a summarizer. You will be given the text of an article and will produce a summary / meta description which summarizes the article. The summary / meta descriptions you produce must be between ${minChars} and ${maxChars} characters long. If they are longer or shorter than that they cannot be used. Avoid using the \`'\` character as it is not supported by the blog website - you may use the \`’\` character instead. Do not use the expression "the author" or "the writer" in your summary.`,
},
{
role: 'user',
content: `Here is an article to summarize:
${article}`,
},
];
let attempts = 0;
const maxAttempts = 10;
let summary = '';
while (attempts++ < maxAttempts) {
console.log(`Attempt ${attempts} of ${maxAttempts}`);
// This will use the Azure CLI's currently logged in user;
// make sure you've done `az login` and have the correct subscription selected
const result = await openAIClient.getChatCompletions(
deploymentName,
messages,
{
temperature: 0.9,
},
);
for (const choice of result.choices) {
summary = choice.message?.content || '';
if (summary.length >= minChars && summary.length <= maxChars) {
return summary;
}
console.log(`Summary was ${summary.length} characters long; no good`);
}
messages.push(
{
role: 'assistant',
content: summary,
},
{
role: 'user',
content:
summary.length < minChars
? `Too short; try again please - we require a summary that is between ${minChars} and ${maxChars} characters long.`
: `Too long; try again please - we require a summary that is between ${minChars} and ${maxChars} characters long.`,
},
);
console.log(messages);
await sleep(500);
}
console.log(`Failed to produce a summary in ${maxAttempts} attempts`);
return '';
}
function sleep(ms: number) {
return new Promise((resolve) => setTimeout(resolve, ms));
}
There's a lot going on here, so let's break it down.
Oftentimes the fiddliest part of using Azure Open AI is authentication. In this case, I'm using the Azure CLI credential. So to run this you need to authenticate with the Azure CLI with az login
. (Remember to make sure you have the correct subscription selected. You can check this by running az account show
and checking that the isDefault
property is set to true
.)
Once you're logged in, this code will use the currently logged in user to authenticate with Azure Open AI.
const credential = new AzureCliCredential();
const openAIClient = new OpenAIClient(endpoint, credential);
const deploymentName = 'OpenAi-gpt-35-turbo';
You need to set the endpoint
variable to the endpoint of your Azure Open AI resource. You can find this in the Azure Portal by going to your Azure Open AI resource. Look for something like https://<resource name>.openai.azure.com
. You'll also need to get the name of your deployment. In my case this is OpenAi-gpt-35-turbo
.
Once you've authenticated and got the client you can start to summarise. The first thing to do is provide a system message to prime the model with context on what we're trying to do. As part of writing a good description, there's a sweet spot to hit in terms of length; too short and it's not useful, too long and it gets truncated. So we're going to aim for between 120 and 156 characters. We're also going to encourage the AI to avoid certain wording constructs and also avoid using the '
character as it upsets the front matter.
Once primed, we hand over the blog content to the AI and ask it to produce a summary.
const minChars = 120;
const maxChars = 156;
const messages = [
{
role: 'system',
content: `You are a summarizer. You will be given the text of an article and will produce a summary / meta description which summarizes the article. The summary / meta descriptions you produce must be between ${minChars} and ${maxChars} characters long. If they are longer or shorter than that they cannot be used. Avoid using the \`'\` character as it is not supported by the blog website - you may use the \`’\` character instead. Do not use the expression "the author" or "the writer" in your summary.`,
},
{
role: 'user',
content: `Here is an article to summarize:
${article}`,
},
];
The way we interact with the Azure Open AI is with the getChatCompletions
method, which is effectively a strongly typed wrapper for the chat-completions endpoint in Azure.
let attempts = 0;
const maxAttempts = 10;
let summary = '';
while (attempts++ < maxAttempts) {
console.log(`Attempt ${attempts} of ${maxAttempts}`);
// This will use the Azure CLI's currently logged in user;
// make sure you've done `az login` and have the correct subscription selected
const result = await openAIClient.getChatCompletions(
deploymentName,
messages,
{
temperature: 0.9, // the closer to 1, the more creative the AI will be
},
);
for (const choice of result.choices) {
summary = choice.message?.content || '';
if (summary.length >= minChars && summary.length <= maxChars) {
return summary;
}
console.log(`Summary was ${summary.length} characters long; no good`);
}
messages.push(
{
role: 'assistant',
content: summary,
},
{
role: 'user',
content:
summary.length < minChars
? `Too short; try again please - we require a summary that is between ${minChars} and ${maxChars} characters long.`
: `Too long; try again please - we require a summary that is between ${minChars} and ${maxChars} characters long.`,
},
);
console.log(messages);
}
console.log(`Failed to produce a summary in ${maxAttempts} attempts`);
return '';
What's quite interesting, is that you really can't rely on the AI do what you ask it to do. It may create a description of an appropriate length. It may not. So we need to check what it gives us, and if it doesn't satisfy our needs, then we ask it to try again. We'll give it a maximum of 10 attempts per post, as surprisingly, every now and then it struggles to meet the brief and infinite loops are to be avoided.
When the script ran (after I'd az login
-ed) it produced descriptions for all my blog posts. I reviewed each summary and tweaked them where necessary. If I really didn't like a description I'd delete and run the script again. In the end I had descriptions for all my blog posts that I was pretty happy with. If you take a look at this giant PR you can see them all landing.
Hopefully this post provides a useful example of how to use the TypeScript Azure Open AI SDK to generate article metadata. You can see the raw code here.
]]>I had the good fortune to be involved in the making of the documentary. In part this was thanks to my work on Definitely Typed. Another reason was my work on recording the history of DefinitelyTyped.
You can see it on YouTube here or hit play on the embedded video above. Thanks to the Keyboard Stories team for making this happen!
]]>One of the components that I use frequently is the tabs component. However, I've found that it can be a little tricky to use in a "text-first" way, that also remains strongly typed. This post documents how to do just that!
What does the tabs component look like? Well, here's a screenshot of it in action:
It's very useful if you'd like your users to be able to switch between different views easily. The official MUI documentation provides an example of how to use the tabs component:
import * as React from 'react';
import Tabs from '@mui/material/Tabs';
import Tab from '@mui/material/Tab';
import Typography from '@mui/material/Typography';
import Box from '@mui/material/Box';
interface TabPanelProps {
children?: React.ReactNode;
index: number;
value: number;
}
function CustomTabPanel(props: TabPanelProps) {
const { children, value, index, ...other } = props;
return (
<div
role="tabpanel"
hidden={value !== index}
id={`simple-tabpanel-${index}`}
aria-labelledby={`simple-tab-${index}`}
{...other}
>
{value === index && (
<Box sx={{ p: 3 }}>
<Typography>{children}</Typography>
</Box>
)}
</div>
);
}
function a11yProps(index: number) {
return {
id: `simple-tab-${index}`,
'aria-controls': `simple-tabpanel-${index}`,
};
}
export default function BasicTabs() {
const [value, setValue] = React.useState(0);
const handleChange = (event: React.SyntheticEvent, newValue: number) => {
setValue(newValue);
};
return (
<Box sx={{ width: '100%' }}>
<Box sx={{ borderBottom: 1, borderColor: 'divider' }}>
<Tabs
value={value}
onChange={handleChange}
aria-label="basic tabs example"
>
<Tab label="Item One" {...a11yProps(0)} />
<Tab label="Item Two" {...a11yProps(1)} />
<Tab label="Item Three" {...a11yProps(2)} />
</Tabs>
</Box>
<CustomTabPanel value={value} index={0}>
Item One
</CustomTabPanel>
<CustomTabPanel value={value} index={1}>
Item Two
</CustomTabPanel>
<CustomTabPanel value={value} index={2}>
Item Three
</CustomTabPanel>
</Box>
);
}
This example is great, but (personally) I find it a little hard to read. There's a direct relationship between the tabs and the tab panels, but it's not immediately obvious. When you see the 0
passed to a11yProps
and the 0
passed to CustomTabPanel
, it's not clear that they're related. And if the a11yProps
function call was not present, it would be even less clear.
I'd like to see the tabs and tab panels presented together in a more text-first way, that makes the relationship between tab and tab panel more apparent.
The code I'd like to see would look something like this:
<Box sx={{ width: '100%' }}>
<Box sx={{ borderBottom: 1, borderColor: 'divider' }}>
<Tabs
value={selectedTab}
onChange={handleChange}
aria-label="basic tabs example"
>
<Tab {...customTabProps('Item One')} />
<Tab {...customTabProps('Item Two')} />
<Tab {...customTabProps('Item Three')} />
</Tabs>
</Box>
<CustomTabPanel selectedTab={selectedTab} tab="Item One">
Item One
</CustomTabPanel>
<CustomTabPanel selectedTab={selectedTab} tab="Item Two">
Item Two
</CustomTabPanel>
<CustomTabPanel selectedTab={selectedTab} tab="Item Three">
Item Three
</CustomTabPanel>
</Box>
In this code snippet, the tabs and tab panels have more of a linkage, in a text-first way. It's hopefully clear that the "Item One" Tab
and the "Item One" CustomTabPanel
are related.
In the code above, the customTabProps
function is used to generate the props for the tabs (it's an evolution of the a11yProps
function that handles accessibility props as well as all others). Meanwhile, the CustomTabPanel
component is used to render the tab panels. The selectedTab
state is used to keep track of the selected tab.
How does this work? And is it strongly typed? Let's find out.
Yes, it's strongly typed! We achieve this by defining a mapping of tab text to tab index:
const tabs = {
'Item One': 0,
'Item Two': 1,
'Item Three': 2,
} as const;
type TabText = keyof typeof tabs;
type TabIndex = (typeof tabs)[TabText];
So "Item One" is 0
, "Item Two" is 1
, and "Item Three" is 2
.
We then do some TypeScript magic to strongly type this. We use as const
to tell TypeScript this is an immutable object. With that done we can then extract the keys and values from the object and use them to create the derived types TabText
and TabIndex
.
TabText
is the keys of the tabs
object and TabIndex
is the values. So TabText
is "Item One" | "Item Two" | "Item Three"
and TabIndex
is 0 | 1 | 2
. If we should subsequently amend the tabs
object in our code, TypeScript will ensure that the TabText
and TabIndex
types are updated accordingly.
We can then use these types in our components:
function customTabProps(tab: TabText) {
const index = tabs[tab];
return {
id: `simple-tab-tab-${index}`,
'aria-controls': `simple-tab-tabpanel-${index}`,
label: tab,
};
}
interface CustomTabPanelProps {
children?: React.ReactNode;
tab: TabText;
selectedTab: TabIndex;
}
function CustomTabPanel(props: CustomTabPanelProps) {
const { children, selectedTab, tab, ...other } = props;
const index = tabs[tab];
return (
<div
role="tabpanel"
hidden={selectedTab !== index}
id={`simple-tab-tabpanel-${index}`}
aria-labelledby={`simple-tab-tab-${index}`}
{...other}
>
{selectedTab === index && (
<Box sx={{ p: 3 }}>
<Typography>{children}</Typography>
</Box>
)}
</div>
);
}
Then our final example code looks like this:
export default function BasicTabs() {
const [selectedTab, setSelectedTab] = React.useState<TabIndex>(
tabs['Item One'],
);
const handleChange = (event: React.SyntheticEvent, newValue: TabIndex) => {
setSelectedTab(newValue);
};
return (
<Box sx={{ width: '100%' }}>
<Box sx={{ borderBottom: 1, borderColor: 'divider' }}>
<Tabs
value={selectedTab}
onChange={handleChange}
aria-label="basic tabs example"
>
<Tab {...customTabProps('Item One')} />
<Tab {...customTabProps('Item Two')} />
<Tab {...customTabProps('Item Three')} />
</Tabs>
</Box>
<CustomTabPanel selectedTab={selectedTab} tab="Item One">
Item One
</CustomTabPanel>
<CustomTabPanel selectedTab={selectedTab} tab="Item Two">
Item Two
</CustomTabPanel>
<CustomTabPanel selectedTab={selectedTab} tab="Item Three">
Item Three
</CustomTabPanel>
</Box>
);
}
Note how we use our TabIndex
types to strongly type the selectedTab
state and the handleChange
function. And also how the TabText
type is used to strongly type the tab
prop in the CustomTabPanel
component and the tab
argument in the customTabProps
function. With this in place, we cannot provide invalid tab text to the customTabProps
function or the CustomTabPanel
component. TypeScript would fight us every step of the way if we tried.
So now we have a strongly typed, text-first way to use the MUI tabs component. We've used TypeScript to ensure that our tabs and tab panels are related in a way that is clear and easy to understand. This approach makes our code more maintainable and easier to work with. The full code is below:
import * as React from 'react';
import Tabs from '@mui/material/Tabs';
import Tab from '@mui/material/Tab';
import Typography from '@mui/material/Typography';
import Box from '@mui/material/Box';
const tabs = {
'Item One': 0,
'Item Two': 1,
'Item Three': 2,
} as const;
type TabText = keyof typeof tabs;
type TabIndex = (typeof tabs)[TabText];
function customTabProps(tab: TabText) {
const index = tabs[tab];
return {
id: `simple-tab-tab-${index}`,
'aria-controls': `simple-tab-tabpanel-${index}`,
label: tab,
};
}
interface CustomTabPanelProps {
children?: React.ReactNode;
tab: TabText;
selectedTab: TabIndex;
}
function CustomTabPanel(props: CustomTabPanelProps) {
const { children, selectedTab, tab, ...other } = props;
const index = tabs[tab];
return (
<div
role="tabpanel"
hidden={selectedTab !== index}
id={`simple-tab-tabpanel-${index}`}
aria-labelledby={`simple-tab-tab-${index}`}
{...other}
>
{selectedTab === index && (
<Box sx={{ p: 3 }}>
<Typography>{children}</Typography>
</Box>
)}
</div>
);
}
export default function BasicTabs() {
const [selectedTab, setSelectedTab] = React.useState<TabIndex>(
tabs['Item One'],
);
const handleChange = (event: React.SyntheticEvent, newValue: TabIndex) => {
setSelectedTab(newValue);
};
return (
<Box sx={{ width: '100%' }}>
<Box sx={{ borderBottom: 1, borderColor: 'divider' }}>
<Tabs
value={selectedTab}
onChange={handleChange}
aria-label="basic tabs example"
>
<Tab {...customTabProps('Item One')} />
<Tab {...customTabProps('Item Two')} />
<Tab {...customTabProps('Item Three')} />
</Tabs>
</Box>
<CustomTabPanel selectedTab={selectedTab} tab="Item One">
Item One
</CustomTabPanel>
<CustomTabPanel selectedTab={selectedTab} tab="Item Two">
Item Two
</CustomTabPanel>
<CustomTabPanel selectedTab={selectedTab} tab="Item Three">
Item Three
</CustomTabPanel>
</Box>
);
}
It's certainly more complicated than the official example (and this may well be why the official example is the way it is), but it matches my preferences.
As an aside, I'd like the code even more if I had the following instead of using Tab
with customTabProps
:
<CustomTab tab="Item One" />
I avoided that in this post because it would have made the example more complicated. But I think it would be a nice improvement.
]]>To generate a Word document in .NET, the most straightforward way is to use the Open XML library. We can install the library using the following command:
dotnet add package DocumentFormat.OpenXml
With the Open XML library installed, we can create a new Word document in the context of an ASP.NET controller. The following code demonstrates how to do this:
using Microsoft.AspNetCore.Mvc;
using DocumentFormat.OpenXml;
using DocumentFormat.OpenXml.Packaging;
using DocumentFormat.OpenXml.Wordprocessing;
namespace MyApp.Controllers;
[ApiController]
public class WordDocumentController() : ControllerBase
{
[HttpGet("api/generate-word-document")]
public IActionResult GetWordDocument()
{
// Create a new Word document
using var stream = new MemoryStream();
using var document = WordprocessingDocument.Create(stream, WordprocessingDocumentType.Document);
var mainPart = document.AddMainDocumentPart();
mainPart.Document = new Document();
// Add content to the document
var body = mainPart.Document.AppendChild(new Body());
var paragraph = body.AppendChild(new Paragraph());
var run = paragraph.AppendChild(new Run());
run.AppendChild(new Text("Hello, World!"));
// Save the document to a memory stream
document.Save();
var byteArray = stream.ToArray();
// Return the document as a file
return File(byteArray, "application/vnd.openxmlformats-officedocument.wordprocessingml.document", "document.docx");
}
}
In this example, the GetWordDocument
method creates a new Word document and adds the text "Hello, World!" to it. If we navigate to the /api/generate-word-document
endpoint, we will receive a Word document with the text "Hello, World!" in it.
The document is then saved to a memory stream and returned as a file. The File
method is used to return the document as a file with the MIME type application/vnd.openxmlformats-officedocument.wordprocessingml.document
(which basically is the server saying "Hey! This is a Word document!").
Generating a Word document in an ASP.NET controller is quite simple to do using the Open XML library. We can create a new Word document, add content to it, and return it as a file using the File
method.
To learn more about how to add content to a Word document using the Open XML library, it's worth reading the Open XML SDK documentation.
I hope this post helps you to generate Word documents in your ASP.NET applications!
]]>If you're making use of Azure's Open AI resources for your AI needs, you'll be aware that there are limits known as "quotas" in place. If you're looking to control how many resources you're using, you'll want to be able to control the capacity of your deployments. This is possible with Bicep.
This post grew out of a GitHub issue around the topic where people were bumping on the message the capacity should be null for standard deployment
as they attempted to deploy. At the time that issue was raised, there was very little documentation on how to handle this. Since then, things have improved, but I thought it would be useful to have a post on the topic.
If you take a look at the Azure Open AI Studio you'll notice a "Quotas" section:
You'll see above that we've got two deployments of GPT-35-Turbo in our subscription. Both of these contribute towards an overall limit of 360K TPM. If we try and deploy resources and have an overall capacity total that exceeds that, our deployment will fail.
That being the case, we need to be able to control the capacity of our deployments. This is possible with Bicep.
Consider the following account-deployments.bicep
:
@description('Name of the Cognitive Services resource')
param cognitiveServicesName string
@description('Name of the deployment resource.')
param deploymentName string
@description('Deployment model format.')
param format string
@description('Deployment model name.')
param name string
@description('Deployment model version.')
param version string = '1'
@description('The name of RAI policy.')
param raiPolicyName string = 'Default'
@allowed([
'NoAutoUpgrade'
'OnceCurrentVersionExpired'
'OnceNewDefaultVersionAvailable'
])
@description('Deployment model version upgrade option. see https://learn.microsoft.com/en-us/azure/templates/microsoft.cognitiveservices/2023-05-01/accounts/deployments?pivots=deployment-language-bicep#deploymentproperties')
param versionUpgradeOption string = 'OnceNewDefaultVersionAvailable'
@description('''Deployments SKU see: https://learn.microsoft.com/en-us/azure/templates/microsoft.cognitiveservices/2023-05-01/accounts/deployments?pivots=deployment-language-bicep#sku
eg:
sku: {
name: 'Standard'
capacity: 10
}
''')
param sku object
// https://learn.microsoft.com/en-us/azure/templates/microsoft.cognitiveservices/2023-05-01/accounts?pivots=deployment-language-bicep
resource cog 'Microsoft.CognitiveServices/accounts@2023-05-01' existing = {
name: cognitiveServicesName
}
// https://learn.microsoft.com/en-us/azure/templates/microsoft.cognitiveservices/2023-05-01/accounts/deployments?pivots=deployment-language-bicep
resource deployment 'Microsoft.CognitiveServices/accounts/deployments@2023-05-01' = {
name: deploymentName
parent: cog
sku: sku
properties: {
model: {
format: format
name: name
version: version
}
raiPolicyName: raiPolicyName
versionUpgradeOption: versionUpgradeOption
}
}
output deploymentName string = deployment.name
output deploymentResourceId string = deployment.id
We can use this to deploy.... deployments (naming here is definitely confusing) to Azure like so:
var cognitiveServicesDeployments = [
{
name: 'OpenAi-gpt-35-turbo'
shortName: 'gpt35t'
model: {
format: 'OpenAI'
name: 'gpt-35-turbo'
version: '0301'
}
sku: {
name: 'Standard'
capacity: repositoryBranch == 'refs/heads/main' ? 100 : 10 // capacity in thousands of TPM
}
}
]
// Model Deployment - one at a time as parallel deployments are not supported
@batchSize(1)
module openAiAccountsDeployments35Turbo 'account-deployments.bicep' = [for deployment in cognitiveServicesDeployments: {
name: '${deployment.shortName}-cog-accounts-deployments'
params: {
cognitiveServicesName: openAi.outputs.cognitiveServicesName
deploymentName: deployment.name
format: deployment.model.format
name: deployment.model.name
version: deployment.model.version
sku: deployment.sku
}
}]
We're currently only deploying a single account deployment in our array, but we do it this way as it's not unusual to deploy multiple deployments together. Notice the sku
portion above:
sku: {
name: 'Standard'
capacity: repositoryBranch == 'refs/heads/main' ? 100 : 10
}
Here we provision a larger capacity
for our feature branch deployments than our main
branch deployments. This demonstrates our own usage, whereby we deploy a smaller capacity for our feature branches so that we can test things out, but then deploy a larger capacity for our main branch deployments.
Significantly, we're controlling the capacity of our deployments. The way in which you choose to decide on the capacity of your deployments is up to you, but the above demonstrates how you can do it with Bicep and stick within your quota limits.
]]>This post assumes we have a Vitest project set up and an Azure Pipeline in place. Let's get started.
First of all, lets get the tests running. We'll crack open our azure-pipelines.yml
file and, in the appropriate place add the following:
- bash: npm run test:ci
displayName: 'npm test'
workingDirectory: src/client-app
The above will, when run, trigger a npm run test:ci
in the src/client-app
folder of the project (it's here where the app lives). What does test:ci
do? Well, it's a script in the package.json
that looks like this:
"test": "vitest",
"test:ci": "vitest run --reporter=default --reporter=junit --outputFile=reports/junit.xml",
You'll note above we've got 2 scripts; test
and test:ci
. The former is the default script that Vitest will run when you run npm test
. The latter is the script that we'll use in our pipeline. The difference between the two is that the test:ci
script will:
The test results are written to reports/junit.xml
which is a path relative to the src/client-app
folder. Because you may test this locally, it's probably worth adding the reports
folder to your .gitignore
file to avoid it accidentally being committed.
Our tests are running, but we're not seeing the results in the Azure Pipelines UI. For that we need the PublishTestResults
task.
We need to add a new step to our azure-pipelines.yml
file after our npm run test:ci
step:
- task: PublishTestResults@2
displayName: 'supply npm test results to pipelines'
condition: succeededOrFailed() # because otherwise we won't know what tests failed
inputs:
testResultsFiles: 'src/client-app/reports/junit.xml'
This will read the test results from our src/client-app/reports/junit.xml
file and pump them into Pipelines. Do note that we're always running this step; so if the previous step failed (as it would in the case of a failing test) we still pump out the details of what that failure was.
And that's it! Azure Pipelines and Jest integrated.
The complete azure-pipelines.yml
additions look like this:
- bash: npm run test:ci
displayName: 'npm test'
workingDirectory: src/client-app
- task: PublishTestResults@2
displayName: 'supply npm test results to pipelines'
condition: succeededOrFailed() # because otherwise we won't know what tests failed
inputs:
testResultsFiles: 'src/client-app/reports/junit.xml'
Please note, there's nothing special about the reports/junit.xml
file. You can change the name of the file and/or the location of the file. Just make sure you update the testResultsFiles
value in the PublishTestResults
task to match.
This post will instead look at how we can use the "bring your own certificates" approach in Azure Container Apps using Bicep. Well, as much as that is possible; there appear to be limitations in what can be achieved with Bicep at the time of writing.
This post assumes you already have an Azure Container App in place and you want to bind a custom domain to it. If you don't have an Azure Container App in place, then you might want to take a look at this post on the topic.
So the first thing we need to do is make a certificate. This is pretty simple, and in my case amounts to the following command:
sudo openssl req -x509 -newkey rsa:4096 -sha256 -days 3650 -nodes \
-keyout poorclaresarundel.org.key -out poorclaresarundel.org.crt -subj "/CN=poorclaresarundel.org" \
-addext "subjectAltName=DNS:poorclaresarundel.org,DNS:www.poorclaresarundel.org,IP:8.8.8.8"
sudo chmod +r poorclaresarundel.org.key
cat poorclaresarundel.org.crt poorclaresarundel.org.key > poorclaresarundel.org.pem
This makes a certificate with a 10 year expiry date. You'll note the 8.8.8.8
listed as the IP address above. You should replace that with the static IP address of your Container Apps Environment. You can find that in the Azure Portal. It's listed on the "Overview" tab of your Azure Container App. It's the "Static IP" value:
You'll also note the poorclaresarundel.org
listed as the domain name above. You should replace that with your domain name. You'll need to do that in two places; in the openssl
command and in the subjectAltName
value. More on the significance of that particular domain name later.
Now, we'll crack open our Azure Container App's environment in the Azure Portal and navigate to the "Certificates" tab. Then we need to click on the "Bring your own certificates (.pfx)" tab, and we are presented with a screen like this:
Note the "Add certificate" button. Use that to upload your certificate - in my case that's the poorclaresarundel.org.pem
file. You'll need to provide the password for the certificate too. Oh and you'll be asked for a friendly name. We'll remember the friendly name you use - we'll need it later.
This is a real world example; my aunt's website. My aunt is Poor Clare nun and, for years, I've done an average job of maintaining her convent's website. If I was her, I'd be wishing her nephew was a designer rather than an engineer. Or maybe an engineer with more of a sense what looks good. But here we are - she's a nun and so consequently much too nice to say that. Anyway, I digress. The point is, I've got a certificate for her website and I'm going to use it here.
I haven't managed to work out how one would handle the certificate upload in Bicep (and I rather suspect it is not supported). However, I have worked out how to handle the certificate in the Azure Container App once it's been uploaded with regards to custom domains. Find the Microsoft.App/containerApps
resource in your Bicep and add a customDomains
property to it. It should look something like this:
resource environment 'Microsoft.App/managedEnvironments@2022-10-01' = {
name: environmentName
// ...
}
resource webServiceContainerApp 'Microsoft.App/containerApps@2022-10-01' = {
name: webServiceContainerAppName
tags: tags
location: location
properties: {
// ...
configuration: {
// ...
ingress: {
// ...
customDomains: [
{
name: 'www.poorclaresarundel.org'
// note the friendly name of "poorclaresarundel.org" forms the last segment of the id below
certificateId: '${environment.id}/certificates/poorclaresarundel.org'
bindingType: 'SniEnabled'
}
]
}
}
// ...
}
}
With the above post you should be able to deploy an Azure Container App with a custom domain and provide your own certificate, which is then referenced using Bicep. If you'd like to see the full Bicep file, then you can find it here.
I'm writing this post as, whilst it ends up being a relatively small amount of code and configuration required, if you don't know what that is, you can end up somewhat stuck. This should hopefully unstick you.
To query the Graph API, we'll need:
GroupMember.Read.All
and User.Read.All
API application permissionsOf the above, it's the API permissions that I want to draw your attention to. Ultimately, you'll need to get your Azure AD admin to grant consent to these permissions for your app registration so you have the necessary permissions to query the Graph API:
How did I know that I needed these permissions? Great question! I found the answer in the "directoryObject: getMemberGroups / group memberships for a user" documentation.
The Application User.Read.All
and GroupMember.Read.All
permissions are the least privileged permissions that allow you to query the Graph API for the information we need. If you're interested in the other permissions available, check out the Microsoft Graph permissions reference.
Now we've configured our app registration, we can query the Graph API. I'm going to show you how to do this with the C# SDK. If you're using a different language, you'll need to find the equivalent SDK for your language.
The first thing we need to do is install the SDK. I'm using the Microsoft.Graph package. At the time of writing, the latest version is 5.35.0.
Once you have added it, you'll have an entry in your .csproj
along these lines:
<PackageReference Include="Microsoft.Graph" Version="5.35.0" />
If you're interested in the underlying project, you can find it as msgraph-sdk-dotnet
on GitHub.
Next, we need to create a GraphServiceClient
. This is the object that we'll use to query the Graph API. For that we'll need a credential which we'll construct using the tenantId
, clientId
and clientSecret
that we got from our app registration:
using Azure.Identity;
using Microsoft.Graph;
// ...
// The client credentials flow requires that you request the
// /.default scope, and pre-configure your permissions on the
// app registration in Azure. An administrator must grant consent
// to those permissions beforehand.
var scopes = new[] { "https://graph.microsoft.com/.default" };
// using Azure.Identity;
var options = new ClientSecretCredentialOptions
{
AuthorityHost = AzureAuthorityHosts.AzurePublicCloud,
};
// https://learn.microsoft.com/en-us/graph/sdks/choose-authentication-providers?tabs=csharp#using-a-client-secret
// https://learn.microsoft.com/dotnet/api/azure.identity.clientsecretcredential
var clientSecretCredential = new ClientSecretCredential(
tenantId, clientId, clientSecret, options);
_graphClient = new GraphServiceClient(clientSecretCredential, scopes);
The above code is based on the "client credentials provider / using a client secret" documentation. It's possible that you might want to use a different authentication provider. If so, check out the "choose authentication providers" documentation. Because we're querying for application permissions, we need to use the client credentials provider. We could, if we wanted to, use the client certificate approach instead. I'm not going to cover that here.
Now we come to the fun part. We're going to query the Graph API for the groups that a user is a member of. We'll need to pass in the usernameOrId
of the user that we're interested in. I'm using the user's email address as the usernameOrId
in my application. You might want to use the user's id instead. It's up to you.
List<GroupIdDisplayName> groupIds = new();
Microsoft.Graph.Models.GroupCollectionResponse? response = await _graphClient.Users[usernameOrId].MemberOf.GraphGroup.GetAsync(requestConfiguration =>
{
requestConfiguration.QueryParameters.Select = ["id", "displayName"];
requestConfiguration.QueryParameters.Top = 100;
});
var pageIterator = PageIterator<
Microsoft.Graph.Models.Group,
Microsoft.Graph.Models.GroupCollectionResponse?
>.CreatePageIterator(_graphClient, response, (group) =>
{
groupIds.Add(new GroupIdDisplayName(Id: group.Id, DisplayName: group.DisplayName));
return true;
});
await pageIterator.IterateAsync();
Let's unpack the code above. It does the following:
GroupIdDisplayName
objects. This is the type that I'm using to store the group ids and display names. You can use whatever type you want.Users
collection on the GraphServiceClient
using the usernameOrId
that we passed in. From there we make use of MemberOf.GraphGroup
and call GetAsync
on that, passing in a request configuration method. This is where we specify the Select
and Top
query parameters. We're using Select
to specify only the properties that we want to return - by default lots more data would come back than we require. We're using Top
to specify the maximum number of results that we want to return. I'm using 100 as that's the maximum that the Graph API will allow. We'll need to use a PageIterator
to get all of the results. More on that in a moment.PageIterator
. We create one using the CreatePageIterator
method on the PageIterator
class. We pass in the GraphServiceClient
that we created earlier, the response
that we got from the GetAsync
call and a delegate that will be called for each result. In our delegate, we add the GroupIdDisplayName
to our list and return true
to indicate that we want to continue iterating. If we wanted to stop iterating, we'd return false
. Finally we call IterateAsync
on the PageIterator
to get all of the results.The PageIterator
is somewhat confusing, in my opinion. I'm used to paging through results, but this way of doing it did puzzle me. If you're interested in learning more about the PageIterator
, check out the "paging" documentation. I also found this documentation helpful.
Initially the PageIterator
was throwing an exception when I used it. I found the answer to that on StackOverflow. It turns out that the types you specify for the PageIterator
need to be correct for the request above; and if not you may be impaced by type mismatches at runtime. The entries that are returned from the API may be wider or simply different to what you require. I've the correct types in my code now - but I mention it just in case.
Ultimately I wrote a GroupService
that I could use to get the groups for a user. That service looks like this:
using Azure.Identity;
using Microsoft.Graph;
namespace ZebraGptContainerApp.Services.Implementations;
public record GroupIdDisplayName(
string? Id,
string? DisplayName
);
public interface IGroupService
{
Task<List<GroupIdDisplayName>> GetGroups(string usernameOrId);
}
public class GroupService : IGroupService
{
private readonly GraphServiceClient _graphClient;
private readonly ILogger<GroupService> _logger;
// new this up in program.cs
public GroupService(ILogger<GroupService> logger, string tenantId, string clientId, string clientSecret)
{
_logger = logger;
// The client credentials flow requires that you request the
// /.default scope, and pre-configure your permissions on the
// app registration in Azure. An administrator must grant consent
// to those permissions beforehand.
var scopes = new[] { "https://graph.microsoft.com/.default" };
// using Azure.Identity;
var options = new ClientSecretCredentialOptions
{
AuthorityHost = AzureAuthorityHosts.AzurePublicCloud,
};
// https://learn.microsoft.com/en-us/graph/sdks/choose-authentication-providers?tabs=csharp#using-a-client-secret
// https://learn.microsoft.com/dotnet/api/azure.identity.clientsecretcredential
var clientSecretCredential = new ClientSecretCredential(
tenantId, clientId, clientSecret, options);
_graphClient = new GraphServiceClient(clientSecretCredential, scopes);
}
public async Task<List<GroupIdDisplayName>> GetGroups(string usernameOrId)
{
// https://stackoverflow.com/questions/75860298/graphserviceclient-pageitarator-failes-with-the-parsable-does-not-contain-a-col
try
{
List<GroupIdDisplayName> groupIds = new();
Microsoft.Graph.Models.GroupCollectionResponse? response = await _graphClient.Users[usernameOrId].MemberOf.GraphGroup.GetAsync(requestConfiguration =>
{
requestConfiguration.QueryParameters.Select = ["id", "displayName"];
requestConfiguration.QueryParameters.Top = 100;
});
var pageIterator = PageIterator<
Microsoft.Graph.Models.Group,
Microsoft.Graph.Models.GroupCollectionResponse?
>.CreatePageIterator(_graphClient, response, (group) =>
{
groupIds.Add(new GroupIdDisplayName(Id: group.Id, DisplayName: group.DisplayName));
return true;
});
await pageIterator.IterateAsync();
return groupIds;
}
catch (Exception ex)
{
_logger.LogError(ex, "Problem getting groups with usernameOrId {usernameOrId}", usernameOrId);
throw new Exception($"Problem getting groups with usernameOrId {usernameOrId}", ex);
}
}
}
It is constructed in Program.cs
:
// ...
builder.Services.AddSingleton<IGroupService>(serviceProvider =>
new GroupService(
logger: serviceProvider.GetRequiredService<ILogger<GroupService>>(),
tenantId: builder.Configuration.GetValue<string>("TenantId")!,
clientId: builder.Configuration.GetValue<string>("ClientId")!,
clientSecret: builder.Configuration.GetValue<string>("ClientSecret")!
)
);
// ...
Then I can use it in my controller:
using Microsoft.AspNetCore.Mvc;
namespace MyApp.Controllers;
[ApiController]
public class GroupsController : ControllerBase
{
readonly IGroupService _groupService;
readonly ILogger<MeController> _log;
public GroupsController(IGroupService groupService, ILogger<MeController> log)
{
_groupService = groupService;
_log = log;
}
[HttpGet("api/groups")]
public async Task<IActionResult> GetGroups()
{
string? email = User?.Identity?.Name; // email in my context
try
{
if (string.IsNullOrEmpty(email))
return Unauthorized();
var groups = await _groupService.GetGroups(email);
return Ok(groups);
}
catch (Exception ex)
{
_log.LogError(ex, "Problem getting groups for {email}", email);
return BadRequest($"Problem getting groups for {email}");
}
}
}
When you browse to /api/groups
, you'll get a JSON response with the group ids and display names:
And if you're anything like me, you'll discover that you are in way more groups than you thought you were.
This post has demonstrated both the configuration and the code required to query the Graph API for the groups that a user is a member of. Hopefully it also provides something of a reference for acquiring different types of Graph API permissions and using the C# SDK to query the Graph API.
]]>In this evaluation guide, we’ll explore the features that make Bun an excellent choice for developing fast, performant, error-free frontend apps. By the end of this article, you’ll have a clear understanding of when and why you should use Bun in your projects.
Bun was created by Jarred Sumner with the intention that, if you currently use Node.js, you should easily be able to swap it out and replace it with Bun instead.
In other words, you should be able to quickly take advantage of Bun’s awesome features without having to deal with a steep learning curve. Generally, you can use the same frameworks, libraries, and conventions you’re used to, often without issue. Keep in mind there may be exceptions to this.
The Bun team released v1.0 on 8 September 2023. However, Bun was already widely known and used long before that. Jarred Sumner even appeared on PodRocket on 5 August 2022 to discuss Bun, more than a year before the release of v1.0.
Part of the reason for Bun’s rise in popularity even before its stable release is its speed and ease of use.
While Node.js and Deno both use V8 — the Chrome JavaScript engine — Bun uses Apple’s JavaScriptCore. This choice of JavaScript engine supports the Bun team’s self-described goal of “eliminating slowness and complexity without throwing away everything that's great about JavaScript.”
Additionally, Bun is written using Zig, a systems programming language that allows you to write extremely performant software — exactly what Bun intends to enable you to do. Interestingly, while Bun has now hit 1.0, Zig, which has been around longer, has not!
In contrast, Node.js is written in C++, and Deno is written in Rust. Zig makes it possible to write code that is both fast and safe, contributing to Bun’s speed and performance.
JavaScript is a great language, and the npm ecosystem is vast. These combined have powered large numbers of systems and development experiences for years. Given that this is the case, why would you want to use Bun in particular? What are the benefits that make it such a great choice?
If I were to explain why I've been using Bun, it's because it is like using Node.js, but simply a better version of it. Bun is the Node.js I want to use. Let's go through some of the reasons why:
Bun also provides a built-in bundler, although it’s still in beta and may not be widely used. However, it’s fast, works as a bundler should, and was created by the Bun team, so it's likely to be supported on an ongoing basis.
Keep in mind that — like any tool — Bun isn’t perfect. Let’s take a look at some things you should keep in mind before or while using Bun.
Whilst Bun is a fantastic tool with tremendous goals, it’s worth thinking about some of the friction and imperfections around Bun:
bun.lockb
lock file: One of Bun’s more unusual aspects is the lock file it employs. Lock files are well established in the Node.js ecosystem, allowing deterministic installs of packages — like yarn.lock
for Yarn or package-lock.json
for npm. Bun has trodden a slightly different path with a binary lock file, bun.lockb
. Using a binary lock file reduces the file size and improves efficiency in areas like parsing, processing, and version control. However, it also makes working out the difference between an old and a new lock file quite involved. There’s some friction here both in terms of seeing the diff as well as plugging into third-party tools that manage dependency upgradesIt’s worth noting that these are not particularly significant hindrances, and many are not necessarily long-term issues either. As time passes, they’ll likely become even less problematic. However, as of now, they are worth taking into account before you go all-in on Bun.
We’ve touched on why Bun was created and looked at some high-level reasons why you should consider using it. Now, it’s time to dive deeper into what exactly Bun offers. Let's take a look at some of its features.
The aim is that Bun will entirely support Node.js APIs. Consequently, the majority of npm packages — which were originally written for the browser and for Node.js — should just work with Bun.
Bun also includes a package manager, bun install
, which is compatible with npm as well as faster than npm, Yarn, and pnpm by some margin! You can use bun install
to install packages from npm or from a Bun registry.
Notably, you still use package.json
to manage your dependencies in Bun. This makes migrating from Node.js to Bun very easy and reduces the learning curve. If you just want to dip your feet in the water of Bun, you could just use it for installs and speed up your GitHub Actions!
Bun extends the JavaScriptCore engine — the engine that hails from the Safari browser — but with incredible performance, thanks to its Zig implementation. It's fast. Really fast. It's not only way faster than Node.js but also faster than Deno in most benchmarks.
You'll hear about speed in almost everything that Bun does. Speed is a feature. Bun runs fast. Bun starts fast. Bun installs fast. Bun hot reloads fast. Bun bundles fast. Bun runs tests fast. Bun is fast.
You’re likely now tired of reading the word “fast” — but hopefully, you get the point!
If you use .ts
and .tsx
files, Bun can make your life much easier. It can execute these files in the same way Node.js can execute JavaScript. No need to set up a build step or add ts-node — it just works.
One exciting aspect of this feature is that you can write TypeScript directly and execute it in a Node.js style. If you had been relying upon a build step or on JSDoc for strong typing in JavaScript, now you can just use TypeScript!
Similarly, when it comes to React, Bun transpiles it into JavaScript internally — you don’t have to worry about it.
Bun has great support for watch mode and hot reloading. These features help you automatically monitor your source files for changes and then update your running application accordingly.
You’re likely used to hot reloading taking some time; it’s usually not instant. But with Bun, it pretty much is. How, you ask? I’ll paraphrase the docs:
Detects changes to files using OS native filesystem watcher APIs Can scale to larger projects thanks to optimization techniques like setting a higher limit for file descriptors, statically allocating file path buffers, reusing file descriptors whenever possible, and more
In the Bun docs, you can also find a demonstration from the Bun team showing the performance of watch mode:
That is fast.
As part of the JavaScript world, you’re likely very aware that the long-standing but nonstandard CommonJS modules are starting to be displaced by standard ES modules.
TypeScript enforces its own set of rules around import extensions that aren't compatible with ESM. Different build tools support path re-mapping via disparate, non-compatible mechanisms. Node.js, for instance, has many “gotchas” around ESM resolution, making it difficult to use in practice.
Bun aims to provide a consistent and predictable module resolution system that just works. It does this very well indeed. Code that Node.js might struggle with, Bun handles with ease.
This is a great example of Bun taking the pain out of using JavaScript. It’s a great example of how Bun saves time and increases happiness.
Some web APIs aren't relevant in the context of a server-first runtime like Bun — for instance, the DOM API or History API. But many APIs are, and Bun embraces them. It doesn't reinvent the wheel; it leans into the ecosystem.
A great example is the Fetch API, an API that has been around for a long time in the browser but is relatively new to Node.js. Bun supports Fetch out of the box, along with providing partial or complete support for many other Web APIs.
Another built-in Bun feature is its ultra-fast test runner, which is compatible with the Jest testing framework. The Bun runtime itself executes the tests, contributing to a smooth DX and optimal performance.
Bun’s testing capabilities support features like the following:
--watch
: You may prefer to have your tests running in the background and getting that feedback as files are updated. Bun tests give you just what you need with the speed you’ve likely come to expectThese comprehensive testing features are a compelling alternative to what’s out there already. But perhaps more significantly, if you have tests you’re already writing in another test framework — which likely describes the majority of cases — then you can still use them with Bun.
All platforms benefit from the ability to debug. Presently, Bun supports debugging in two ways:
Of these two, the recommended approach at the time of this writing is to use the web debugger. The VS Code debugger is still in beta and has some bugs.
Bun implements a set of native APIs on the Bun global object and through a number of built-in modules. You can use these APIs, but they generally amount to being aliases for the Node.js API equivalents.
If you’re concerned about vendor lock-in, or if you want your code to be more typical to readers, you may want to use the Node.js APIs over Bun’s.
So where would you use Bun? What are some of its ideal use cases?
There are many, but here are some of the use cases that I've used Bun for:
You can explore many of the other use cases for Bun in Bun's guides.
Bun is a JavaScript runtime. So is Node.js, and so is Deno. So what's the difference between them? Why would you use Bun over Node.js or Deno?
It's possible to look at Bun as "Node.js - but better!". By this I mean it supports the same APIs as Node.js, but it's faster and often easier to use.
For example, you can write TypeScript with Node.js, but you will need to do some work to get it to work. With Bun, you can just write TypeScript. The speed of Bun is also a big selling point. Bun is much faster than Node.js — not just a little bit faster, but a lot faster.
Deno is a very similar project to Bun: a fast JavaScript runtime that supports TypeScript. However, though Deno itself is very fast, Bun is faster in many benchmarks than Deno.
Additionally, Deno is a great project, but at launch, it intentionally didn't support npm. This lack of npm compatibility increased friction in the process of migrating a Node.js app to Deno.
Times have changed; Deno now supports npm, and that friction is lessened. But Bun has supported npm from the start and always intended to.
Here’s a comparison table you can use as a quick resource for comparing Node.js, Deno, and Bun:
Node.js | Deno | Bun | |
---|---|---|---|
JavaScript support | ✅ | ✅ | ✅ |
TypeScript support | ❌ | ✅ | ✅ |
JSX / TSX support | ❌ | ✅ | ✅ |
Speed | Reasonable | Faster than Node.js | Faster than Node.js and Deno by many benchmarks |
Node.js API compatible | ✅ | ✅ — permissions will need to be granted at runtime | ✅ |
Module support - CommonJS | ✅ | ❌ | ✅ |
Module support - ECMAScript | ✅ — although not without friction | ✅ | ✅ |
If you love writing JavaScript and TypeScript, then you'll love Bun. It's fast, easy to pick up, and a joy to use. Bun is a great project and I'm excited to see where it goes next.
Do take a look at the Bun website to see its documentation and developer guides. I hope this resource will be helpful as you consider a JavaScript runtime to use in your next project.
This post was originally published on LogRocket and edited by Megan Lee.
]]>This is a follow up to my "How I ruined my SEO" post. That was about how my site stopped ranking in Google's search results around October 2022. This post is about how Growtika and I worked together to fix it.
As we'll see, the art of SEO (Search Engine Optimisation) is a mysterious one. We made a number of changes that we believe helped. All told, my site spent about a year out in the cold - barely surfacing in search results. But in October 2023 it started ranking again. And it's been ranking ever since.
I put that down to the assistance rendered by Growtika. What was the nature of that assistance? I'll tell you. This post is a biggie; so buckle up!
I wrote "How I ruined my SEO" almost as self therapy. I was frustrated that my site's traffic had dropped. I knew it didn't really matter; my motivation for writing my blog is, in large part, about creating a long term memory for myself. But I was still frustrated. I write things that I know people find useful, and so it was suboptimal that my posts were no longer being found.
I should include myself in that. When I'm trying to remember how to do something (and I know I once knew how to do it) I'll often Google it. Hoping to see a blog post I once wrote that answers my question. But, no more. My own site was no longer being found by me. I was missing me. Vanity.
I shared the post on Hacker News, not really expecting much to happen. But it ranked, and in amongst the conversation that followed, someone named Growtika offered to help.
I hadn't heard of Growtika before; SEO is not my world. But it turned out that Growtika specialise in helping organisations with that. Out of the goodness of their hearts, they offered to assist me. Never one to look a gift horse in the mouth, I leapt at the offer.
I spent some time with Growtika talking through my site. They made some suggestions around getting my site to align with best practices. They also schooled me on some of the basics of SEO. I was very much a novice in this area, and so I was grateful for the education.
Here's the thing: SEO is a mystery. Or at least, it's not fully understood. Like Coke haven't published their recipe, Google doesn't publish its (ever evolving) algorithm. They do publish SEO guidelines, but they are just that: guidelines. And so, whilst there are best practices, there is no guarantee that following them will result in success.
What's more, the feedback loop for changes is long. It's not like fixing a program with a bug, where you tweak the code, run the tests and see if it's fixed. It's more like making a change to a program, and then waiting weeks or months to see if it's fixed. And if it's not, you have to wait again to see if the next change you make fixes it.
Cause and effect are just not as obvious as you might like, when it comes to SEO. So whilst I'm going to share what we did, I can't say for sure what actually lead to the improvement in my site's SEO. I'm confident that they were all good things to do. But I cannot be certain which of them made the difference.
As an aside, Growtika think that it's pretty absurd that developers who write high quality technical articles (and for the sake of this point, let's say mine fit into this category sometimes), need to run the gauntlet of SEO best practices to get ranked. It really shouldn't be necessary. With one of the recent Google algorithm updates; the helpful content algorithm update, it feels like Google are starting to understand that it and prioritize it in their search engine. But there's still a long way to go.
Over the time we worked together, Growtika made a number of suggestions. Changes we might make that could improve my SEO. I'm going to go through the suggestions over the rest of the post. I'll also share some of the rationale as I go along.
There's a concept used by Google for ranking known as Experience, Expertise, Authoritativeness, and Trust (E-E-A-T). It's about how much Google trusts the content on your site. When evaluating an author's profile for a blog post, it's worth considering the following E-E-A-T aspects:
I didn't have much that addressed these points on my site.
On each blog post I had a profile picture. But it wasn't being all it could be; it looked like this:
It's my face and the text "John Reilly" which linked through to my Twitter (now X) profile page. Nice enough but not really demonstrating my expertise and authority on topics. I updated it to look like this:
Alongside my picture and name I added a byline to demonstrate my expertise and authority on topics: "OSS Engineer - TypeScript, Azure, React, Node.js, .NET". Alongside that, I switched the link to the about page on my site instead of Twitter.
Since the author profile at the top of each post didn't answer all the E-E-A-T points, enriching my about page was the way forward. I updated the about page to include a richer bio and a list of places where my site has been featured:
This was to demonstrate my expertise and authority on topics. We even snuck in some structured data - more on that later!
My site is built using Docusaurus. Now I love Docusaurus, but it's not perfect. One of the problems with it is that it generates a number of pages that are not helpful for SEO as they duplicate content. (This is a bad thing for SEO.) Consider this report:
The report suggests there was a good amount of duplicate content on my site. This was because Docusaurus generates "pagination" pages which allow you to navigate click by click through the whole history of a blog.
Also Docusaurus creates "tag" (or category) pages that reproduce blog posts under tags that have been used to categorise the posts:
In both cases, these pages duplicate content - which lead to the above report. Rather frustratingly, the pages also feature canonical
link tags which rather suggest that they are the canonical source of the content:
In my case, some of these pagination or tags pages were being prioritised over the original blog posts. This felt quite strange from a readers point of view. We took a number of steps to address this.
noindex
unnecessary pagesI wanted to remove or noindex
the pagination and tags pages to provide a clear signal to search engines about which pages were the most important. I couldn't remove the pages without breaking the navigation on my site, so I chose instead to noindex
them. My site is hosted on Azure Static Web Apps and so I was able to achieve this fairly easily by adding the following to my staticwebapp.config.json
file:
{
"route": "/page/*",
"headers": {
"X-Robots-Tag": "noindex"
}
},
{
"route": "/tags/*",
"headers": {
"X-Robots-Tag": "noindex"
}
},
This meant that the pagination and tags pages (which were served up under URLs beginning /page/
and /tags/
respectively) were still available, but search engines were encouraged not to index them by the X-Robots-Tag: noindex
header that these pages now served.
I might see if I could land a change like this in Docusaurus itself. I think it would be helpful for others. The mechanism would need to be slightly different, as Docusaurus doesn't have control over headers your site serves. But I think it would be possible to add a noindex
meta tag to the pagination and tags pages HTML as is suggested here:
<meta name="robots" content="noindex" />
As I've mentioned, we have pagination and tags pages that duplicate content. It's possible to make this slightly better by truncating the content on the pagination and tags pages. It amounts to adding a truncate
marker early into the content of each blog post.
With this in place, the pagination and tags pages don't duplicate the content of the blog posts in full, but instead just feature a snippet of the content. There's documentation on how to do this in Docusaurus here.
I did this for every page on my blog. Given I have quite a lot of posts, doing it manually would have been tedious. So I scripted the insertion of a truncate marker after the first paragraph of each post, and it was done in a jiffy.
I also performed something of a tag rationalisation. I had a lot of tags, and many of them were not used on more than one blog post. Also, many of them were not the greatest of tags, as this slightly embarrassing screenshot demonstrates:
As is probably apparent, I'd not really thought about tags much. I'd just added them as I'd written blog posts. I'd tagged first, asked questions later. I removed a lot of the (rather pointless) tags I had and also added a tags to blog posts that were missing them. This removed the "noise", so search engines understand the content of my blog posts, and readers also. Less is more.
Much better!
sitemap.xml
and robots.txt
Alongside noindex
ing the pagination and tags pages, we took a look at my sitemap.xml
- this is automatically generated by Docusaurus. I removed the pagination and tags pages from the sitemap.xml
as it's a little confusing to noindex
a page and then include it in the sitemap.xml
.
Further to that, I write posts for other websites sometimes and cross post it on my own blog, with a canonical pointing to the original post. Having these posts in the sitemap.xml
wasn't quite right as they are not the canonical source of the content. I removed them.
I've a number of post processing steps that run in my build step of my site, and I included this filtering in it. In the end it amounted to filtering an XML file; which is pretty straightforward - I wrote about it to demonstrate.
As well as filtering my sitemap.xml
, I went a little further and added lastmod
timestamps to the sitemap.xml
based on the git commit date of the markdown file that the blog post was generated from. This was to help search engines understand how recent the content on my site is. I wrote about how I did this. Google have subsequently announced that they use lastmod
as a signal for scheduling re-crawling URLs and so this turns out to have been a helpful change to make.
Alongside this, I added a robots.txt
to my site. These are files that search engines use to understand the structure of a site and what they should and should not index. I didn't previously have one and the one I added was pretty rudimentary:
User-agent: *
Allow: /
Sitemap: https://johnnyreilly.com/sitemap.xml
I don't know how much this helped, but it certainly didn't hurt.
We next looked at our internal linking strategy. This is about how we link to other pages on our blog from within our blog posts. The idea is that we should link to other pages on our blog that are relevant to the topic of the blog post. This helps search engines understand the structure of our blog and the relationships between the pages.
This was something that I did a little, but I didn't really think about. I'm now much more intentional around internal linking. I'm very much an editor of my content, and as I'm editing my posts / writing new posts I'll take a look at whether I'm linking to other relevant posts on my blog and whether perhaps I should be.
You'll possibly have noticed a good number of internal links in this post! I'm careful about how I do this - I have internal links where they are relevant and where I think it adds value. I don't have internal links for the sake of it. Whilst I want to improve my SEO, the main readers of my blog are humans, and I want to make sure that I'm not making the experience worse for them.
Alongside upping my internal linking game, Growtika suggested that the footer of my site was a prime place to add links to notable posts on my site, and also to provide an indication of topics that this site seeks to cover.
A picture is worth a thousand words, so here's what the footer of my site used to look like:
And here is what it looks like now:
As you can see, the difference is significant. With the new enhanced footer I can call out particular articles around themes that I cover, I can highlight popular articles, and I can also emphasise articles that I think are particularly important, or recently updated. This is both about helping search engines understand what I consider to be important in my site, it's also helpful for humans that might scroll that far down. And goshdarnit, I think it looks rather fine too!
You likely know that a primary way that search engines find content on your site is by following links. They have crawlers that do this. The more links that a crawler has to follow to get to a page, the more difficult it is for the crawler to find that page. This is known as the pages crawl depth.
My initial site structure was not great. I had a number of pages that were 4+ clicks away from the home page:
A primary reason for my pages crawl depth this was the pagination and tags pages I mentioned earlier. We originally displayed a single full length (not truncated) blog post per page, and so the pagination pages were many. Likewise, we had a lot of tags, and so the tags pages were many. This meant that the pages crawl depth was high. You want to keep the pages crawl depth as low as possible. Less is more.
We increased the number of blog posts displayed per page from 1 to 20 which dramatically reduced the amount of work the crawlers had to do. So instead of having few hundred pagination pages we reduced it to 16. Much better.
It used to be the case that the URLs for my blog posts always featured the date of publication. This was a hangover from when I used to use Blogger as my blogging platform. I'd migrated from Blogger to Docusaurus, and I'd kept the date in the URL. It so happens that Docusaurus has a similar behaviour too.
Growtika suggested that I remove the date from the URL. This was to make the URLs shorter and more readable. Search engines also have a preference both for shorter URLs and for pages that are recent, rather than pages that are old. So removing the date from the URL would help with both of those things. Or at least it would stop search engines that looked for the date in the URL from thinking that older content was irrelevant. (And with our lastmod
timestamps in the sitemap.xml
we were already helping search engines understand how recent the content on my site was.)
I must admit, I didn't really want to make this change. I rather liked having the date in the URL. But, in Growtika we trust. I did it.
Where you used to go to:
https://johnnyreilly.com/2019/10/08/definitely-typed-movie
You now go to:
https://johnnyreilly.com/definitely-typed-the-movie
And of course, we made sure a redirect mechanism was in place to ensure that the old URLs still worked. More on that later - you can test the redirect if you like!
To implement this we used the slug feature of Docusaurus. If you want to see the mega PR that implemented this on nearly 300 blog posts it's here. You won't be surprised to learn I scripted this change - life's too short to do boring things by hand.
As I've said, Docusaurus is great but it historically has had some defaults that hurt SEO from a blogging perspective. One of these I identified when I was first planning to migrate from Blogger to Docusaurus. Docusaurus didn't ship with a blog archive. This is a page that allows you to browse through your historic blog posts. A place where you can see all that you've written and when. I find this very helpful. It's also helpful for search engines to understand the structure of your site.
I hand-rolled my own blog archive for Docusaurus before I migrated. It looked like this:
My implementation was later made part of Docusaurus itself by Gabriel Csapo in this PR. So now by default, all Docusaurus sites have a blog archive that lives at /archive
in the blog. This is great news for Docusaurus users!
In one if the more speculative changes we made, we changed the URL of the blog archive from /archive
to /blog
(and the associated navbar label).
It was a wild guess (and it may not have made any difference) but the thinking was that it might affect the CTA (call to action) of people who see my site on Google. If they saw old date in the URL and "archive" in the breadcrumbs, maybe they'd think the site is "not relevant for the search I have now"?
So our tweaked blog archive page now looked like this:
We also added a 301
redirect from /archive
to /blog
to ensure that any links to the old URL would still work.
One of the most intriguing strategies we followed was to build on the structured data support in my site. Structured data is a way of providing metadata about a page in a machine readable format. It's a way of providing a clear signal to search engines about the content of a page; it makes their lives easier.
As it turned out, I already had some structured data support in my site; I'd written about how to add it previously. But we were to go further!
One of the experiments we ran was to add FAQs to a post, and with that, the equivalent FAQ Structured Data. The intent being to see if this would help with the SEO for that post. So, because I'm super meta, I wrote a post about how to do that which included FAQs and the equivalent structured data.
I also added FAQ structured data to another post and Growtika resubmitted it to Google for indexing. Then two things happened. Firstly, the page was indexed:
And then the page started featuring FAQs in the search results:
I've included the reactions at the bottom of each screenshot above - we were quite excited!
Beyond adding individual structured data to each page and post, I added site wide structured data. This would proclaim from the rooftops about the nature of my site.
So I decided to add site wide structured data of the following types: (there are many types of structured data which you can read about at https://schema.org/ and in this Google document on the topic)
You can see how the structured data is implemented in this PR. We used the headTags
API in Docusaurus to add site wide JSON-LD structured data. Funnily enough, I contributed the headTags
API to Docusaurus long before I thought I'd end up using it for this!
In this change we are heavily inspired by the full structured data graph work Yoast have done. With site wide structured data in place, every page that search engines index on my site will have structured data that describes the site as a whole.
Finally I added breadcrumbs to my blog posts. Breadcrumbs are a way of indicating to search engines where a page sits in the hierarchy of a site. I wrote about how I did this. It's worth noting that the approach outlined in that post I've subsequently simplified. Originally I added a breadcrumb for the page structure and also one per tag on the post. I've since removed the tag breadcrumbs as they were not adding much value. Less is more.
I mentioned in "How I ruined my SEO" that I had a number of backlinks to my site. I also mentioned that I had broken a number of the backlinks by my carelessness. I planned to fix the broken backlinks and also do a better job of backlinks in general.
I'd already implemented support for dynamic redirects on my site. What this meant was, if someone linked through to a non-existent page on my site, rather than having just a 404 Not Found, I could do some fairly sophisticated work to redirect them to the correct URL. Thus protecting (and unbreaking previously broken) backlinks. By the way, using Azure Static Web Apps as my hosting mechanism really helped me out here as the dynamic redirect mechanism I had was super powerful - I wasn't limited to regexes. If you want see how I did that have a read of this.
What I had was good, but I could do better. I did the following:
Again, less is more. With these changes made, I had a much better backlink story.
Another aspect which factors into SEO is performance. Google has a Core Web Vitals measurement which is about evaluating the performance of websites. It covers how fast a website loads, how responsive it is / how quickly it becomes interactive.
The thing that was hurting my site's performance was images. The images on my site were generally not optimised at all. They were also not lazy loaded. This meant that the images were slowing down the loading of my site, and this reflected in my Lighthouse scores.
I took a number of actions to improve the site image performance.
The first, and most obvious, was to optimise the images on my site. There's many ways you can do this; I chose to use TinyPNG's API. I wrote about how I did this. Ultimately I wrote a script that optimised all the images on my site, and allowed me to run it on demand for the images of a particular post.
This shrunk the file size of images on my site served significantly, and improved the performance. Once again, less is more.
I also added Lighthouse to my site's build step, so I could get some performance measurements surfaced into my pull requests. This made it easy to catch potential regressions, where I might accidentally add unoptimised images to my site.
Having tackled the low hanging fruit of images not being optimal in the first place, I then went further. Cloudinary offer a CDN that can transform images on demand. This means that you can serve the same image in different sizes, formats and qualities depending on the device that is requesting it. This is a great way to improve performance.
I was able to plug the Cloudinary CDN into my site using by building a the rehype-cloudinary-docusaurus
plugin which can be used to integrate Cloudinary into Docusaurus. You can read more about how it works here.
Now when my site serves an image, it serves the optimal image for the device that is requesting it. This improves the performance of my site. (And you can do this too if you're using Docusaurus!)
In fact I went a little further and scripted the patching of my Open Graph images to make use of Cloudinary too. This meant that the images that were shared on social media were also optimised for the device that was requesting them. I don't think this helped with SEO, but I'd noticed that large and / or slow loading Open Graph images aren't always used by platforms that support the Open Graph protocol. With the Cloudinary image transformation CDN in place, this became much less of an issue.
Incidentally, Cloudinary got wind of this change and invited me onto their DevJams live stream to talk about it. I was very flattered to be asked and it was a lot of fun!
fetchpriority="high"
and loading="lazy"
So far we'd handled the performance of images on my site by optimising them and serving them in an optimal way. But there was more we could do. We could also make sure that the images on my site were loaded in an optimal way.
We did this by adding fetchpriority="high"
to the first image on each blog post; the "title" image if you will. This is a hint to the browser that the image is important and should be loaded as soon as possible. We also added loading="lazy"
to all the other images on a given blog post. This is a hint to the browser that those images (the "below the fold" images) are not as important, and can be loaded later if and when they are required.
The effect of these two changes combined, is that when a browser lands on a blog post it loads the first image as soon as possible, and then it doesn't immediately load the rest images; it focuses on giving the user a usable page. The upshot of this is that the Largest Contentful Paint (LCP) is loaded as soon as possible and the browser remains more responsive. The remaining images may be loaded... Or they may not - it depends on whether people scroll down to them. This translates into improved perceived performance / user experience.
And again: less is more. I've written about how using fetchpriority="high"
and loading="lazy"
was implemented in depth here.
One of the ideas I'd had as I attempted to fix my SEO was to publish my content on other sites. I'd seen other people do this, and I thought it might be a good idea. So I set up a mechanism that published my blog posts to dev.to with the canonical pointing back to my site. I was so pleased with my idea I even wrote about how I did it.
Publishing content on other sites isn't inherently negative. However, in my case, it created confusion. I'd hoped that publishing my content on dev.to would help my SEO. It didn't. My content on dev.to ranked higher than the original content on my own website (most times my site didn't rank at all).
Growtika were keen to "cancel the noise", which would improve their understanding of my ranking situation. Since dev.to was ranking instead of my site, it was difficult to analyze how long it took for articles to rank on my site. Stopping the submission of content to external sites would help clarify the situation.
I turned the mechanism off. This helped them determine the time it took for my site to achieve a ranking.
Whilst actually publishing your content on other sites with a canonical turns out not to be the best idea, getting your site featured in relevant places is a good idea. This is about getting your site linked to from other sites. This is a signal to search engines that your site is relevant and important.
I already had a number of links to my site from other sites. For instance, I'd regularly show up in Azure Weekly, The Morning Brew and a number of other sites. Many of these use my RSS feed.
I also submitted my site to a number of other places. For instance, I submitted my site to daily.dev. A good rule of thumb here for picking places tended to be "where do you go to read about the topics you write about?". But crucially I didn't place my content on others sites, I just linked to my site from other sites.
As a side note, I rather wish that RSS still thrived. I support it - people still use it to read my blog. But it's not as popular as it once was.
The final tweak we're going to cover, is adding meta descriptions to my posts. This is a short description of the content of the blog post that is included in the HTML of the page:
<meta
name="description"
content="In October 2022 traffic to my site tanked. Growtika collaborated with me to fix it. This is what we did. Read it if you're trying to improve your SEO."
/>
It's not visible to humans, but it is visible to search engines. And it's kind of visible to humans at one remove, in that it is often used as the description of the page in search results.
We followed these meta description guidelines:
I'd previously not included meta descriptions on my blog posts. I found myself with the daunting task of writing meta descriptions for nearly 300 blog posts. I decided to script it. I wrote a script that would generate a meta description for each blog post based on the content of the blog post, powered by Azure Open AI. I then ran the script and added the meta descriptions to my blog posts. I wrote about how I did this here.
This left me with meta descriptions for all my blog posts. It was also rather fun to use AI for something that was not GPT or copilot related!
As I mentioned earlier, my SEO has now recovered. I'm ranking again in search results. I'm not ranking quite as highly as I was before; I think my site is possibly still in the throes of recovery. But without a doubt, it's definitely trending in the right direction. I want to show you some graphs and numbers to demonstrate this.
Here's a graph from Ahrefs showing my site's performance over the last two years:
Notably, you can see the drop off happened around the time of a Google algorithm update. Likewise, it's really important to highlight that the traffic increased around the time Google made another algorithm update. It's impossible to know if the traffic would still have gone up if we had not made our changes. Perhaps it would, maybe to a lesser extent. We just don't know. It does demonstrate the power of Google's algorithm updates though. The graph alone tells a story, a phenomenal drop off in traffic followed by a recovery.
A couple of simpler graphs from Google Search Console tell a similar story. Here's a graph showing my site's performance over time:
Observe the massive drop off in October 2022. And then the recovery in October / November 2023. It's a similar story to the ahrefs graph above. Further to that, here is the traffic for 28 days in February 2023 vs 28 days in October / November 2023:
If you're reading on a mobile device you may not even realise that there are two lines on the graph; the February 2023 line is so low and flat as to be almost invisible.
The difference is stark. I couldn't get arrested from a search perspective in February 2023; showing up a mere 5,000 times in search results. That sounds like a big number, but in search terms it's really not. But by October / November 2023 I was back in the game, showing up 274,000 times in search results. That's a 55x increase!
An interesting aside, that I've excluded from the graph, is that my total clicks increased from 1,160 to 5,470 when comparing Februray to October / November. This is a mere factor of 4. What does that mean? It means when my site was showing up very rarely in search results, it enjoyed a very healthy click through rate. People clicked on my search results when they saw them!
This post has mostly been driven by discussing how Google stopped ranking my site in their search results. What I haven't really mentioned is that other search engines did not stop ranking my site. I was still showing up in Bing and DuckDuckGo's search results. Here's a graph from Bing webmaster tools (Bing's equivalent to Google Search Console) showing my site's performance over the last six months:
There's two things to take from the above graph:
You might be curious as to what the actual impact of that is on my sites traffic. It's fairly significant; around 80% of my sites traffic comes from search engines. So when my Google SEO tanked, my traffic was significantly (but not terminally) impacted. Consider the graph below from Google Analytics:
It covers a similar time frame to the Google Search Console and ahrefs graphs above. It shows that now, on a weekday (weekends are quieter) I get around 350 unique visitors to my site. Whereas for the comparison period it was more like 100 unique users per day. That's roughly a 3.5x% increase in traffic; which is not to be sniffed at.
This is not a post I'd expected to write. SEO used to be something I didn't think about much. But it turns out that a way to get my attention, is taking away my traffic! I'm actually rather grateful that all this happened as it got me to thinking and learning about SEO in a way that I quite enjoyed.
As I say, whilst I can't be certain which of the changes we made made the difference, I'm confident that my site now is better than my site a year ago. It loads faster, it's more performant, it's more structured, it's better linked, it's better optimised. It's better. It looks the part too. I'm really quite proud of it.
I'm also phenomenally grateful to Growtika for their help. I should say that a few others offered pointers and suggestions which I was thankful for. But it was Growtika who stuck by my side for the long haul. For nearly a year they worked with me; and for a long time saw no improvements in my sites traffic at all. They didn't give up. They were patient with me, and they were generous with their time and expertise. I'm very grateful to them for all their help.
If you're looking for help with SEO, I'd recommend you check out Growtika. They're fantastic folk!
]]>This post fills in the gaps for a TypeScript Azure Function. It's probably worth mentioning that my blog is an Azure Static Web App with a TypeScript Node.js Azure Functions back end. So, this post is based on my experience migrating my blog to v4.
I'm going to walk through the migration of my blog from v3 to v4. This takes place in this pull request. I'll probably cover some of the ground of the offical JavaScript upgrade docs, but I'll also cover some of the TypeScript specific stuff.
There will be two main parts to this post:
package.json
The second part will be the bulk of the post, but the first part is important too.
package.json
So, starting with the first part, there are three changes to make to the package.json
:
"dependencies": {
+ "@azure/functions": "^4.0.1",
},
"devDependencies": {
- "@azure/functions": "^3.5.0",
},
+ "main": "dist/src/functions/*/index.js"
@azure/functions
dependency to ^4.0.1
(or later)@azure/functions
dev dependency becomes a regular dependency - this is because we'll be using the package at runtime now - previously we just used it to get the types at build timemain
property to the package.json
with a glob that matches the functions in your project; in my case dist/src/functions/*/index.js
- which will be our output from the TypeScript buildAs I took care of 3., I found myself changing the folder structure of my functions. Actually, this isn't mandatory, but it was tricky for me to come up with a glob for my current structure. So I moved things around - you may not need to. All that matters is that your glob matches the output of your build.
In order that we can understand what migration looks like, we must first take a look at the v3 version of a function. Here's the fallback
function from my blog:
import type { AzureFunction, Context, HttpRequest } from '@azure/functions';
import { redirect } from './redirect';
import { saveToDatabase } from './saveToDatabase';
const httpTrigger: AzureFunction = async function (
context: Context,
req: HttpRequest,
): Promise<void> {
try {
const originalUrl = req.headers['x-ms-original-url'];
const { status, location } = redirect(originalUrl, context.log);
await saveToDatabase(originalUrl, { status, location }, context.log);
context.res = {
status,
headers: {
location,
},
};
} catch (error) {
context.log.error(
'Problem with fallback',
error,
req.headers['x-ms-original-url'],
);
}
};
export default httpTrigger;
The above is the code I use to power dynamic redirects in my Azure Static Web App with the Azure Function back-end. It's a TypeScript Azure Function that takes a request, redirects to a new location and saves metadata about the redirect to a database.
Looking at the code now, I rather think I should have called the function redirect
rather than fallback
. I'll leave it as is for now, but I'll probably change it in the future.
What the fallback
function does isn't significant for this post, but the structure is. Now let's look at the migrated version:
import type {
HttpRequest,
HttpResponseInit,
InvocationContext,
} from '@azure/functions';
import { app } from '@azure/functions';
import { redirect } from './redirect';
import { saveToDatabase } from './saveToDatabase';
export async function fallback(
request: HttpRequest,
context: InvocationContext,
): Promise<HttpResponseInit> {
try {
const originalUrl = request.headers.get('x-ms-original-url') || '';
const { status, location } = redirect(originalUrl, context);
await saveToDatabase(originalUrl, { status, location }, context);
return {
status,
headers: {
location,
},
};
} catch (error) {
context.error(
'Problem with fallback',
error,
request.headers.get('x-ms-original-url'),
);
return {
status: 500,
body: 'something went wrong',
};
}
}
app.http('fallback', {
methods: ['GET'],
handler: fallback,
});
As we can see, the logic looks pretty much the same. But a lot has changed. What's different? We'll go through the changes one by one.
import
s usedStarting at the top, the import
s we use are different:
-import type { AzureFunction, Context, HttpRequest } from '@azure/functions';
+import type {
+ HttpRequest,
+ HttpResponseInit,
+ InvocationContext,
+} from '@azure/functions';
+import { app } from '@azure/functions';
We're no longer just importing types, we're importing the app
function from @azure/functions
also. The types that are being imported are different too. We're no longer importing AzureFunction, Context, HttpRequest
- instead we're importing HttpRequest, HttpResponseInit, InvocationContext
.
app
, goodbye function.json
As we saw, we're making use of the app
function from @azure/functions
. This is a new function that we use to register our Azure Functions. We no longer use function.json
. Instead we use app
. In the case of my fallback
Azure Functions, we register it like this:
app.http('fallback', {
methods: ['GET'],
handler: fallback,
});
We're registering an HTTP trigger called fallback
that responds to GET
requests. The handler
is the function that will be called when the trigger is invoked. There's more options available, but this is the minimum we need to register our function.
This minimal TypeScript/JavaScript replaces the more verbose function.json
that used to sit alongside:
{
"bindings": [
{
"authLevel": "anonymous",
"type": "httpTrigger",
"direction": "in",
"name": "req",
"methods": ["get", "post"]
},
{
"type": "http",
"direction": "out",
"name": "res"
}
],
"scriptFile": "../dist/fallback/index.js"
}
All of this is gone, replaced by the app
function usage. There's one part of the function.json
that isn't covered by the app
function, and that's the scriptFile
property. This is covered by the main
property we added to the package.json
earlier.
The rest of the function.json
is covered by the app.http
call. Much terser.
function
The signature of our function has changed too:
-const httpTrigger: AzureFunction = async function (
- context: Context,
- req: HttpRequest
-): Promise<void> {
+export async function fallback(
+ request: HttpRequest,
+ context: InvocationContext,
+): Promise<HttpResponseInit> {
You'll see here that we're using a function declaration rather than a function expression. Our new function takes our new types, the subtly different HttpRequest
and InvocationContext
, which are similar to, but different from, the previous Context
and HttpRequest
types. The order of these parameters has changed also.
The return type of the function is now Promise<HttpResponseInit>
rather than Promise<void>
. What this means is, we're going to return values from our function, which we didn't do previously. Let's look at the implications of this.
context.res
to Promise<HttpResponseInit>
With a v3 function, we'd set the context.res
property to return values from our function. With a v4 function, we return values from our function directly. What does this look like?
- context.res = {
- status,
- headers: {
- location,
- },
- };
+ return {
+ status,
+ headers: {
+ location,
+ },
+ };
I rather like this change. My reasoning is that, in the event that there is subsequent code that would otherwise run after context.res
was set, we no longer need to remember to subsequently return
to prevent that running. (And yes, I have made that mistake on multiple occasions.) All we need do is return the value we want to return from our function.
body -> jsonBody
Another difference is that we no longer set the body
property of the context.res
. Instead we return an object with a jsonBody
property, assuming we're returning JSON from our API. (And that's the most common case, right?)
This wasn't illustrated in the fallback
function above, but here's an example of migrating a function that returns JavaScript object literal named redirectSummary
:
- context.res = {
- status: 200,
- body: redirectSummary,
- };
+ return {
+ status: 200,
+ jsonBody: redirectSummary,
+ };
The body
property still exists, but it's for returning strings and other things, rather than JSON. If you're returning JSON, the easiest approach is to use the jsonBody
property.
Finally, the APIs offered by the request
and context
objects are different. I shan't go into detail here as it's well covered in the official documentation. But I will show you one of the changes I made to my fallback
function:
- const originalUrl = req.headers['x-ms-original-url'];
+ const originalUrl = request.headers.get('x-ms-original-url') || '';
Not too significant a tweak, but there's a number of slight changes like this to make. (Related to this, the logging API on the context
object is also different - but not significantly.)
If you're a fan of running locally then you may need to make this tweak to your host.json
:
"extensionBundle": {
"id": "Microsoft.Azure.Functions.ExtensionBundle",
- "version": "[3.*, 4.0.0)"
+ "version": "[4.*, 5.0.0)"
}
A complete host.json
might look like this:
{
"version": "2.0",
"logging": {
"applicationInsights": {
"samplingSettings": {
"isEnabled": true,
"excludedTypes": "Request"
}
}
},
"extensionBundle": {
"id": "Microsoft.Azure.Functions.ExtensionBundle",
"version": "[4.*, 5.0.0)"
}
}
As discussed with Eric Jizba on this GitHub issue, updating the host.json
may not actually be necessary. For me it seemed to be the thing that turned a not working setup into a working setup; but it's possible I was mistaken. Certainly if I revert the change now I'm still able to run locally. What I'm saying is: your mileage may vary.
Migrating an Azure Function from v3 to v4 with TypeScript is a little more involved than I'd expected. But I do like that this moves us to a code style that feels more "Node-y". The official documentation is good, but it's not complete right now. There's now a decent upgrade guide available: https://learn.microsoft.com/en-us/azure/azure-functions/functions-node-upgrade-v4?tabs=v4
Hopefully this post will help you migrate your TypeScript Azure Functions to v4.
]]>This post will show you how to do that using Bicep.
What we want to achieve can be summmed up by this screenshot:
Inside the Azure Portal, inside our Static Web App, we want to see the App Insights tab and we want to see a linked App Insights instance. We do. But how?
The Bicep code to achieve this is pretty simple:
var tagsWithHiddenLinks = union({
'hidden-link: /app-insights-resource-id': appInsightsId
'hidden-link: /app-insights-instrumentation-key': appInsightsInstrumentationKey
'hidden-link: /app-insights-conn-string': appInsightsConnectionString
}, tags)
resource staticWebApp 'Microsoft.Web/staticSites@2022-09-01' = {
name: staticWebAppName
location: location
tags: tagsWithHiddenLinks // <--- here
sku: {
name: 'Free'
tier: 'Free'
}
properties: {
repositoryUrl: 'https://github.com/johnnyreilly/blog.johnnyreilly.com'
repositoryToken: repositoryToken
branch: branch
provider: 'GitHub'
stagingEnvironmentPolicy: 'Enabled'
allowConfigFileUpdates: true
buildProperties:{
skipGithubActionWorkflowGeneration: true
}
}
}
Consider the code above; it's a fairly standard Bicep resource declaration for a Static Web App. The only difference is the tags
property. We're using the union
function to add three additional tags to the tags
property that has been passed into the static-web-app.bicep
module. These tags are the hidden-link
tags that link the Static Web App to the App Insights instance.
Luke Murray has a great post on hidden-
tags in Azure that I recommend you read if you want to know more about them. Essentially, hidden tags are tags that don't show up in the Azure Portal and have some metadata purpose. hidden-link
tags are a subset of those that are used to link resources together.
hidden-link
tags?In the case of the hidden-link
tags we're using here, we need to know the id
, InstrumentationKey
and ConnectionString
of the App Insights instance we want to link to. We can get these values from the App Insights instance itself. Because of the way Bicep works, it's necessary to get those in a parent module to the Static Web App module. Here's an example:
resource appInsightsResource 'Microsoft.Insights/components@2020-02-02' existing = {
name: appInsightsName
}
module staticWebApp './static-web-app.bicep' = {
name: '${deployment().name}-staticWebApp'
params: {
// ...
appInsightsId: appInsightsResource.id
appInsightsConnectionString: appInsightsResource.properties.ConnectionString
appInsightsInstrumentationKey: appInsightsResource.properties.InstrumentationKey
// ...
}
}
With this is place we have the linking we need to get to our logs quickly as we navigate around inside the Azure Portal. And we have it in place in a way that's repeatable and consistent. I hope you find this useful.
]]>.env
files to configure my applications. They're a great way to keep configuration in one place and to keep it out of the codebase. They're also a great way to keep secrets out of the codebase.
But what if you need to use a multiline string in a .env
file? How do you do that? You just do it:
SINGLE_LINE="you know what..."
MULTI_LINE="you know what you did
LAST SUMMER"
That's right, you just use a newline character. It's that simple. Oddly, searching for that on the internet didn't yield the answer I was looking for. So I'm writing it down here for posterity.
With your .env
file in place, you can then consume it in your application using a package like dotenv
. Or if you'd like to use a bash script to consume the .env
file, you can do it like this:
#!/usr/bin/env bash
set -a
source test.env
set +a
npm run start # or whatever you need to do
I use Azure Pipelines for much of my day to day work. However, my OSS work is (unsurprisingly) all GitHub oriented. Using Bun in GitHub Actions is straightforward; you just make use of the Setup Bun GitHub Action to install Bun and you're off to the races. There isn't an equivalent for Azure Pipelines but that doesn't matter.
It turns out there's a great variety of ways to install Bun. However the simplest of the lot is to install it via npm, like so:
npm install -g bun
This installs the bun
command globally from the bun
package. You can then use it to run your TypeScript / JavaScript. This is the approach we'll use in Azure Pipelines.
Since there's already a dedicated Azure Pipelines task for npm, we can use that to install Bun. Here's an example of how to do that:
- task: Npm@1
displayName: setup bun
inputs:
command: 'custom'
customCommand: 'install -g bun'
verbose: true
Now we're able to use Bun in our Azure Pipelines. Here's an example of how to use it to install dependencies and run a build:
- bash: bun install
displayName: 'install'
- bash: bun run build
displayName: 'build'
We're able to use Bun in Azure Pipelines by installing it via npm and then using it as we would Node.js. This is a great way to speed up your TypeScript / JavaScript builds. I hope you find it useful!
]]>A wonderful aspect of this approach is that no human need ever get to see the connection strings / access keys. They'll be discovered and consumed by Azure during a deployment, and known to your application at runtime, but untrustworthy humans need never get to see them. This is secure, and therefore good.
The blog you are reading this on is hosted on Azure Static Web Apps and deployed with Bicep. It also has an Azure Cosmos DB database and an Application Insights instance. The Azure Static Web App has access to the database via its access key and has access to the Application Insights instance through a connection string. The key and connection string are supplied to the configuration of the SWA during deployment.
Let's look at the Bicep configuration that deploys a database. Here's a snippet of the Bicep template:
resource databaseAccount 'Microsoft.DocumentDB/databaseAccounts@2023-04-15' = {
name: cosmosDbAccountName
kind: 'GlobalDocumentDB'
location: location
tags: tags
properties: {
consistencyPolicy: { defaultConsistencyLevel: 'Session' }
locations: locations
enableAutomaticFailover: true
databaseAccountOfferType: 'Standard'
publicNetworkAccess: 'Enabled'
ipRules: [for ipAddress in ipAddresses: {
ipAddressOrRange: ipAddress
}]
backupPolicy: { type: 'Periodic', periodicModeProperties: { backupIntervalInMinutes: 240, backupRetentionIntervalInHours: 720 }}
}
}
Here's a snippet of the Bicep template that deploys the Application Insights instance:
resource appInsights 'Microsoft.Insights/components@2020-02-02' = {
name: appInsightsName
location: location
kind: 'other'
properties: {
Application_Type: 'web'
Flow_Type: 'Bluefield'
WorkspaceResourceId: workspace.id
RetentionInDays: 90
IngestionMode: 'LogAnalytics'
publicNetworkAccessForIngestion: 'Enabled'
publicNetworkAccessForQuery: 'Enabled'
}
}
Given that both of these resources are deployed, we can reference them subsequently and acquire connection strings / access keys.
So when we're getting ready to deploy the Azure Static Web App, we are able reference both the database and the Application Insights instance. Here's a snippet of the Bicep template that acquires the references:
resource databaseAccount 'Microsoft.DocumentDB/databaseAccounts@2023-04-15' existing = {
name: cosmosDbAccountName
}
resource appInsightsResource 'Microsoft.Insights/components@2020-02-02' existing = {
name: appInsightsName
}
With those references in hand, we can now configure the Azure Static Web App with the connection string and access key. Here's a snippet of the Bicep template that configures the Azure Static Web App with the connection string and access key:
// deploy the Azure Static Web App
resource staticWebApp 'Microsoft.Web/staticSites@2022-09-01' = {
name: staticWebAppName
location: location
tags: tagsWithHiddenLinks
sku: {
name: 'Free'
tier: 'Free'
}
properties: {
repositoryUrl: 'https://github.com/johnnyreilly/blog.johnnyreilly.com'
repositoryToken: repositoryToken
branch: branch
provider: 'GitHub'
stagingEnvironmentPolicy: 'Enabled'
allowConfigFileUpdates: true
buildProperties:{
skipGithubActionWorkflowGeneration: true
}
}
}
// configure the Azure Static Web App with the connection string and access key
resource staticWebAppAppSettings 'Microsoft.Web/staticSites/config@2022-09-01' = {
name: 'appsettings'
kind: 'config'
parent: staticWebApp
properties: {
APPINSIGHTS_INSTRUMENTATIONKEY: appInsightsResource.properties.InstrumentationKey
APPLICATIONINSIGHTS_CONNECTION_STRING: appInsightsResource.properties.ConnectionString // <-- connection string
COSMOS_ENDPOINT: databaseAccount.properties.documentEndpoint
COSMOS_KEY: databaseAccount.listKeys().primaryMasterKey // <-- access key
}
}
I've slightly tweaked the code to make it more readable, if you'd like to see the full configuration of the Azure Static Web App in the source of my blog, you can find it here.
You can see the effect of this configuration in the Azure Portal. Here's a screenshot of the configured environment variables of the Azure Static Web App:
What's hopefully apparent from the previous section is that in the end this amounts to injecting a string to the appropriate place in the configuration of the resource. This is true for Azure Container Apps as well. Here's a snippet of the Bicep template that configures an Azure Container App with a connection string and access key:
var appInsightsInstrumentationRef = 'app-insights-instrumentation-key'
var appInsightsConnectionStringRef = 'app-insights-connection-string'
var cosmosKeyRef = 'cosmos-key'
resource webServiceContainerApp 'Microsoft.App/containerApps@2023-05-01' = {
name: webServiceContainerAppName
tags: tags
location: location
properties: {
// ...
configuration: {
secrets: [
{
name: appInsightsInstrumentationRef
value: appInsightsResource.properties.InstrumentationKey
}
{
name: appInsightsConnectionStringRef
value: appInsightsResource.properties.ConnectionString // <-- connection string
}
{
name: cosmosKeyRef
value: databaseAccount.listKeys().primaryMasterKey // <-- access key
}
// ...
]
// ...
}
template: {
containers: [
{
// ...
env: [
{
name: 'APPINSIGHTS_INSTRUMENTATIONKEY'
secretRef: appInsightsInstrumentationRef
}
{
name: 'APPLICATIONINSIGHTS_CONNECTION_STRING'
secretRef: appInsightsConnectionStringRef
}
{
name: 'COSMOS_ENDPOINT'
value: databaseAccount.properties.documentEndpoint
}
{
name: 'COSMOS_KEY'
secretRef: cosmosKeyRef
}
// ...
]
}
]
// ...
}
}
}
The mechanism is slightly different, as befits the different service being used, but the principle is the same. We're injecting the connection string and access key into the configuration of the resource.
In this post we've demonstrated how to deploy resources, acquire reference to them and safely configure Azure Static Web Apps and Azure Container Apps such that they can access the resources.
The pattern we've used here is generally applicable in the Azure world. The same technique can be used to configure Azure Functions, Azure KeyVault, and many other Azure resources. The key is to understand the configuration of the resource you're working with and to understand how to inject the relevant secrets into that configuration.
]]>noUnusedLocals
and noUnusedParameters
settings. I like to avoid leaving unused variables in my code; these compiler options help me do that.
I use ESLint alongside TypeScript. The no-unused-vars
rule provides similar functionality to TypeScripts noUnusedLocals
and noUnusedParameters
settings, but has more power and more flexibility. For instance, no-unused-vars
can catch unused error variables; TypeScript's noUnusedLocals
and noUnusedParameters
cannot.
One thing that I missed when switching to the ESLint option is that, with noUnusedLocals
and noUnusedParameters
, you can simply ignore unused variables by prefixing a variable with the _
character. That's right, sometimes I want to declare a variable that I know I'm not going to use, and I want to do that without getting shouted at by the linter.
It turns out you can get ESLint to respect the TypeScript default of ignoring variables prefixed with _
; it's just not the default configuration for no-unused-vars
. But with a little configuration we can have it. This post is a quick guide to how to implement that configuration.
There are various scenarios when I want to ignore unused variables. Here are a few:
Not everyone will agree with these reasons, but they work for me in certain situations.
Just to offer the counterpoint, let me quote Brad Zacher who works on TypeScript ESLint:
On the one hand it is nice to have a short-hand to ignore things.
On the other hand it is terrible having a short-hand to ignore things - it's a single character that's easy to miss in code review - so it's easy to sneak into a commit.
For example I recently reviewed a PR where someone innocently did something like
import { promisify } from 'node:util';
import { exec as _exec } from 'node:child_process';
const exec = promisify(_exec);And they didn't realise that doing this would define a variable that would never get flagged if it's unused! Really bad!
Brad has a valid point, but let's say you've decided to --ignore-pattern 'brad'
, and want to make use of the _
prefix anyway. (Sorry Brad!) Here's how you can do it.
I mentioned that I like to use the TypeScript noUnusedLocals
and noUnusedParameters
settings. Here's how they would be configured in a tsconfig.json
:
{
"compilerOptions": {
"noUnusedLocals": true,
"noUnusedParameters": true
}
}
Given we're moving to ESLint so we'll explicitly turn these off in our tsconfig.json
so we can use ESLint to do the same job:
{
"compilerOptions": {
"noUnusedLocals": false,
"noUnusedParameters": false
}
}
With those off in TypeScript, we can now configure ESLint to respect the _
prefix. Here's how you can do that in your .eslintrc.json
:
{
"rules": {
"@typescript-eslint/no-unused-vars": [
"error",
{
"args": "all",
"argsIgnorePattern": "^_",
"caughtErrors": "all",
"caughtErrorsIgnorePattern": "^_",
"destructuredArrayIgnorePattern": "^_",
"varsIgnorePattern": "^_",
"ignoreRestSiblings": true
}
]
}
}
The argsIgnorePattern
, caughtErrorsIgnorePattern
, destructuredArrayIgnorePattern
, and varsIgnorePattern
settings are the ones that respect the _
prefix. You have to set them all to ^_
to make it work. ^_
is a regular expression that matches any string that starts with an underscore. So if you actually had a different convention for ignoring variables, you could change this to match your convention.
Incidentally, you have to explicitly set args
to "all"
and caughtErrors
to "all"
to make the argsIgnorePattern
/caughtErrorsIgnorePattern
settings work. If you don't, the settings are ignored.
There's an ignoreRestSiblings
setting specified above that we'll get to in a minute. First of all, let's see how the linting we've activated works in practice. Here's some code that demonstrates the settings in action:
export function demoTheProblems(
unusedAndReportedArg: boolean,
_unusedButIgnoredArg: boolean, // argsIgnorePattern
someArray: string[],
) {
try {
const unusedAndReportedVar = true;
const _unusedAndButIgnoredVar = false; // varsIgnorePattern
const [
unusedAndReportedDestructuredArray,
_unusedButIgnoredDestructuredArray, // destructuredArrayIgnorePattern
] = someArray;
// caughtErrors
} catch (unusedAndReportedErr) {
// ...
}
try {
// caughtErrorsIgnorePattern
} catch (_unusedButIgnoredErr) {
// ...
}
}
In this code, the unusedAndReportedArg
, unusedAndReportedVar
, unusedAndReportedDestructuredArray
, and unusedAndReportedErr
variables are all reported as unused. ESLint considers them errors and shouts about them.
By contrast, the _unusedButIgnoredArg
, _unusedAndButIgnoredVar
, _unusedButIgnoredDestructuredArray
, and _unusedButIgnoredErr
variables are all ignored, because they are prefixed with an underscore. ESLint notices them but lets them past.
If we run ESLint on this code, we get the following output:
2:3 error 'unusedAndReportedArg' is defined but never used. Allowed unused args must match /^_/u @typescript-eslint/no-unused-vars
7:11 error 'unusedAndReportedVar' is assigned a value but never used. Allowed unused vars must match /^_/u @typescript-eslint/no-unused-vars
11:7 error 'unusedAndReportedDestructuredArray' is assigned a value but never used. Allowed unused elements of array destructuring patterns must match /^_/u @typescript-eslint/no-unused-vars
15:12 error 'unusedAndReportedErr' is defined but never used. Allowed unused args must match /^_/u @typescript-eslint/no-unused-vars
Perfect! This is exactly what we wanted. You can see this in action in the playground.
ignoreRestSiblings
settingThe ignoreRestSiblings
setting is also useful. You may find the need to use the rest operator in a destructuring assignment to omit properties from an object and hold onto the rest. Here's an example:
const { formattedDate, date, ...totals } = payload;
In this case I don't want to use formattedDate
or date
but I do want to use totals
. I can use the ignoreRestSiblings
setting to ignore the unused variables without even needing a _
prefix or similar. So I do.
I hope this post has been helpful. I've shown you how to configure ESLint to respect the TypeScript default of ignoring variables prefixed with _
.
Many thanks to Brad Zacher for his input on this post. You can read our discussion on the TypeScript ESLint GitHub repo here.
]]>Loving snapshot testing as I do, I want to show you how to write high quality and low effort log assertions using snapshot testing. The behaviour of logging code is really important; it's this that we tend to rely upon when debugging production issues. But how do you test logging code? Well, you could write a bunch of assertions that check how your logger is used. But that's a lot of work, it's not super readable and it's not fun. (Always remember: if it's not fun, you're doing it wrong.)
Instead, we'll achieve this using snapshot testing.
I've written previously about manually implementing snapshot testing with .NET. That was great, but I subsequently moved to use the excellent Snapshooter instead. In this post we'll use that. I'm using XUnit as my test framework; but Snapshooter also supports MSTest and NUnit if they are your preference.
ILogger
?Before we get into the details of how to test logging code, let's first consider how we might test a method that uses ILogger
. Here's a simple example:
using Microsoft.Extensions.Logging;
namespace MyApp.Tests.Services;
public class GreetingService(ILogger<GreetingService> log)
{
private readonly ILogger<GreetingService> _log = log;
public string GetGreeting(string name)
{
_log.LogInformation($"Greeting {{{nameof(name)}}}", name);
return $"Hello, {name}!";
}
}
If we look at the class above, we can see it has a dependency on ILogger<GreetingService>
. This is a common pattern in .NET applications. The ILogger
interface is used to write log messages. I wouldn't be surprised if it's the most commonly used interface in .NET applications.
If we execute the GetGreeting
method above, we'll both get a greeting returned and a log message will be written. In order that we can test this method, we need to be able to verify that the log message was written correctly. We're going to do that by making use of a fake logger.
FakeLogger
As of .NET 8, there is a FakeLogger
that ships as part of .NET. You can read more about that here: https://devblogs.microsoft.com/dotnet/fake-it-til-you-make-it-to-production/#logging-fake
We're going to make use of the official FakeLogger
in this post. However, I'm mindful that not everyone is on .NET 8 yet. So, I'm going to show you how to implement a fake logger yourself. So if you're on .NET 6 / .NET 7 then you can use this.
Whichever fake logger we use, it is a simple implementation of ILogger<T>
. It's a fake because it doesn't actually write anything to a log. Instead, it records the log messages it's asked to write in a list of FakeLogRecord
. We can then use this list to verify that the log messages were written correctly.
FakeLogger
for .NET 8+If you're working with .NET 8 or later, you can use the FakeLogger
that ships with .NET. To add this to your project, add the Microsoft.Extensions.Logging.Testing
and Microsoft.Extensions.TimeProvider.Testing
packages to your test project:
dotnet add package Microsoft.Extensions.Diagnostics.Testing
dotnet add package Microsoft.Extensions.TimeProvider.Testing
FakeLogger
for earlier .NET versionsIf you're working with an earlier version of .NET, you can implement a fake logger yourself:
using Microsoft.Extensions.Logging;
namespace MyApp.Tests.TestUtilities;
public record FakeLogRecord(LogLevel Level, string Message, Exception? Exception);
public class FakeLogger<T> : ILogger<T>
{
public IReadOnlyList<FakeLogRecord> GetSnapshot() => _records;
readonly List<FakeLogRecord> _records = [];
public IDisposable? BeginScope<TState>(TState state) where TState : notnull => NullScope.Instance;
public bool IsEnabled(LogLevel logLevel) => true;
public void Log<TState>(LogLevel logLevel, EventId eventId, TState state, Exception? exception, Func<TState, Exception?, string> formatter) =>
_records.Add(new FakeLogRecord(logLevel, formatter(state, exception), exception));
/// <summary>
/// Reference: https://github.com/aspnet/Logging/blob/master/src/Microsoft.Extensions.Logging.Abstractions/Internal/NullScope.cs
/// </summary>
sealed class NullScope : IDisposable
{
public static NullScope Instance { get; } = new NullScope();
private NullScope()
{
}
public void Dispose()
{
}
}
}
This implementation is inspired by both the .NET 8 FakeLogger
implementation and David Nguyen's post. It's not identical to the official implementation, but it's close enough for our purposes.
ILogger
with SnapshooterNow that we have a fake logger, we can use it to test the logging caused by calling our GetGreeting
method.
ILogger
with Snapshooter for .NET 8+Here's how we might do that with our .NET 8 FakeLogger
:
using Microsoft.Extensions.Logging.Testing;
using Microsoft.Extensions.Options;
using Microsoft.Extensions.Time.Testing;
using Snapshooter.Xunit;
namespace MyApp.Tests.Services;
public class GreetingServiceTests
{
[Fact]
public void GetGreeting_greets_and_logs()
{
// Arrange
var fixedTimeLogCollector = new FakeLogCollector(Options.Create(new FakeLogCollectorOptions {
TimeProvider = new FakeTimeProvider()
}));
var log = new FakeLogger<GreetingService>(fixedTimeLogCollector);
var greetingService = new GreetingService(log);
// Act
var greeting = greetingService.GetGreeting("John");
// Assert
Snapshot.Match(new { log = log.Collector.GetSnapshot(), greeting });
}
}
Here, we create an instance of FakeLogger<GreetingService>
. That FakeLogger
is instantiated with an instance of FakeLogCollector
. We're doing this because we want to stub out the time provider. If we don't do this, the log messages will contain the current time. That would will make our snapshot tests brittle. By stubbing out the time provider, we can ensure that the log messages will always contain the same time. This will make our snapshot tests more robust.
If you'd like to learn more about stubbing out the time provider, I recommend you read Andrew Lock's post on the subject.
Our FakeLogger
is passed into the GreetingService
constructor, ready to be used. We call GetGreeting
, and then we use Snapshooter to verify that the log messages were written correctly and that the greeting generated is what we expect. The log messages are acquired by calling the GetSnapshot
method of the FakeLogCollector
.
When the test is first run, a GreetingServiceTests.GetGreeting_greets_and_logs.snap
snapshot is created. This snapshot contains the serialised log
and greeting
objects. Subsequent runs of the test will compare the current output with the snapshot. If the output matches the snapshot, the test passes. If it doesn't, the test fails. Here is what the contents of the snapshot should look like:
{
"log": [
{
"Level": "Information",
"Id": {
"Id": 0,
"Name": null
},
"State": [
{
"Key": "name",
"Value": "John"
},
{
"Key": "{OriginalFormat}",
"Value": "Greeting {name}"
}
],
"StructuredState": [
{
"Key": "name",
"Value": "John"
},
{
"Key": "{OriginalFormat}",
"Value": "Greeting {name}"
}
],
"Exception": null,
"Message": "Greeting John",
"Scopes": [],
"Category": "MyApp.Tests.Services.GreetingService",
"LevelEnabled": true,
"Timestamp": "2000-01-01T00:00:00+00:00"
}
],
"greeting": "Hello, John!"
}
And that's it, we're done! We've tested our logging code with minimal effort; the only assertion we wrote was Snapshot.Match
. If we change behaviour of the GetGreeting
method, the test will fail. To remedy we can then update the snapshot and we're good to go.
ILogger
with Snapshooter for earlier .NET versionsIf we were using our own FakeLogger
implementation, we'd almost the same thing:
using Snapshooter.Xunit;
namespace MyApp.Tests.Services;
public class GreetingServiceTests
{
[Fact]
public void GetGreeting_greets_and_logs()
{
// Arrange
var log = new FakeLogger<GreetingService>();
var greetingService = new GreetingService(log);
// Act
var greeting = greetingService.GetGreeting("John");
// Assert
Snapshot.Match(new { log = log.GetSnapshot(), greeting });
}
}
You'll notice that actually less code is involved this time. That's because we don't need to stub out the time provider. Our simple implementation of FakeLogger
doesn't bother with time. So we can just call GetSnapshot
on the FakeLogger
instance.
The snapshot generated by this test will look like this:
{
"log": [
{
"Level": "Information",
"Message": "Greeting John",
"Exception": null
}
],
"greeting": "Hello, John!"
}
There's a lot less information in this snapshot. That's because our simple implementation of FakeLogger
doesn't bother with all of the things the official FakeLogger
does. But for what we're doing here, it's enough.
In this post we've seen how to use Snapshooter to test logging code.
This approach is easy to implement, easy to maintain and easy to read. Significantly: it involves very little work on our part. If you're not already using snapshot testing, I hope this post has inspired you to give it a try. If you are already using snapshot testing, I hope this post has inspired you to use it to test your logging code.
]]>Specifically, how can we validate structured data in the context of a GitHub workflow? I've created a GitHub Action called Schemar that facilitates just that. In this post we'll see how to use it.
If you'd like to read more about structured data, you might like to read these posts:
Schemar is a GitHub Action that validates structured data. It's a wrapper around the Schema Markup Validator tool.
If you haven't heard of Schema.orgs validator; it originally started at Google as the Structured Data Testing Tool but was repurposed and gifted to the community.
That tool is a website; Schemar is a wrapper around the tool that makes it easy to validate structured data in the context of a GitHub workflow. Let's imagine it's very important to you that your structured data is both present and valid. You could use Schemar to validate your structured data as part of your CI/CD pipeline.
Imagine Schemar to be the structured data equivalent of the lighthouse-ci-action
GitHub Action.
I'm going to take my blog (that's what you're reading right now BTW) and use Schemar to validate the structured data on it. I already have a GitHub Action that builds and deploys my blog to a staging environment in Azure Static Web Apps and validates it with Lighthouse. So I'm going to add Schemar to that.
But before we do that, let's look at simple usage of Schemar. If you were to add a .github/workflows/schemar.yml
file to your repo with the following contents:
jobs:
release:
runs-on: ubuntu-latest
steps:
- uses: johnnyreilly/schemar@v0.1.1
with:
urls: https://johnnyreilly.com
name: Validate structured data
on:
pull_request: ~
push:
branches:
- main
Then you'd have a GitHub workflow that would validate the structured data on https://johnnyreilly.com
and fail if it wasn't valid.
The urls
input of Schemar is a list of URLs to validate. In this case, we're just validating only one. The results look like this:
Validating https://johnnyreilly.com for structured data...
https://johnnyreilly.com has structured data of these types:
- Organization / Brand
- WebSite
- Blog
For more details see https://validator.schema.org/#url=https%3A%2F%2Fjohnnyreilly.com
We can see that the home page of my blog has structured data of the types Organization / Brand
, WebSite
and Blog
. And we can even click into the Schema Markup Validator to see the details.
If at some point I were to omit or break the structured data on my blog, then Schemar would fail the build. This is a great way to ensure that your structured data is always present and valid.
We're going to see what usage looks like in a minute, as we dive into a more sophisticated example.
Now that we've seen a basic example, let's see what it looks like to use Schemar in a more sophisticated way. We're going to add Schemar to run against my blogs pull request previews, in the same way we're already running Lighthouse against them.
I won't reiterate the whole GitHub workflow that spins up a preview environment here, but I'll show the key parts. You can see the whole thing in the build-and-deploy-static-web-app.yml
of the blog repo. You'll note I'm using Azure Static Web Apps to host my blog - but any web platform will do.
Here is the key part of the GitHub workflow:
structured_data_report_job:
name: Structured data report 📝
needs: build_and_deploy_swa_job
if: github.event_name == 'pull_request' && github.event.action != 'closed'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Wait for preview ${{ needs.build_and_deploy_swa_job.outputs.preview-url }} ⌚
id: static_web_app_wait_for_preview
uses: nev7n/wait_for_response@v1
with:
url: '${{ needs.build_and_deploy_swa_job.outputs.preview-url }}'
responseCode: 200
timeout: 600000
interval: 1000
- name: Audit URLs for structured data 🧐
id: structured_data_audit
uses: johnnyreilly/schemar@v0.1.1
with:
urls: |
${{ needs.build_and_deploy_swa_job.outputs.preview-url }}
${{ needs.build_and_deploy_swa_job.outputs.preview-url }}/about
${{ needs.build_and_deploy_swa_job.outputs.preview-url }}/blog
${{ needs.build_and_deploy_swa_job.outputs.preview-url }}/definitely-typed-the-movie
- name: Format structured data results
id: format_structured_data_results
if: always()
uses: actions/github-script@v7
with:
script: |
const structuredDataCommentMaker = (await import('${{ github.workspace }}/.github/workflows/structuredDataCommentMaker.mjs')).default;
const results = ${{ steps.structured_data_audit.outputs.results }};
core.setOutput("comment", structuredDataCommentMaker('${{ needs.build_and_deploy_swa_job.outputs.preview-url }}', results));
- name: Add structured data results as comment ✍️
id: structured_data_comment_to_pr
if: always()
uses: marocchino/sticky-pull-request-comment@v2
with:
number: ${{ github.event.pull_request.number }}
header: structured_data
message: ${{ steps.format_structured_data_results.outputs.comment }}
Along with the following structuredDataCommentMaker.mjs
script:
// @ts-check
/**
* @typedef {Object} Result
* @prop {string} url
* @prop {ProcessedValidationResult} processedValidationResult
*/
/**
* @typedef {Object} ProcessedValidationResult
* @prop {boolean} success
* @prop {string} resultText
*/
/**
* @param {string} baseUrl
* @param {Result[]} results
*/
function createStructuredDataReport(baseUrl, results) {
const comment = `### 📝 Structured data report
${results
.map((result) => {
const shortUrl = result.url.replace(baseUrl, '') || '/';
return `#### ${
result.processedValidationResult.success ? '🟢' : '🔴'
} [${shortUrl}](${result.url})
${result.processedValidationResult.resultText}`;
})
.join('\n')}
`;
return comment;
}
export default createStructuredDataReport;
Let's break this down:
nev7n/wait_for_response
GitHub Action to wait for the preview to be available. This is because the preview URL is not available immediately after the preview is created.structuredDataCommentMaker.mjs
script.marocchino/sticky-pull-request-comment
GitHub Action.Let's see what this looks like in action. I've created a pull request that breaks the structured data from my blog. This is what the pull request looks like:
-'@type': 'Person',
+'@type': 'Blarg', // let's break the schema!
The question is, what does the pull request look like after the GitHub Action has run? Here's the answer:
It failed! And it put a comment on the PR that looks like this:
Let's unbreak the structured data and see what happens:
-'@type': 'Blarg', // let's break the schema!
+'@type': 'Person',
It succeeded! And it put a comment on the PR that looks like this:
This is great! It means that I can be confident that my structured data is always present and valid. And if it isn't, then I'll know about it. I can even click through to the Schema Markup Validator to see the details.
My hope is that Schemar can be used to increase the quality of structured data on the web. I'm using it to increase the quality of structured data on my blog. I hope you'll find it useful too.
I've also shared this with the good folk of Schema.org in the hope they'll find it useful too. The source code of Schemar can be found here.
]]>