Inexactly benchmarking Eleventy vs Astro build times

The Eames Institute team is checking out some frameworks and static site generators for a project, and I wanted to see how Eleventy and Astro compare in terms of build time.

Zach Leatherman’s 2022 article “Which generator builds Markdown the fastest?” is probably the most thorough resource I’ve come across along these lines, and I’d recommend checking that out if you want to do some serious comparisons.

But I was curious about a “real world” test in 2024, so I decided to do some inexact benchmarking using a Markdown export of this blog. A caveat up front: I’m much more familiar with Eleventy than I am with Astro, which will likely be apparent when I get to the incremental build tests later in this post.

For Eleventy I used eleventy-netlify-boilerplate with zero modifications, and for Astro I used the blog template as described in their docs with some small modifications along the lines of this wordpress-to-astro repo to get categories and tags working. I didn’t want to use wordpress-to-astro directly since it was last updated two years ago, but it is a good reference point.

My blog has 770 posts which were exported to 770 Markdown files. With a paginated feed, categories, and tags, the total number of built pages is around 2550.*

Based on an average taken from 10 builds, Astro took 10.07 seconds and Eleventy took 4.29 seconds to build.

Incremental builds can speed things up significantly since the only built content is that which is relevant to the modified files.

Eleventy has supported incremental builds since December 2022 (I believe!), but it doesn’t yet support it on a CI server. There is an open issue for it which looks like it has traction.

To test incremental builds, I added and removed the same single tag on the same post 10 times.** Based on an average taken from 10 incremental builds, Eleventy took 2.17 seconds skipping 777 files. I would have expected it to skip more, but this might have to do with not being able to incrementally build paginated data.

I wanted to test the same content change in Astro… but it isn’t clear to me that there is an apples-to-apples comparison. Astro introduced an experimental Incremental Content Caching feature in v4.0 (not sure if this is supported on CI servers). When I added experimental.contentCollectionCache to the config, there was no difference between basic build times when I made a content change. I’m not sure if this is because having all of my content in Markdown makes the caching a mute point, or if it’s something else. If anyone has further context on how best to test Incremental Content Caching in Astro, would love to know.

For what it’s worth, running astro dev is extremely quick, just 125ms before it’s ready.

I’d be curious to do a similar benchmark using WordPress’s REST API but am not sure I’ll have the time… Will update here if I do.

* I give a rough number because the Eleventy boilerplate and Astro template generate a few additional pages, but the page total difference is in the single digits so I didn’t waste time evening them up perfectly.

** For my own future reference in case I do further tests: Add and remove the tag hello from this post.