The rocky road to implementing link prefetching in Rails

Web performance matters for many reasons: a better, more inclusive user experience; less waste of resources (your user’s devices will thank you); and an increase in business metrics like conversion, SEO traffic, and even revenue.

It also happens that it is one of my favorite technical topics, and I fell down the rabbit hole of optimizations (and got to talk about it) more than one time.

Yesterday, I found a new itch I had to scratch, and it all started by the release of InstantPage v5 by Alexandre Dieulot.

https://twitter.com/Dieulot/status/1261665346030895105

Instant Page has a very enticing promise: what if, by dropping a single line of code into your app, you could make it feel instant?

InstantPage achieves this by using a technique called link prefetching. Traditionally, a website loads the HTML contents of a new page when the user has clicked on its link. InstantPage takes full advantage of the fact that to click a link, the user has to get there, and usually spends a few hundred milliseconds hovering on it. By triggering the page load on hover, instead of on click, we can shave off those few hundred milliseconds of load time, making the transition to the new page feel instant.

You can see this pattern in action on this very website, dev.to! Feels fast, doesn't it?

So I set up to implement that in our Ruby on Rails application, and boy was it a wild ride. Buckle up!

Note: Although my engineering background is heavily leaning on JavaScript, I joined the fine folks at Orbit a few weeks ago and this is my first experience with Ruby on Rails. So please, if I made a mistake somewhere, or missed an opportunity for a more idiomatic solution, let me know in the comments! Consider this my first attempt to #LearnInPublic.

The naive approach: using InstantPage itself

So, InstantPage says that a single line of code can make this work. Well… in our case, it didn’t. I could see the prefetching happen in the DevTools, but clicking on a link resulted in the same experience as before.

It turns out that InstantPage and Turbolinks (Rails integrated library to make navigation faster and Single Page App-like) do not pair well together:

I’ve talked privately with @dhh and he said he’s interested in bringing the just-in-time preloading mechanism into Turbolinks. I also plan to make an alternative to Turbolinks that uses them (in fact I already did so with InstantClick, but it lacks good documentation and a bunch of other things, I plan to reboot it). So maybe I won’t make instant.page compatible with Turbolinks in the next version, it will depend on ease of implementation, we shall see.

(source)

Damn. Well, maybe Turbolinks already solved that problem and I don’t even need InstantPage?

A prefetching solution based on Turbolinks

A quick search in the Turbolinks repository issues showed that I was not the first one to want link prefetching: https://github.com/turbolinks/turbolinks/issues/313

Reading through that 50+ comments discussion (!), I found that GitHub user hopsoft helpfully shared a gist implementing a link prefetching strategy leaning on Turbolinks cache.

We’re making progress! I can see the prefetch request going out as I hover a link, and the navigation after I click feels faster.

However… I was not 100% convinced by this approach. Leveraging Turbolinks cache meant relying on Turbolinks preview behavior: if a page is warm in the cache, then Turbolinks will show that cached version as a preview (a static, non-interactive version) and then trigger a new request, using its results to really update the page.

With that in mind, this solution had the drawback of making two requests:

  • One on hover, which would warm Turbolinks cache;
  • One on click, which would be displayed after “flashing” the cached version.

This seemed a bit wasteful, as there was a very small chance that those two requests would differ—they were triggered a few hundred milliseconds apart after all.

Back to the drawing board!

Getting closer with InstantClick, InstantPage’s predecessor

In the GitHub comment highlighted previously, Alexandre piqued my curiosity:

I also plan to make an alternative to Turbolinks that uses them (in fact I already did so with InstantClick, but it lacks good documentation and a bunch of other things, I plan to reboot it).

Undeterred by the lack of documentation (I can’t decide if that’s brave or just plain dumb), I set out to try InstantClick and see if it could solve that duplicate request issue.

The InstantClick repository is pretty straightforward: an instantclick.js file that implements link prefetching, and a loading-indicator.js file that takes care of showing a fake loading indicator if a page load takes too long. Fake because it doesn’t reflect the real progress of the page, it just goes forward until the page finishes loading. This is a technique used by GitHub and dev.to that is easy to set up and is good enough for the vast majority of use cases, so that’ll do!

I copied and pasted both those files, and after a bit of Rails plumbing (I had to remove Turbolinks as it was clashing with InstantClick) and fixing some problems with React and forms (more on that later), it was all set up.

And… it worked!

Prefetching happened on hover, and no extra request went off on click. The app felt much faster, with most page transitions appearing instant. Happy times!

However, I noticed something off: hovering the same link multiple times triggered multiple requests. Again, this seemed a bit wasteful! I was curious about whether dev.to showed this behavior (they use a custom implementation of InstantClick). I fired up the Dev Tools on this very website, and lo and behold, it didn’t. Meaning that they found a way to fix it.

How? Well, let’s find out!

The beauty of Open Source: diving into dev.to’s codebase

The dev.to codebase is open source (which, by the way, is awesome), which meant that the solution to my problem was somewhere in the repository.

A quick GitHub in-repository search for InstantClick led me directly to their custom implementation which, to my surprise, was quite heavily integrated with their codebase. So copy-pasting the whole file wasn’t an option, and I had to put on my detective hat and figure out what is going on.

I knew I was looking at some kind of cache pattern, so I tried and find the method that was responsible for making HTTP requests—I figured that that method would check whether the results were already in said cache.

That was a hit!

The dev.to folks added a variable $fetchedBodies to InstantClick code that would save the URL, title, and body of any preload, which would then be available as a cache for subsequent requests.

Here is a simplified representation of that mechanism:

var $fetchedBodies = {}

function processXHR(xhr, url) {
  // Makes the XHR call, the response body and title are available
  // Use that response to add a new cache entry
  $fetchedBodies[url] = {body:body, title:title};
}

function preload(url) {
  // Responsible for preloading URLs
  // If the URL is already in the cache, then do not make the request
  if (!$fetchedBodies[url]){
    $url = url
    $xhr.open('GET', internalUrl)
    $xhr.send()
  }
}

function removeExpiredKeys(option) {
  // Handles the cache expiration
  if ( Object.keys($fetchedBodies).length > 13 || option == "force" ) {
    $fetchedBodies = {};
  }
}

Once I felt I had sufficiently grasped how it worked on dev.to, I ported it to our codebase, nearly as is. The only difference is in the cache expiration mechanism, where we forced an expiration with removeExpiredKeys("force") after each navigation to make sure that users do not see stale versions of a page.

Hurrah! No more multiple requests if a user hovers the same link multiple times. We just got ourselves a working, optimized, and mobile-friendly link prefetching implementation in Rails.

Getting it to work with React and interactive navigations

As mentioned previously, we had a bit more work to do to make InstantClick work with our existing app. In case it might help anyone else, I’m going to go over those bumps in the road and the fix we found.

First, it appeared that our React components were broken after navigating a link. According to this issue, React Rails do not automatically mount components when using prefetched links and we had to do that ourselves by calling ReactRailsUJS.mountComponents() in the JS initialization step.

Second, after the initial move to InstantClick some of table filtering/searching features stopped working, because they relied on programmatically tell Turbolinks to visit a URL with the proper query params:

<select onchange="if(event.target.value) Turbolinks.visit(event.target.value);")>…</select>

The InstantClick source code does not provide a method to navigate to a given URL. Luckily, this GitHub comment offered a clever solution: have JavaScript create a new link in the DOM with that URL and click on it.

InstantClick.go = function(url) {
  var link = document.createElement('a');
  link.href = url;
  document.body.appendChild(link);
  link.click();
}

We can now change the previous HTML code into this:

<select onchange="if(event.target.value) InstantClick.go(event.target.value);")>…</select>

Unfortunately… this crashes the JS of the page: the console tells us that InstantClick is undefined. The workaround to that came from Stack Overflow, and required some Webpack black magic to make InstantClick available as a global variable:

// in webpack.environment.js

// run yarn add --dev expose-loader exports-loader beforehand

const { environment } = require("@rails/webpacker");

const webpack = require("webpack");

environment.loaders.append("InstantClick", {
  test: /instantclick/,
  use: [
    {
      loader: "expose-loader",
      options: "InstantClick",
    },
    {
      loader: "exports-loader",
      options: "InstantClick",
    },
  ],
});

module.exports = environment;

It took many different helpful answers from around the internet, but all our problems are now solved!

Going further

Our custom InstantClick setup is available as a gist, feel free to use it!

We are pretty happy with our implementation for now, but it is important to point out that it can be improved even further:

  • Featured in the tweet that sparked all this (but absent from our implementation), the new release of InstantPage uses a clever trick: trigger the click event on mousedown, instead of the usual mousedown then mouseup. While this is promising in terms of perceived performance, I’m curious to hear about people’s reaction to this change in such a foundational experience as a click;
  • Our implementation does not respect the Save Data header, which might be an issue for users looking to reduce their bandwidth consumption (e.g. when traveling abroad);
  • Guess.js is a library that takes this whole idea of link prefetching one step further: using your analytics data and machine learning, it prefetches links the user is most likely to click on next. Ain’t that amazing?

Thanks for reading!