Link shorteners are superbly useful, but really bad for the longevity of the world wide web. Every time you use tinyurl.com, bit.ly, or similar you are taking something open and direct (the web address of the think you're pointing people to) and hiding it within the black box that is the company behind that site.
What happens if tinyurl.com closes its doors? Or if bit.ly is bought by a billionaire who charges people to see the long link behind the short one? This is the way the web "rots"; by the removal of the content that holds it together.
However short links are incredibly useful; humans can't remember the long sequences of letters and numbers that some web addresses need to be, so what's to be done?
Ugh. JP. I get it. Let me skip to the cool part.
What is IPFS?
A few days ago the InterPlanetary FileSystem (IPFS) released a new version of their flagship server with an exciting new feature — the ability to perform redirects (geekier folks can read about it here). To explain why this is so valuable, I'll quickly talk about IPFS, and why it's already solving many similar problems.
IPFS is a way of storing & requesting data on the internet by referring to it based on its content, rather than its location.
A web address today looks like
www.byjp.me/posts. This contains the name of the server you should contact to get my site (
www.byjp.me), and where on that server the page is that lists all my posts (
/posts) — it's all location based. If my server dies, then everyone has forever lost access to what's on my site — this is what's known as "link rot".
In the world of IPFS data is accessed, instead, by a Content ID ("CID") — a name that is derrived from, and unique to, the data it points to, rather than where it's located. The way this works is fairly technical, but the same data always produces the same CID, and the CID unquely refers to that data1.
Another of my sites (a 3D model of my brain) has the CID
QmPgEidvA67eUxtSgLQy2N5cWXh8WQ8r7iYQn1ecjrL8Mz. You can see it at ipfs.io, at Cloudflare, dweb.link or any other IPFS server, as I have it stored on an IPFS server connected to the internet.
This means that if Jo Bloggs makes a copy of my site ("pins" that CID), and then my site goes down forever, everyone can still view my site, exactly as it was meant to be, by getting it from Jo Bloggs instead of me. I'm glossing over some details here but, broadly, so long as someone cares about the data in question: link rot defeated.
How it works
So, you may be asking, how does this help with URL shorteners? This new 'redirects' feature of the IPFS server (called kubo) means information on how to redirect people (from a shortlink to a long URL) can be stored in a file on IPFS and used to do redirections.
byjp.fyi is configured to serve content from IPFS, in particular a
_redirects file (see it live here) that holds all of my shortlinks and their destinations. Anyone who pins
/ipns/byjp.fyi on their IPFS server will always be able to know what my shortlinks were pointing to, even if my site disappears.
For convenience's sake, I keep my
_redirects file and other files associated to byjp.fyi on github, with a deploy workflow which pins all the files on Filebase, and updates Cloudflare with the new DNS entry every time I publish a new shortlink.
How do I do this too?
If you too want to run your own (free!) URL shortener that is resilient to link rot then follow the steps below — then get in touch, I'll happily pin your shortened domain locally, so we've got some more protection against link rot.
These instructions assume you're confident with Github, Cloudflare & code, but you shoud be able to make it through even if you're not, and you can always reach out to me if you have questions!
- Set up the prerequisites:
- You'll need a domain you want to host your shortlinks on. I tend to buy mine on Cloudflare (hit "Register domains" in the dashboard)
- You'll need to be using Cloudflare as your Nameserver for this domain to follow these instructions exactly (though other DNS providers that have APIs to update DNS records should work)
- You'll need an account with Filebase, or any other API-capable IPFS pinning service.
- To follow my code exactly you'll need a bucket in Filebase called "microsites".
- A Github account
- Fork my byjp.fyi repo, and rename your fork to be whatever domain you registered.
- I'll refer to this as github.com/yourghname/your.tld from here on. You can calle the repo something different to what your domain is, but you'll need to edit the deploy scripts.
- Add some secrets to your repo (see your repo's Settings > Secrets and variables > Actions):
CLOUDFLARE_ZONE_IDneeds to be the Zone ID for the domain you're using (find it on the side pane of the Overview page for your domain in Cloudflare)
CLOUDFLARE_TOKENneeds to be a Cloudflare token with
Zone.Web3 Hostnames:Editpermissions for the Zone (domain) you're using.
FILEBASE_KEYneeds to be the "Key" from your Filebase access keys page.
FILEBASE_SECRETneeds to be the "Secret" from the same place.
- Create a Web3 Gateway for your domain in Cloudflare:
- There's a tab called "Web3" on the left when you're viewing your domain, click "Create Gateway"
- The "Hostname" should be the same as your repo name, and the domain you've bought
- "Gateway Type" should be "IPFS DNSLink"
- You can leave the DNSLink field empty (we'll be replacing it via the API shortly)
- Nearly there! In Github, manually edit the
public/_redirectsfile in your new repo
- Perhaps change the
/melink to point to your own homepage
- You probably want to remove all my other shortlinks, but keep the bottom one (starting
/*), as it manages shortcodes that don't exist.
- Commit your changes
- Perhaps change the
- A few minutes after you committed you should see a green tick in Github, and your shortlinks site should be active! Visit your domain at a random path to see the 404 page, and at any shortlink you added to see it work.
If you'd also like an API to add a new shortlink, you'll need a val.town account:
Go to my val and fork it: byjp.fyi/new
Create another val with
export const shortlink_domain = 'your.tld'in it, referring to your domain.
And another again with
export const shortlink_repo = 'yourghname/your.tld'in it, referring to your github repo.
You can now send a query like below to have a new shortlink added to your repo!
curl "https://yourvtname-addshortlink.express.val.run/example?to=http://example.com" \ -H 'Authorization: Bearer abc123
- The host refers to your val.town username, so here
yourvtname-addshortlink.express.val.runwould only be accurate if your val.town account name was
- The path refers to the shortlink you want to make, so here
/examplemeans you'd be hitting
toquery parameter is the destination, here that's
http://example.com. It'll fail if you try and redirect to something on
- The Bearer token (
abc123above) needs to be a fine grained Github personal access token (or the organisation equivalent) with at least "Read and Write access to code" for the
- The host refers to your val.town username, so here
Caveats & thoughts
I'd like to build a Raycast extension for making new shortlinks. It's stopped raining now, so maybe later!
If you try to post new shortlinks with the val.town API faster than the deploy time for the repo then your API call will be rejected, and you'll have to try again. I'd like to get some auto-retries in there.
With val town and Raycast being drivable by Typescript I'd like to release this as a package, so it's mega easy to use. Perhaps it'll even have tests 😲
I'd really like to be able to publish this blog with "other things I'd like to be included in the root hash", so I could say "resolve
/ipns/byjp.fyi/ and include that IPFS data here", so this blog always includes a 'backup' of the shortlinks. This feels like it's the tip of a bigger feature IPFS might include around deeper archival support.
Removing shortlinks is still a manual edit of the Github repo away. I don't see much value in allowing that to be automated any time soon.
I do need to trust val.town with write access to my shortlinks repo — if something leaked out of that part of the flow then back folks could hijack links folks may trust are mine isn't great. I should probably check the shortlinks every now and again.
It's not technically unique, there are just more different combinations of these names (Content IDs, or "CIDs") than there are atoms in the universe, so the likelihood of two being the same for different data is vanishingly small. ↩︎