Published by

Don’t leave DNS records lying around

I just had my first experience with a subdomain takeover due to a dangling DNS record. It’s been lovely. /s

This note outlines how I came across it, what I had to do to fix it, and what I’ve learned from it.

How I found out

Last weekend I got a notification from Google Search Console that a subdomain associated with a domain I own had been added as a new property. I got a quick followup notification that someone with a totally random Gmail address had verified themselves as an “owner” of this property. Alarming.

I visited the subdomain and saw that it had a junky 404 page. I then did a site search on Google for the subdomain and saw that they’d added a *lot* of garbage PDFs, seemingly for shady SEO purposes or maybe for spreading malware.

Sorting it out

The first thing I did was add the property to Google Search Console so that I could manage the users. At first I made the mistake of adding only the HTTP site, whereas this person had verified the HTTPS version of the site. Once I added the HTTPS version, I checked the list of users and saw this new “owner”. They’d verified themselves via HTML, meaning they’d uploaded a file to the server that the subdomain was pointed at.

Next, I logged in to the DNS host (Netlify). I wrote down the contents of the record first for reference, then deleted the offending DNS record. This took the whole subdomain offline, including the file that had been used for verification in Google Search Console. I then went back in to Google Search Console and removed the user.

Now I needed to sort out the indexing for this site. Search bots would eventually remove all pages associated with this subdomain since the pages no longer exist, but I wanted to get rid of them as fast as possible. I started individually removing URLs using the tool in Google Search Console, but this was tedious considering the number of URLs and would do nothing to sort things out on other search engines such as Duck Duck Go. I dropped that effort and instead made a new site at this subdomain that returns a 410 error for all pages. This indicates that the resource has been permanently, purposefully, removed. I then visited a bunch of these pages directly from Google’s search results hoping to expedite the reindexing. Note that I have no idea if this is effective, but I feel like it might be since so much information is passed to Google via URL parameters. This technique seems to work, but it takes a while. It took me longer since I had a robots.txt file blocking crawlers from re-indexing the site. It’s counterintuitive (ideally the site wouldn’t be indexed at all), but it’s necessary so that the crawlers can receive the 410 error.

The cause of it all

I was initially in the dark about how this had all happened. It had been such a long time since I’d worked with that subdomain, and I’ve never encountered a subdomain takeover before.

My biggest fear was that the login details for my DNS host had been compromised, but it quickly became obvious that this is unlikely. I use two-factor authentication for my most important accounts and use very long, unique, and usually unreadable passwords for all services (thumbs-up for password managers). As a precaution though, I changed the passwords for related services and checked the security logs offered by each service. Everything looked A-OK.

So I still didn’t understand why the DNS record was there in the first place, especially since it seemed likely that I had added it myself.

At first I thought it had been pointed at GitHub Pages. I’ve hosted sites there in the past, and I’ve heard of this happening to GitHub Pages websites (see step 2 of the documentation on setting up custom GitHub Pages domains). I sent an email to the GitHub support staff to ask if the IP address was one of theirs, and they quickly replied to let me know that it was not. I then looked up the IP (should have probably done this before getting in touch with GitHub) and found out it was a Digital Ocean server. This jogged my memory.

When I was first experimenting with Homebase late last year, I was thinking of getting a website set up with Dat + HTTPS. I dipped my toe in the water, setting up a DigitalOcean droplet and adding a DNS record pointing at DigitalOcean. Ultimately, I ended up shelving the project. I didn’t worry too much about tidying up at the time, I thought I might revisit the idea. Regular billing reminders from DO wore me down though, and about a week ago I finally got round to deleting the droplet. At this point, I had completely forgotten about the DNS record.

What I’ve learned

Sorting it out has been a faff, but I’ve learned a lot.

  1. Don’t neglect your DNS records. I thought my DNS hygiene was pretty decent, but clearly I was wrong if I ended up leaving a dangling DNS record for an unrealised project in place. I’m setting a monthly or quarterly reminder to check my most important domains.
  2. Subdomain takeovers are a risk with all sorts of hosting providers, not just GitHub Pages. They can also be much more complex and wide-reaching than my experience was. See this StackExchange thread for a practical example, this post by Patrik Hudak, and this Medium article about a potential takeover of 55,000+ Shopify subdomains.
  3. Register your domains in Google Search Console. This one isn’t a mandatory “always do this” recommendation, but it is worth noting that this is how I found out about the subdomain takeover in the first place. I’m not quite sure how else I would have found out, and I imagine it may have been weeks later. Registering with Google Search Console is probably a very common “first step” for a person executing a subdomain takeover once they’ve got their content online since Google only allows you to submit a domain for indexing via the Console.
  4. Act as quickly as you can. Subdomain takeovers can be used for all sorts of unsavoury activity, and a lot of that activity can get your domain blacklisted in spam filters, search engines, and other lists or services. The longer that the takeover goes unchecked, the bigger the headache you have to sort out.

I feel pretty thankful that I was in a place that allowed me to sort it out quickly, but it was really not the nicest Monday morning I’ve ever had.