Yesterday, my host, DigitalOcean, experienced a sudden and massive DNS outage. Their DNS services are operated by CloudFlare. It put me in a straight panic, believing that I had put my own server out of connection from the rest of the Internet. The previous day I had spent a lot of time tinkering around with the DNS records to setup SPF, DKIM, and DMARC hoping to put a few more soldiers in the field against the regular influx of spam. (So far, trends say that they have).
This post isn’t about the outage, but the response of some towards it.
For the family members who read this and aren’t familiar with DNS, it’s the part of the Internet that turns stevenski.com into 220.127.116.11 in the background and you don’t even know that it’s happening. Your computer asks for DNS results dozens of times an hour, give or take, depending on what your doing with your computer during that time. So, my machine became accessible only by IP address and I had a bit of a freak out. Where I should have started was checking DigitalOcean’s server status page, then going there, but I didn’t, and fumbled through a bunch of stuff on my machine like a bull in a china shop.
Finally, getting to the server status page, I saw that the problem was squarely on DigitalOcean and there wasn’t anything I could do about it.
For me, this isn’t the end of the world. I had to go about my day knowing that things were broken, but it just meant that some people wouldn’t be able to read this and I couldn’t check my email. No big deal.
However, at DigitalOcean’s @DOStatus Twitter page, there were tons of people pledging to abandon DigitalOcean for a competitor because of this outage and “lack of communication to their customers”. First of all, their communication was fine, “Things are broke, we’re in full fledge freak out mode fixing this right now, “ was my translation of their updates. People kept demanding an ETA and they’d answer that it was their number one priority and it would be fixed as soon as they fixed it. That answer was not satisfactory for some people.
Secondly, quite a few of their angry customers claimed that they had business that they were losing or was being damaged by this outage. Probably true. So I started thinking- I have no clue if a domain can be served by multiple DNS. I can’t recall a time that any of the big boys like Google.com, Facebook.com, Yahoo.com, etc. had been put down by a DNS outage, and I’m sure that they have DNS that goes down. Is redundancy an option here? I’m sure it must be.
Five minutes of searching the Internet turned up my answer: yes.
And five more minutes later, I used my domain registrar, hover, to turn on their DNS for me as well (which cost me nothing more than what I was already paying) and stevenski.com started to work as normal while the good people at DigitalOcean had a terrible day and these web professionals continued to lose money and reputation. I mean that second part as sarcastic as possible. My job is not web development, but I figured out how to mitigate this issue within minutes of concluding that there was a solution to this issue that might be in my ability. How are these people not baking DNS redundancy into their web apps as a fundamental part of their process?