“After looking into this, it seems you have a hit song on your site,” the email from Netlify customer support reads. “Maan Bou Jan Sang Lou by Teresa Tang. I was not aware of her, but she seems to be a popular Taiwanese singer. This song is 99% of your bandwidth usage over the past 30 days.”
The letter further explained that a lot of bandwidth was generated from user agents that “are quite ancient using Google Cloud addresses”.
“This would include devices such as circa 2010 iPads, Windows 98 & Windows 6 computers. So either you have a fanbase with a passion for older technology, or this was likely a DDoS attack. To me, this seems to be the latter,” the email continued and suggested hosting such files on third-party platforms, such as YouTube or SoundCloud.
After explaining the standard practice of reducing the bill to 20% after such attacks, which would be $20,900 in this case, the Netlify support team offered a better deal.
“I’ve currently reduced it to about 5%, which is $5,225. I know this is still a lot of money, and I apologize for the inconvenience. If you like, I can raise this internally to see what else can be done.”
The user wasn’t happy with that and decided not to pay but post their story on Reddit and Hacker News instead.
One user on Hacker News with the alias ‘bobfunk’ introduced himself as the Netlify CEO and assured users that the bill would be forgiven. Cybernews was unable to verify the CEO’s identity independently. However, many previous posts from the same user and his bio support the claim of him being Matt Biilmann, the founder of Netlify.
In another twist, the DDoS attack version of the story is being ruled out
“Since the user opened a ticket with us this past Sunday, we’ve been actively researching this situation. Initially, we thought it might have resulted from a DDoS attack, which we stated in our first response. After some investigating, it looks as though the spike in traffic was not caused by a DDoS after all,” Dorian Kendal, CMO at Netlify, told Cybernews.
Instead, now they believe that this was a sustained download event of an mp3 file over a stretch of multiple days.
“We’re working directly with the user to better understand what’s happening on their end, so we can uncover what caused the dramatic increase in downloads,” Kendal said.
“We’ve confirmed that the user was notified multiple times about the additional bandwidth that was being consumed on their site, but given their lack of response to these notifications, we believe that we should revisit and improve the messaging and urgency that’s being communicated.”
I’m kind of impressed by the amount of research they did to figure out why this guy’s bill was so high, then immediately offered a resolution, and then immediately offered another avenue if the resolution wasn’t good enough. Shout out to the customer service rep.
I’m kind of impressed by the amount of research they did to figure out why this guy’s bill was so high, then immediately offered a resolution, and then immediately offered another avenue if the resolution wasn’t good enough. Shout out to the customer service rep.