The sinkhole that saved the internet

It was late afternoon on May 12, 2017. Two exhausted security researchers could barely unpack the events of what had just happened.

Marcus Hutchins and Jamie Hankins, who were working from their homes in the U.K. for Los Angeles-based cybersecurity company Kryptos Logic, had just stopped a global cyberattack dead in its tracks. Hours earlier, WannaCry ransomware began to spread like wildfire, encrypting systems and crippling businesses and transport hubs across Europe. It was the first time in a decade a computer worm began attacking computers on a massive scale. The U.K.’s National Health Service (NHS) was one of the biggest organizations hit, forcing doctors to turn patients away and emergency rooms to close.

Hours after the disruption began to break on broadcast news networks, Hutchins — who at the time was only known by his online handle @MalwareTech — became an “accidental hero” for inadvertently stopping the cyberattack by registering a web domain found in the malware’s code.

The internet, still reeling from the damage, had gotten off lightly. The two researchers, at the time both in their early 20s, had saved the internet from a powerful nation-state attack launched by an enemy using hacking tools developed by the West.

But the attack was far from over.

Hutchins and Hankins knew if the kill switch went down, the malware would pick up where it left off, infecting thousands of computers every minute. Puffy eyed and sleep deprived, they knew the domain had to stay up at all costs. The researchers fended off several attacks from an angry operator of a botnet trying to knock the domain offline with junk internet traffic. And, at one point, law enforcement seized two of their servers from a datacenter in France amid confusion that the domain was helping to spread WannaCry and not preventing it.

With the pressure on but running on empty, Hankins — who was also only pseudonymously known as @2sec4u — fought to stay awake, and would fall asleep on his couch where he worked for hours at a time, laptop still open, only to be jolted awake by messages on Slack or Skype, which the researchers used to talk.

Every time he heard an alert, he feared the kill switch had gone offline.

“Being responsible for this thing that’s propping up the NHS? Fucking terrifying,” Hankins told TechCrunch. “The last thing you need is the idea of the entire NHS on fire.”

“It was probably the most stressful thing happen to me,” he said.

‘I think we can stop it’

U.K. news networks began rolling coverage of the cyberattack hours after it began on May 12. Hankins had the television on in the background.

The chyrons reported disruption at several major London hospitals. Staff were locked out of their computer stations, files were encrypted, and their screens were demanding a ransom with a timer ticking down. The NHS had declared a major incident. Telecoms giant Telefonica was also hit, as well as shipping giant FedEx, car maker Renault, Germany’s rail system and several Russian government departments.

British prime minister Theresa May called it an “international cyberattack,” one the government seemed powerless to stop.

WannaCry was spreading from computer to computer, a feature not seen in ransomware before. Blame quickly fell on hacking tools developed by the National Security Agency that had been stolen and published on the web for anyone to use weeks earlier. One such exploit, DoublePulsar, backdoored vulnerable computers, while another, EternalBlue, was used to deliver and spread the ransomware inside a network.

Microsoft released patches for the hacking tools months earlier. The many who had not patched saw their systems go down, one after the other.

“It was just indiscriminately wiping things out,” Hutchins said.

By registering the domain, Hutchins had “sinkholed” the ransomware, allowing him to capture and dispose of malicious internet traffic. It was not unusual for Hutchins to find and register a domain found in a malware sample. As part of his botnet and malware tracking efforts he would often take control of unregistered domains — assuming they were a malware control server — to see how far and fast the malware was spreading. The end goal was to direct the malicious traffic into a void to identify victims and prevent further infections.

With one domain down, Hutchins suspected the malware could jump to another and asked Hankins to look. It’s not uncommon for malware to generate new domains to try to evade detection.

“Holy shit, I think we can stop it,” Hankins responded.

By 6:30 p.m., there was a frenetic discussion in the researchers’ Slack room, trying to understand what the domain Hutchins had registered actually did. But it took the researchers close to an hour to understand the complex but short fragment of the malware’s code that contained the domain Hutchins sinkholed.

“We were very much looking at an if-else statement,” Hankins told TechCrunch, speaking of the stress in the moment. “It was incredibly hard to think because if we fucked this up it would have been worse.”

For a few minutes the researchers panicked, thinking the domain registration was causing the infections. They went back and forth analyzing the code, unsure if they should keep the domain up or not, fearing they were making matters worse. Then the eureka moment hit. The ransomware would only detonate its payload if the domain did not exist.

“If the domain is reachable it won’t infect — I think,” Hankins wrote.

“You are causing me to have the longest anxiety attack ever,” Hutchins replied. “I think I’m gonna be sick.”

Hankins said the stress of the situation made analyzing the code much more difficult. The news played in the background, adding to the constant pressure.

“It took us 45 minutes to look at this code,” he said. “From a reverse-engineering point of view this is not complicated.”

His Fitbit data showed at one point his heart rate was averaging about 140 beats per minute — the equivalent of intense exercise — while he was sitting at his desk.

Data collected from the kill switch showed it prevented the ransomware triggering on about a million infections in just two days. The figure was likely far higher, not including the vast, unknown number of affected computers under a single internet-connected central server. The world had not seen a computer worm spread with such tenacity since the likes of Blaster and Mydoom in the early 2000s.

“I didn’t think it was a big deal until I started seeing the requests and how many organizations were infected,” said Hutchins. He described how “cognitive distance” helped to keep him focused on the issue and not the damage or human cost that was caused by WannaCry.

Hutchins only wanted an insight into the malware campaign. He did not know that registering the domain hours earlier would stop the ransomware from spreading and encrypting.

Hutchins quickly became known as an “accidental hero.”

Under attack

By 7 a.m. the two researchers were back talking on Slack. An hour later, the kill switch was under attack.

Mirai, a powerful botnet made up of hundreds of thousands of hijacked Internet of Things devices and responsible for the “largest ever” distributed denial-of-service attack, began pummeling the kill switch domain with a deluge of junk internet traffic. Months earlier the botnet targeted Dyn, a critical networking company, knocking it offline — and major tech brands reliant on its service — by overloading it with too much internet traffic. In a separate incident the botnet also knocked Liberia offline, a small coastal African nation, by flooding its single undersea fiber cable with internet traffic.

Before WannaCry, Mirai was one of the many botnets under the watch of the researchers. Each time the botnet struck, a dedicated Twitter account would tweet out the target.

It was their turn to be targeted by the botnet.

“We were quite public in tracking Mirai,” said Hankins. “They weren’t fans of us.”

The kill switch held its ground by automatically scaling up the number of Amazon-hosted servers to absorb as much of the traffic as possible. Mirai was hitting the sinkhole hard but the server stayed up.

“We were being hammered,” said Hankins.

Kryptos Logic’s chief executive Salim Neino was in regular contact with the researchers but largely left them to manage the situation themselves. In the late evening, Hankins briefed his boss on the events.

“You’re saying [if] our sinkhole dies those devices get infected?” asked Neino.

“Yes,” Hankins responded.

“Who is watching this?” asked Neino.

“The entire world,” Hankins replied.

“Marcus and I had never dealt with a real-time incident for that long,” Hankins said as he looked back at the Slack messages from the end of the second day after WannaCry hit. “We didn’t have anyone guiding us. You see all these very senior network defenders and companies with all this experience. Meanwhile Marcus landed this very important domain and now we’re at the heart of this global disaster.”

As the internet breathed a sigh of relief thinking the danger was over, most had no idea that any downtime would result in devastating consequences. Even though the ransomware was no longer encrypting files, the now-dormant malware still posed a risk if the kill switch went offline — or if an infected computer or network could no longer communicate with the kill switch. Other attackers were fast to reengineer WannaCry to change the kill switch domain, but other security researchers quickly sinkholed new variants, reducing the spread of the ransomware.

Many thought the researchers were “going to fuck this up,” said Hankins.

Doxxed

After having been awake for more than 30 hours since the attack, Hutchins eventually got some sleep. The next morning, he woke up to find his face plastered on the front of the Sunday editions of the British tabloid newspapers. The media had found him.

Some reporters called Hutchins a “hero,” while others worked unscrupulously to uncover his identity. Given his work uncovering and researching malware and criminal botnets, Hutchins only ever went by his online handle, MalwareTech. Only a trusted few knew both his handle and his name.

Hutchins said he was not expecting a media swarm.

He had not left his room in days, his head down trying to understand more about the scope and impact of the malware. He said reporters came to his house; his parents told him they were camped out on his front lawn.

He did not fear for his safety but was frustrated by the attention. “I’m just unhappy with trying to help clear up Friday’s mess with the doorbell going constantly,” he tweeted.

Unwavered, Hutchins stayed at his desk and continued to work. “I’ve been replying to [direct messages] for three hours,” Hutchins told Hankins in a Slack message about the deluge of press inquiries and support from fellow security researchers. “Still can’t see the bottom.”

The media’s obsession with Hutchins did not go away. Notwithstanding his role in registering the kill switch, he was also an active tweeter, quickly becoming the public face of WannaCry and its ongoing developments.

Determined to know more about the secretive then-22-year-old, reporters contacted his friends, turned up at their houses, and offered them money for information.

The security community was furious. His allies took to Twitter to denounce efforts to dox Hutchins. It’s not uncommon for security researchers to go by pseudonyms or online handles. So much so, even the U.K. National Cyber Security Center recognize him as “MalwareTech” as his byline in a post on the organization’s blog.

Now with his identity out, Hutchins knew it would be easier for criminal groups to target him for his previous unrelenting work to uncover their malicious online operations. But in the midst of fighting off a multitude of threats targeting the kill switch, he feared the unwanted attention would distract him from his current work.

“It was a huge problem,” he told TechCrunch. While the media swarmed his friends and family, the researchers were still battling attacks and efforts to knock the kill switch offline.

“I don’t work well with that kind of attention,” said Hutchins. “I can deal with stress, but attention is not something I’m very good with.”

“Having like a million journalists all over you for weeks on end? It’s not fun,” he said.

Later that day Hankins went out and bought all of the Sunday newspapers for Hutchins as a keepsake.

Hutchins absorbed most of the media attention. But Hankins, whose real-name was also not public at the time of the WannaCry attack and only in recent months began to use his real-world name with his Twitter account, feared his identity would also be uncovered.

“I was worried [reporters] were going to turn up my place next,” Hankins said. He said how he devised a plan in the event that reporters also found his home address.

“My plan was instead of going out the front door where the journalists would have been to go through my side door and then out the back — which had like a back street — and a friend would pick me up in their car and I’d go and stay with them,” he explained.

But even with the attention Hutchins said he did not regret his role in stopping WannaCry. “I probably would have tried to hide a bit better,” he joked. “But yeah, I did not really enjoy any of this.”

The cavalry arrives

The following day on Monday, Britain went back to work for the first time since the cyberattack.

Many businesses had fallen victim to WannaCry, and their systems were offline. Others whose systems had not yet been ransomed had no idea their systems were also infected. The kill switch was the only thing preventing another outbreak. The U.K.’s National Health Service was on high alert in anticipation of a “second spike,” amid ongoing disruption across the organization. U.K. authorities had joined the global manhunt for the attackers behind the attack days earlier.

But when the researchers weren’t being hit by a barrage of attacks, they knew that the cumulative pressure, exhaustion, and lack of sleep was untenable.

“I wasn’t desperate to hand it off,” Hutchins admitted. “I wanted to keep control of it.” He feared handing it off would make it far more difficult to identify and notify businesses and government organizations infected but not yet ransomed by WannaCry.

“But I came to the realization that there is a huge personal risk of me doing this,” he said. “It was a week of just pure dread every time a server went down. It was more logical just to hand it off and then get some sleep.”

Hankins told TechCrunch that several companies offered to host the kill switch but the researchers were cautious of trusting anyone. “For us it was vital to keep it alive, but for others it was an opportunity to get on this huge press cycle,” he said.

The duo knew people at Cloudflare, a security and networking giant, and reached out for help. The internet company provides many services like domain registration and protection against distributed denial-of-service attacks.

Hutchins and Hankins approached Cloudflare two days after WannaCry hit, said Justin Paine, Cloudflare’s director of trust and safety. Chief executive Matthew Prince had already given Paine the go-ahead to give the researchers what they need, offering its suite of services for free.

Mirai continued to attack the kill switch with everything it had, Paine said. The rush was on to get the kill switch onboarded and protected as soon as possible.

It had just gone past midnight in the U.K. on May 16th when the handover was completed.

For its part, Cloudflare kept quiet about the arrangement. The company did not put out a press release or blog post acknowledging its part in supporting the kill switch. For most it was an invisible partnership, the only giveaway was that the domain name resolves to a Cloudflare name server, which is not noticeable to internet users.

“We couldn’t have done it without them,” said Hankins.

92-hour week

Two years later, the kill switch has not gone down once.

The ransomware continues to lurk in thousands of networks around the world, ready to encrypt the files on millions of computers, despite patches having been available for the past two years. Hankins said that in June 2019 alone the kill switch prevented about 60 million ransomware detonations.

Hackers working for North Korea were later blamed for the cyberattack.

“After it was confirmed it had been stopped, there was a ‘holy shit’ moment that this was one of the biggest things in recent cyber history,” Hutchins said. “This is the first case of any kind of ransomware worm.”

Hankins worked about 92 hours in five days and slept only a few hours a night, according to his Fitbit data. At one point, U.K. government officials privately reached out to the researchers to offer help but also to check on their well-being, knowing the stress they were under.

“I think we struggled but we did a reasonable job,” he told TechCrunch.

All seemed well until last month when a Cloudflare outage knocked a portion of the internet offline for several hours. The cause was blamed on Verizon (which owns TechCrunch) for mishandling the internet traffic. Cloudflare’s Prince tweeted angrily at the telecom giant.

But the kill switch did not buckle. Hankins tweeted that the outage had not affected the WannaCry kill switch. There were 220,000 attempted WannaCry executions during the outage, he told TechCrunch.

“This wasn’t Cloudflare’s fault nor was there really anything we could do about it,” Hankins tweeted. “Outages and issues happen all the time and sometimes they can be incredibly localized and hard to detect.”

As long as computers are infected with WannaCry and are not patched, data remains at risk — and at the mercy of the kill switch.

“Just remove this shit from your networks please,” he tweeted.

Paine said Cloudflare still receives a handful of requests to take down the domain each year, thinking the domain is spreading WannaCry — not preventing it.

“We have to educate people that it’s the exact opposite of what you really want here,” said Paine. “If we took down that domain it would be a much worse day for you.”

Round two: BlueKeep

In August 2017, three months after the WannaCry attack, Hutchins was arrested by U.S. authorities at McCarren International Airport in Las Vegas as he boarded a plane back to the U.K. on charges of creating malware in his teenage years — unrelated to WannaCry. He pleaded guilty and will be sentenced in late July. His supporters have called for clemency given Hutchins’ more recent and concerted efforts to protect users from security threats.

Hankins, now the head of security and threat intelligence at Kryptos Logic, retains control over the kill switch and provides business and governments access to localized infection data.

Almost exactly two years after WannaCry first hit, a new vulnerability appeared. Nicknamed “BlueKeep” by security researcher Kevin Beaumont, the flaw also had a similar worm-like property to WannaCry, allowing it to spread from computer to computer.

“I was panicking,” Hankins said. The emergence of BlueKeep brought back a lot of emotions from the week that WannaCry hit, he said.

Microsoft released patches but about a million computers were still vulnerable by the time the National Security Agency issued its own rare advisory just weeks later. BlueKeep is seen as one of the most significant threats to vulnerable computers since WannaCry. Although no exploit code has yet been made public, Homeland Security has warned that it is only a matter of time before hackers figure out how to abuse the flaw and launch an attack.

“We saw this once before,” he said. “We need to stop this — but obviously there was fuck all we could do,” he said.

“We’re not getting a kill switch this time.”

---

Got a tip? You can send tips securely over Signal and WhatsApp to +1 646-755–8849. You can also send PGP email with the fingerprint: 4D0E 92F2 E36A EC51 DAAE 5D97 CB8C 15FA EB6C EEA5.