Resource

Case Study: HSE Conti Ransomware

What Happened:

In the early hours of May 14th, 2021, hackers began locking up 10’s of thousands of the HSE’s computers and servers, demanding a €20 million ransom (source) in order to unlock them. What’s worse, the malicious actors threatened to release all patient personal health details onto the Dark Web if the ransom demands were not met. In response, the HSE contracted the Irish Defence Forces, the National Cyber Security Centre (NCSC) and critical third parties to aid and provide structure to the recovery of Ireland’s public healthcare system (source). Taking 3 months and a projected €100million (source), this is the story of Ireland’s most damaging cyberattack in history.  

Background:

With over 100,000 staff directly and indirectly employed (source), the HSE is Ireland’s largest employer, with more staff than the 2nd, 3rd,4th and 5th largest Irish employers combined (source). The difference between the HSE and these companies is the nature of the work they do. While the aforementioned companies are in the business of wholesale, retail and banking respectively, the HSE is in the business of saving lives. The Health Service Executive, known formally as the HSE, is Ireland’s public healthcare system. They provide care across the country with over 4,000 locations, 54 acute hospitals and its own ambulance service (source).

But the figures speak for themselves. The HSE deals with over 3,500 Emergency Department patients every single day (source). It’s needless to say that their IT demands are not only gargantuan, but vital. From everyday printer problems that could delay a patient getting lifesaving prescriptions, to million Euro MRIs not being read by a computer and resulting in the cancelation of further appointments involving the device that day. Utilising over 70,000 workstations including PC’s, and servers, the HSE’s information technology demands rival those of many multinational companies (source).

Adidas, the global clothing company, directly employs over 60,000 worldwide staff (source), or about half of the labour force of Ireland’s national health service. While the bulk of Adidas’ employees are contracted in retail – 58% to be exact – they boast an impressive 1,500 employees working in IT. The HSE, despite being a workforce double in size to Adidas, employs 350 staff in IT (source). While they both operate in completely different industries, the staggering difference in human capital available illustrates the complete lack of adequate resources the HSE had dedicated should an IT disaster ever occur. Unfortunately, in the early hours of May 14th, 2021, that IT disaster struck.

What happened?

 While thousands of machines first displayed signs of virtual death and displayed demands for ransom at the dawn of May 14th, hackers actually infiltrated the HSE’s systems earlier. Much earlier, in fact their systems were compromised almost two whole months earlier. The initial breach into the HSE’s network occurred on the March 18th, when pseudonymously named ‘Patient Zero’ opened a malware-infected Microsoft Excel spreadsheet attached to an email.

While this may seem bizarre, it’s a type of impersonation phishing attack that is surprisingly popular in everyday IT communications (known to security specialists as ‘the wild). Hackers will research targets and groups of targets within an organisation and email them, claiming to be a colleague or a senior figure within the organisation, instructing them to open an important file with an inspiring title such as “Christmas Bonus” information or “Important Ward Statistics”. Their aim is to grab enough of your attention to make you open the attachment believing its an important document that you need to access. We may think we’re all impervious to these sorts of scams, but on a busy Monday evening after a long shift, we often may forget to check for the tell-tale signs of a phish (and the data proves it – source).

The hackers, thanks to their dubiously crafted malware, are now in the network. Whilst most would assume their first port of call would be to instantly start scraping for sensitive data, locking or destroying files and propagating onto the rest of the network, reality almost always tells a different story. Malicious actors in actuality love to just sit there and ‘watch’, more accurately – listen – to all of the network traffic. When an IT network is behaving as expected computers ‘talk’ to each other, i.e. “I’m Computer X with Ports A, B and C open, looking to connect to Server Y. If you’re Server Y, please respond. Otherwise, ignore this message. I am also ”. By taking a backseat and observing all of these and similar requests, the hackers get a good idea of how this network normally operates. Their objective is to go unnoticed within the network and learn from user behaviour.  If hackers were to scream out “I’m Computer Z. Can every single computer tell me their names and how I can connect to you?”, they would raise more suspicion and corrective action would be taken to remove them from the network. 

Initially, the hackers just sat and silently waited. As time progressed, the infiltrators began to make more and more noise on the infected computer. In fact, 13 days after their initial entry, they used two of the most renowned hacking tools, MimiKatz and Cobalt Strike, to gain valuable information including passwords and logins. While the Antivirus Software failed to notice the hackers’ initial intrusion, it did alert the computer user upon the use of malicious tools. However, the Antivirus software had been manually set to monitor mode and didn’t block the malicious behaviour. The MimiKatz tool is not used for legitimate purposes and hence if the anti-virus software had of been operating correctly it could have prevented a national disaster. But since no cyber professional was ever alerted, the hacker continued on.

From April 1st to May 7th, the hacker had been relatively quiet in their manoeuvrability around the network. However, once it became clear there was a distinct lack of adequate resources dedicated to stopping hackers like them, they began to commit more and more brazen acts with less and less concern for staying quiet. From May 8th to May 12th, the hacker went on to compromise 6 different hospitals, taking over domain controllers, adding backdoors onto servers and loudly opening sensitive files on systems. Again, with the use of the obviously malicious Cobalt Strike on a system, the hacker was vehemently announcing their presence, only to be responded by blatant silence.

On May 11th, with a growing sense of disregard for keeping a low profile, the hackers compromised a hospital by attempting to brute force a login into a domain. While this method for lateral movement is extremely effective, it’s normally not preferred by higher skilled hackers, as it’s an easy giveaway of their presence and will often get them stopped in their tracks and removed from the environment. Given their blatancy to date and their ability to remain within the network without aversive action by IT personnel, the hacker persisted. When the glaringly conspicuous brute force method didn’t work, the hacker likely took advantage of the system’s lack of implemented updates and exploited a known vulnerability in the system. There is a strong possibility that the exploit was similar to EternalBlue, a hack that was so infamous it made the news for its sheer implications and use in ransomware against 200,000 computers in 150 countries (source). The security patch to prevent this exploit has been available for year, however, it is only effective at stopping attacks if it has been applied to systems.

Undeterred by the fear of creating mass alerts, the hackers started viewing files, opening folders, creating archives, and sharing them freely on May 12th. This brazen recklessness caused concern, not from the Hospital IT staff, but from the Antivirus vendor, as there were so many alerts popping up and so little remedial action. They then emailed the HSE’s Security Operations Team alerting them of some of the suspicious activity. In response, the HSE’s Security Operations Team sought to turn an individual server off and on again. Actions like this are unlikely to have any significant effect on the hacker’s presence as they had already placed backdoors on that and every other machine they touched.

While it’s incredibly easy to exemplify where mistakes were made by the IT team at the HSE it’s also important to acknowledge the good work that was done following this incident. One hospital, moniker Hospital A, correctly identified the malicious alerts and initiated a pre-prepared Incident Response plan on May 12th. They reset “4,500 passwords and made firewall configuration changes to contain the activity”. While incorrectly assuming the origin of the malware, they implemented the correct steps for remediation. They installed improved Endpoint Detection and Response (EDR) on the majority of their systems – likely their saving grace in the days to come.

Presumably concerned by the sudden implementation of newer, improved Antivirus systems, the hackers began executing the ransomware at 01:00AM, May 14th, 2021. By 04:30, the HSE had already received multiple reports of multiple systems being unavailable and locked as a result of the incident. They also became aware that access to their Data Centre which contained sensitive patient information was restricted, essentially lock them out. At 05:10, “Subject Matter Experts” decided the best port of call would be to remove all connectivity to the giant network of machines, disconnecting everything from the internet and everything else. This crippled any and all further paths of spreading the ransomware that the hackers had available to them. Consequently, this also destroyed normal communication channels, like email and networked phone lines, within the HSE network, ushering in the use of analogue phones and fax.  

At 10:00, the Garda National Cyber Crime Bureau, Interpol and the NCSC were brought in to support the response. It was immediately decided that the ransom would not be paid. The Irish Government, Gardai and the NCSC believe that industry and the public should not pay ransom demands from cyber criminals and the funds are seen to enable further criminal activities and encourage the spread of cybercrime. It was not until 16:30 that the Incident Response team began to install defensive EDR on the newly crippled machines.

The response by the HSE was extensive. On May 15th, the HSE senior management commandeered a third party’s office building to enable a coordinated plan of action response to the incident. Senior management received clean Microsoft 365 mailboxes to enable quick and swift communication relating to the attack. Following the 17th of May, solid key recovery plans were implemented.  These included:

  • Identify priority applications to include when rebuilding the foundational IT infrastructure,
  • Implement a monitoring system to detect patient information leaking onto the internet,
  • Contact the Irish Defence Forces to assess aid capabilities and requirements,
  • Distribute clean laptops to select HSE staff members and allow them to use personal emails for crisis communications
  • Establish a “Legal and Data workstream” to aid with the response to GDPR’s Data Protection Office,
  • Have Occupational Health check responders’ health and implement working rotas for this team out of concern for staff burnout.

On May 20th, 2021, in another bizarre twist of seemingly criminal compassion, the hacker released the decryption key. In an unusual twist, they did not ask for any payment for this. After testing its authenticity and efficacy, the HSE’s Incident Response provider developed a more stable version and began to decrypt all locked computers. Despite this good will gesture by the hackers, their ransom demand remained at €20million, threatening to release further patient information into the public domain if it hadn’t been paid. However, the HSE’s position remained steady, and they refused to pay the ransom.

From May 22nd onward, the HSE focused on their recovery efforts, cleansing machines, recovering systems, and restoring applications. Within one month, the HSE had decrypted just under half of the servers and around half of the applications. Three months after the incident, every server had been decrypted and almost all Acute and Business Services applications had been restored. Undoubtedly, recovering from cyber-attack takes time, money and most importantly knowledge.

From the cost of repairing and replacing workstations, to contracting IT support, to rebuilding the IT infrastructure, it’s easy to see how the cost of the attack could reach the projected €100million which had been reported. What is disappointing in this example, is there were many points at which the attack could have been thwarted. Hospital A, in fact, mitigated almost all of the damage incurred by paying attention to the alerts, implementing their incident response playbook and applying appropriate security controls. Unfortunately, this is a reality unique to Hospital A. The approach was not standardised across all of the HSE networks.

How did this happen? (Unless otherwise stated, all the following information is gathered from here)

The following is the personal opinion of the author.

While it’s easy to point fingers at a single point of failure, it’s also naïve. To point and say the staff member who opened the malicious file is singlehandedly responsible for the HSE’s cyber downfall is wrong. The disaster that ensued is the result of a frail IT estate that has lacked investment over many years required to maintain a secure, resilient, modern IT infrastructure. Those are the exact words of the PriceWaterhouseCooper (PwC) – the company hired to conduct a formal investigation into the incident.

The success of this cyber-attack was multifaceted. The National Healthcare Network which is the backbone of the HSE and enables staff to access IT applications across the country is largely running on a frail IT architecture. More than one third of the servers were deemed “end of life” or “out of support” meaning that once the hackers accessed the network, they had the “freedom to roam”, bouncing from machine to machine around the network, across the country. This turned a single hospital’s IT failure into a national healthcare system crisis.   

Additionally, there was a cyber infrastructure to aid in the prevention and remediation of cyber-attacks. While Antivirus had been applied onto the machines, it hadn’t been properly setup and wasn’t sufficiently equipped to deal with modern threats. When the hacker maliciously connected to the computer, no alert warning had been issued. Conversely, when a warning of the malicious use of hacking tools was issued, the improper setup resulted in no action being taken. There was also a distinct lack of an appropriate data and system backup. Should this have been in place, the recovery time for the network would have likely been reduced dramatically. It is inconceivable that an organisation with thousands of machines connected, did not have a suitable system for rapidly restoring them should they become unusable. Instead, the HSE opted to periodically back up the servers to tape, a system that is slow from a recovery perspective. There seems to have been no situational planning for when the entire system would ‘go down’.

Finally, many security standards refer to the approach of ‘lead by example’. Pre-incident, thee HSE lacked a clear figure head responsible for overseeing IT infrastructure and security. As a result, accountability was disseminated and inadvertently diluted. Typically, in organisations of this size, notably identities handling and processing such volumes of sensitive and personal data, have a Chief of Information Security Officer (CISO) in place. Tasked with managing an organisations physical and cyber security, as well as safeguarding sensitive company information, a CISO is essential in any modern environment. Without a strong CISO at the helm of the cyber ship, the HSE was left to flounder at sea when a cyber incident struck.

It could have been worse:

While the HSE did many things right – reverting to pen, paper, and fax, establish communication channels and all third parties going “above and beyond their call of duty” – there was an element of luck in how the recovery proceeded. This was a relatively unsophisticated attack, with the attackers committing brazen acts in the network that would have been detected in a more mature cyber space in a heartbeat. The hackers were generous in their approach to the attack, simply chasing the computer in front of their limited horizon, not prioritizing the most impactful catch. The hackers failed to prioritise the systems and devices that would cause chaos within the network, such as renal dialysis and key operating theatre scheduling devices. Indeed, a more sophisticated hacker would have prioritised such debilitating devices, potentially forcing the hand of the HSE towards paying the ransom request.   

Additionally, the hackers released the decryption tool without payment. Without such help, it is unclear how long the HSE would have taken to decrypt their locked machines, if ever. Using the free decryption tool provided by the hackers, the Incident Response providers were able to reverse engineer it and adapted it to their needs. Despite this ‘gift’, it took three whole months to return to almost full operational capacity. Without an ability to decrypt the computers at scale, the recovery timeline is simply unimaginable, with an associated direct impact on the public’s health and wellbeing.

Conclusion:

The HSE Cyber Incident of 2021 is a series of unfortunate events, littered with prime examples of typical errors within an organisation that can be easily overcome and planned for. It also includes noble and humbling examples of diligent and hardworking staff, exceeding expectations and performing beyond what was expected of them. This real-life example should serve as a warning for all organisations who have continuously underinvested into their IT infrastructure. It is not a matter of ‘if’ you have a cyber-attack but rather ‘when’ you have one!  More so, do you have a disaster recovery plan in place? And if not, many millions of Euros will it cost you?

 

Author: Leo Camacho

Image

Folow us

Image
Image

Folow us

Image
Image

Donegal

191 Colab,
Port Road, 
Letterkenny, 
Co. Donegal

+353 7491 17034
hello@cybrisc.com
www.cybrisc.com

Belfast

3-5 Commerical Court
Co. Antrim
Northern Ireland
BT1 2NB

+44 2896 205140
hello@cybrisc.com
www.cybrisc.com