In July 2023, I commenced a role as a Cloud Engineer at a company specializing in deploying new containers of their software suite within VLANs for customers, either extending a tunnel to clients or leaving them public-facing based on their needs. Upon reviewing network topologies, I identified several security concerns by September and informed the CEO. Specifically, several inherited legacy systems were not protected by a firewall. To allow these systems to interface with internal networks, virtual NICs with internal network connectivity were added to approximately ten legacy VMs used for various purposes, including monitoring.

In late September, the CEO addressed the urgency of this issue with my team and the security team, prioritizing moving these systems behind a firewall. On October 9, 2023, I was alerted by our external monitoring team that several VMs were going offline. Upon investigation, I discovered our data was encrypted, indicative of a targeted attack. All VMs were inaccessible, and backups in our internal repositories had been deleted despite immunity measures.

After attempting to contact the CEO, security team, and coworkers, only an HR representative responded, who then notified the CEO. As more engineers joined the effort, we analyzed the situation and discovered that only the first 10 MB of each VMDK was encrypted. Using specialized tools, we removed this encrypted portion, scanned for partitions, and were able to recover the partition tables. Many VMs required boot record or GRUB reconstruction. Concurrently, we were rebuilding hosts across our data centers and resetting all system passwords. The security team thoroughly reviewed logs to identify the attack's origins.

After four exhaustive days, our systems were operational again. The ransomware attackers demanded $400,000 USD, which we did not pay. However, our recovery method had limitations, particularly with snapshots, as only the parent disk could be restored, rendering older snapshots useless. Additionally, XFS and ZFS virtual disks required complete VM reconstruction due to irreparable partition table damage, though we could retrieve raw data from these disks.

The security team determined the attacker exploited a vulnerability in a public-facing legacy Windows system, which was scheduled to be moved behind a firewall. The system had RDP (port 3389) enabled, allowing the attacker to gather data before targeting our hypervisors via GUI access, enabling SSH, and mounting them via SSHFS to launch the attack.

This incident was a significant learning experience. We have since hired an external security vendor to enhance our security posture, which includes moving all public-facing systems behind firewalls, network and VLAN segregation, regular updates and password changes, offsite backups to tapes and cloud repositories, setting up bastion hosts for administrative tasks, removing unnecessary permissions, and implementing XDR and MDR solutions, among other measures.

I urge you to reassess your internal security measures proactively. Avoid postponing improvements, as the window of opportunity can be narrow. We were fortunate to recover; others may not be as lucky.