Recovering From Ransomware: Lessons Learned From Our Attack

Managing a business requires preparation for many different scenarios. Some events have a low likelihood of occurring, so it’s hard to adequately prepare. Other events, however, are inevitable. These days, one of those inevitabilities is a malware attack. With the number of cases growing each year, IT professionals know that such attacks are not a matter of "if" but "when."

Our "when" came just a few months ago, in the midst of the pandemic. Prior to that attack, I would not have believed that our company would attract the attention of bad actors (criminals). Large financial, data collection, and credit card companies, those were obvious targets. But a small custom fabrication operation? Surely, I thought, we’d be okay. Little did I know how wrong I was. 

Customization, Creativity, and Automation

Since 1957, A & A Machine & Fabrication has provided premier machining and fabrication to companies and industries around the world. We began primarily serving industries in Texas, but our easy access to rail, airport, and shipping ports allowed us to provide custom made parts to companies far and wide. We may be a small company, but we operate on a global stage.

At A&A, we have a unique ability to customize. Our clients come to us with a drawing, and we use our knowledge of physics and materials science to make that concept a reality. One of our specialties is high-pressure tubular reactors and coolers.  We are the only North American manufacturer of high-pressure tubing. With the help of our cutting-edge computer-assisted design and machining process, we can create a wide variety of bespoke equipment.

I arrived at A&A following an employment freeze in the oil and gas industry. I came across the company during my travels in and out of Houston. It just so happened that one of A&A’s key leaders was approaching retirement, and I began working at the company with the intent of becoming his successor. Immediately, however, I found myself in another role, related to updating our processes.

I wanted to make our staff much more productive, and to do that, I needed to leverage automation. Why should a human do mundane tasks when computers could free them from keyboards? Our customers come to us for creative, custom solutions, not cookie-cutter parts. We needed to help our employees fulfill those high-level challenges and creative endeavors. 

Our efforts to update our technology and automation resulted in a number of process improvements. We experienced some well-earned growth and moved into new markets. But success often breeds complacency, and we fell into that trap.

Don’t Forget About Security

In retrospect, we should have spent more time updating our security. But preparing for a cyber attack was pretty far down on our list of priorities. We had no history of serious incidents, so we never thought it was necessary to improve our security much beyond antivirus software and firewalls. 

We were also of the mindset that you can either ignore security altogether or you can spend your entire budget on security measures. Like many things in life, there is a lot more gray area than there is black and white, but we thought we had found the right mix of security features to match our risk. And then, suddenly, we had a pandemic to deal with, so it was definitely not the best time to invest in security.

We were laser focused on keeping the business running. Maintaining the safety of our employees and customers meant finding a way to operate remotely—which is incredibly difficult for a business dealing with machinery and repairs. We needed to reconfigure our working processes and align our technology toward our new needs. 

As I was having these conversations, in the back of my mind I wondered how all these changes would impact our security situation. Six weeks into lockdown, I got my answer.

This Is Not a Drill

For some of our staff, their day begins before dawn. On this particular morning, I received a text around 6:00 a.m. The message included a screenshot of a computer monitor with a message indicating that it was locked—and we’d better pay up to unlock it. 

As A&A’s IT guy, I rushed to work so I could uncover the extent of the damage. There was no question what was happening. The criminals threatened to harm our company unless we paid the ransom. While paying the ransom might have seemed the easiest way out, we felt very strongly that helping the criminals profit only encouraged them. Every time they get paid, it rewards their efforts. My boss was adamant that we would not pay, regardless of what we lost. 

But we still had to figure out a workaround. Fortunately, the attack happened on a Friday so we had the weekend to plan. We started with mitigation. We told the staff to immediately back up anything and everything possible. I was doing my best to stop the bleeding, but I knew we needed more support to get our operation up and running again.

OneNeck To the Rescue

More than a year ago, we used OneNeck IT Solutions to help us launch Microsoft Office 365. Through that project, OneNeck came to know and understand our production processes and technology solutions. They had professional staff and the knowledge to get things done. So when the ransomware attack took place, we knew they were the ones to help us recover.

The attack happened on a Friday morning. OneNeck staff operated as remotely as they could, and when they couldn’t, they stayed on site as late as our own staff. OneNeck made resources available as we needed them and they were willing to do whatever it took to help us recover. 

By Monday morning, we had new desktops in place and an ERP system running. Our email was operational the following day, giving us nearly everything we needed to conduct business. By the end of Tuesday, we even had a working file sharing solution. Only 2.5 days after the initial attack, we were at 95% of our operating capacity.

One of the reasons this was possible was because of backups. I’d put a backup solution in place about 10 years ago—but the solution was largely untested. We never had to utilize it to recover anything of value on such a large scale. But our backups were on the cloud, so we were able to retrieve them right away. 

Because of the existing backup system and fast actions of OneNeck, we barely lost any data. More than a simple contractor, OneNeck was our partner throughout this process. 

How Should You Prepare?

More than anything else, this attack taught us you’re never too small to think about security. We now know that maintaining proper backup schedules is a vital part of enterprise security. The faster information flows within your organization, the more frequent your backup points should occur.

We also learned the critical need for isolated backup locations. If all data is duplicated in the same location as the original data, then you don't really have a backup plan. If you lose that entire system to a ransomware attack—or another site-specific incident—you will lose both your main source of data and the backups. It is important to place your backups in a different location or on a completely isolated system.

In emergency planning, they often say that if you have a plan that is never tested, then you don't have a plan. In a similar fashion, enterprises must practice testing the storage and recovery functions of their backup plan. You do not want your first recovery operation to happen while your business is at a standstill.

At the end of the day, we saw some positive impacts of the ransomware experience. We had a number of technology updates and streamlining projects that we needed an excuse to finish. We didn’t suffer a data breach, and we are implementing employee training so they can better recognize phishing attempts. We also have more comprehensive antivirus protection and a more aggressive data backup schedule.

Know Who’s Got Your Back

Perhaps our biggest takeaway was seeing that having the right partner makes all the difference. OneNeck had the resources and team experience we needed to get back to work. For every barrier we encountered, they had a staff member with experience in that exact arena. That gave us the expertise we needed to make some good decisions and the confidence to trust our future to those choices.

There are other forms of malware out there, and we know we’re likely to be targeted again. We want to know our weaknesses and are undergoing an end-to-end security assessment. OneNeck is conducting a full security audit, the results of which will help us to most efficiently focus our future efforts. OneNeck's role in advising us to prepare for the future is invaluable to a small business like A&A, and their experience will continue to serve us well.

I don’t recommend that other organizations wait for a ransomware attack to kick security preparation into high gear; although we lost very little, the outcome could’ve been much worse. I do, however, recommend that everyone find a great IT partner like OneNeck, and complete security audits in advance. Because when it comes to bolstering your crisis response team, you can never be too prepared.