Environmental Emergency? Technology to the Rescue

Environmental accidents can strike at any time. Whether it’s a leaky furnace in somebody’s home, a chemical spill at a manufacturing plant, or an overturned petroleum tanker truck on the side of the road, the problem has to be solved quickly. Professionals are contacted to secure the site, get everyone to safety, and clean up the mess. It can take months to resolve a major incident, but no matter how severe the crisis, everyone has to move fast to get everything under control.

After more than 20 years in the business, we have the process down pat. The company I work for provides services that identify, monitor, correct, and prevent environmental problems across the country. We assist individuals and companies recovering from natural disasters and environmental accidents, remediate contaminated sites, and remove hazardous materials. We’re first responders and considered an essential service, so our offices remained open for business as usual throughout the coronavirus pandemic.

A Host of Problems 

Whenever a crisis starts to unfold or people discover a problem in their surroundings, they call us. A freight train can derail in the early morning, and a sewer can overflow in the middle of the night. As such, we are not a 9-to-5 type of operation. Our 200+ employees and our network of audited contractors are there to respond to emergencies in all 50 states, 24/7. To support this, we need rock-solid IT infrastructure.

I’m the IT network coordinator. I started here in early 2016, and at the time, this was a solo gig. I was managing our infrastructure with the help of a local Managed Service Provider and, let me tell you, we had problems.

A lot of our hardware was end of life. Our routers were mostly Cisco, but our switches were whatever had been on sale in a given week—I’m talking every vendor under the sun—and they weren’t set up correctly. Only three or four of our offices could access our file servers and email directly. Our WAN was a patchwork that didn’t allow our branches to communicate with each other or with our data center. 

At the time, we had 20 branches across the country, and when something went wrong, we often had no idea what the problem was, or how to find it. A site-to-site VPN connected a subset of 3 or 4 of these locations and let our people log into Citrix virtualized apps, but these connections were unstable and software performance lagged as a result.

At the branch level, our wireless access points weren’t running on a single platform, nor were they feeding into our WAN. Instead, we had a mishmash of consumer-grade Wi-Fi base stations from companies like Belkin, Linksys, and NETGEAR, and every office had basic WEP or WPA security, complete with separate SSIDs and passwords. There was no way to manage or secure these access points from a central location. 

The other pain point was our telephone system. We were using antiquated and overpriced PBX technology, and VoIP was nowhere to be found. Early IP telephony systems couldn’t weather a power failure, so copper-based landlines were the only way to go. This might have made sense back in the ‘90s, but it was no longer good enough. 

From Bush League to Major League

I could have updated the firmware on our existing routers and standardized their configuration, but it was a waste of time because they were already obsolete. Instead, I chose to upgrade our infrastructure to Meraki MX routers. We also standardized on Cisco 2960L, Meraki MS series and a new core infrastructure of Cisco 9200-series switches. Since so much of our infrastructure was already based on Cisco hardware, it made sense to keep it within the same family.

Moving to an all-Cisco/Cisco Meraki setup also helped us resolve our access point issue. I’d used Cisco Meraki at my previous job, and so I knew it offered the right mix of hardware and software to centralize and standardize our branch-office Wi-Fi. I needed a little more proof, so I talked it over with a former co-worker and a local vendor. They both agreed that Meraki was the way to go.

We threw out our consumer-grade gear and migrated our access points to Meraki MR Series APs We also added Meraki Z-Series remote gateways and Meraki MX enterprise security and SD-WAN internet appliances. It’s the perfect solution because the hardware is on-prem, but we manage it in the cloud using a single pane of glass dashboard. We also purchased the licensing for Meraki advanced security as it included Cisco AMP (Advanced Malware Protection), a robust security product that adds intrusion detection and protection functionalities to Meraki’s impressive feature set.

Cisco Meraki took us from bush league to major league. We went from no security and an endpoint and network management nightmare to industry-leading protection and a dashboard that allowed us to remotely configure our routers, switches, and endpoints. With Cisco Meraki, we can set up equipment at our head office, label it appropriately, and courier it to our branch offices for installation. These routers, switches, and endpoints are literally plug-and-play at the other end, and we don’t need an army of technicians to upgrade our offices.

A Walk in the Park Installation

We trialed Meraki at one of our branch offices. We swapped out the old Cisco routers, replaced the HP switches with a new Cisco core switch, and cleaned up all the cables. We installed the new access points, powered up our new equipment, and then only needed to spend a day tweaking our settings. 

We moved on to another local office, and that upgrade also went off without a hitch. After that, we started shipping out Cisco hardware to our offices across the country and managed to move most of the company to our new infrastructure over the next two weeks, typically while people were on a lunch break.

Installation couldn’t have been any easier. We’d ask for a volunteer—typically a manager who was comfortable with tech—and we’d guide them through the process on the phone. It was usually a matter of unplugging a couple of cables, matching them to the new piece of gear, and switching everything back on.

As we were doing this, one of our branches got hit with a ransomware attack. Somebody opened an infected email and it took out our Citrix environment. We were able to recover most of our files, but our Citrix environment never went back to normal, and we took this opportunity to ditch it entirely. 

With only six branches left to go in our Cisco implementation, we decided to switch everyone to Meraki. Now, anyone in the company can log into the network and activate Microsoft 365 directly.

The Only Thing That’s Constant Is Change

In January of 2020, we were acquired, and our new management showed an increased eagerness to embrace new technologies that are both cost-effective and more efficient. 

A few years back, one of our offices in North Carolina lost all connectivity one day because of nearby construction. North Carolina doesn’t have call-before-you-dig laws, and so when construction took out their phone lines, they didn’t have any recourse. It wasn’t the first or the last time that happened, but our former executive team continued to insist on sticking with copper-in-the-ground phone lines. 

During the buyout, we consolidated some of our offices and added a handful of locations belonging to our new owners, bringing our total number of branches close to 30. Recently, we opened a new branch and hired a manager before we’d finalized its physical location. We needed a phone line to put him to work, and I suggested VoIP, explaining that the initial setup and first month of service would cost $250, and it would be just $30 a month after that. Furthermore, we could transfer the line to our new office when it finally opened, at no extra cost. It was reliability and convenience at a bargain-basement cost, and just like that, our new management approved the switch.

Opening the Doors

This first step opened the door to exploring the entire Webex range of solutions. Cisco Webex combines technologies that include intelligent whiteboards, smart cameras, desktop telephone sets, and mobile apps. It offers an end-to-end solution that allows companies to communicate and collaborate with employees and customers no matter where they are or what technology they’re using. 

Cisco recently consolidated several product lines under the Webex banner, including VoIP, and so telephony is now part of the company’s collaborative platform. It is incredibly efficient and cost-effective, and moving to the Webex messaging application not only improves productivity but can also result in substantial savings.

Right now, I’m doing a cost analysis for converting seven branches to Webex for VoIP. One of our offices in Pennsylvania is currently paying $1,200 a month for telephone lines alone. After the switch, we'll be looking at a cost savings of nearly a thousand dollars. If that office in North Carolina loses its connectivity due to construction again, Webex can transfer calls to employees’ smartphones, and they can keep working despite the disruption.

At one point, we combined Meraki and Webex in the field. The company was hired to clean up a 10,000-gallon oil spill, and we needed to coordinate our teams on the ground. I shipped them a Webex phone and a Cisco Meraki Z3C gateway that incorporates a cellular LTE failover. This gear allowed them to set up a VoIP line and an onsite hotspot with full-time access to the WAN in one of our trucks. It wasn’t the intended use of this equipment, but our improvised solution worked.

Connectivity Is Everything

As first responders in environmental emergencies, we spring into action the second the telephone rings. If people can’t get through because our lines are down or our network is buggy, we can lose precious minutes in recovery. Those moments can have catastrophic consequences, and a situation can spin out of control. The sooner our teams are onsite, the easier it is to contain the damage.

Cisco Meraki and Webex have given us the reliable and cost-effective communication tools that we need to react in a crisis. We are doing more, spending less money, and using fewer resources to manage our infrastructure. 

Thanks to Cisco’s intuitive dashboard-based management and diagnostic tools, I can easily set up, maintain, and troubleshoot our infrastructure. I can spot potential issues before they become real problems and ensure that nothing becomes obsolete again. Most importantly, I no longer worry that IT emergencies might prevent us from responding to a real-world disaster.