Overcoming Complex System Management for Remote Locations Through Simplicity and Automation

Career choices often come with a particular lifestyle. School teachers can expect a little more free time during the summer. Military members often face unpredictable deployments. Writers can require quiet and isolation while they work. Those of us in IT know that our work requires us to lead an on-call life.

In addition to traditional working hours, we are often called in to fix things that break. Sometimes we have to stop working on a project to help others troubleshoot a problem, especially having remote locations, with servers outside the data center that are in dire need of firmware updates. And as technology is such a vital part of our activities, we have to ensure that all systems and hardware are up to date at any given time—with minimal or no disruption to the rest of the organization, otherwise this can become the number one issue for the company.

With these pressures and responsibilities, above all, technical staff must have faith in their infrastructure. Once an IT team loses confidence in the architecture they manage, things can quickly turn south. That translates into stress, sleepless nights, and even longer working hours.

Handling Updates as a Dispersed Operation

I learned the IT trade in the military, serving as a radioman. After being discharged, Kimley-Horn hired me to do a lot of the same things. Kimley-Horn is a planning, surveying, engineering, and design consulting firm based in the U.S. I currently act as the server lead on our infrastructure team. 

Our shop is somewhat of an “old school” operation. For most of the staff, the label "Jack of all trades" is a fitting descriptor. We spend about 50% of our time on special projects, and we also delegate random tasks as they arrive. While legacy is part of the reason for this arrangement, we also operate in a more free-flowing manner because of our organizational needs. 

We have about 100 offices around the country, and each has at least one server. My team is responsible for the servers, but we have to work with local IT staff in each office to maintain the technology. The remote nature of our IT operation can make management a headache, especially during activities like firmware updates and server deployment. 

Under the old system, server changes or updates meant substantial data transfers and coordinated installation. Even in cases where everything goes right, that would easily require four to five hours of downtime during scheduled maintenance windows.

But of course, everything doesn’t always go right. Installing firmware required us to push a large bundle of data across the WAN to our servers, and it would timeout in the middle of the process. In those cases, we had to start the process all over again. And because we all multitask, the extra time transferring data comes from the resources allocated to other tasks.

Lost Time, Lost Money

Then, of course, there are the business costs of downtime. Kimley-Horn’s customers span a wide range of industries, including retail, entertainment, public works, and educational institutions. We believe in cultivating long-term relationships, which is why nearly 90% of our business comes from repeat clients.

The nature of our business involves hourly billing and multiple projects at different stages. For us, downtime directly translates to missing revenue and lost progress. In addition, because we deal with so many different industries, our staff don't always work the traditional 9–5 workday. We cannot count on being able to schedule maintenance downtime windows after hours or on the weekends. 

Kimley-Horn was expanding, and that meant IT grew more complicated. We could no longer afford extended downtimes and unreliable maintenance windows. 

A Migration Led to an Imperfect Solution

About four years ago, we made the transition from Dell to HPE, primarily for hardware reliability. As a consulting firm, every moment the system is down means people can’t work, which costs us money. The last thing we need is to select equipment that doesn't contribute to a stable environment.

We were happy with the equipment, but management was still complicated. After our migration, we used HPE OneView to manage the servers, but it was not designed for our environment. We just had too many servers and our operations were too remote. HPE OneView is best for much simpler setups than ours, and it could not handle our needs.

The biggest problems with the system were still related to our scheduled maintenance windows and updates. Because the data transfer process was so cumbersome and failure-prone, our regional IT managers were reluctant to schedule maintenance windows and updates. We had to work through their frustrations and have some difficult conversations about why things weren’t working as they should.

They were losing confidence in the system and the products. With so much depending on our network, we could not afford to remain silent. 

The Final Piece: Building a New System from the Ground Up

We shared our concerns with HPE, specifically with our Account Manager Kerry Perry. We knew that ours could not be the only enterprise out there with this problem, and we told HPE that we needed a new solution to address our needs. 

Eventually, Kerry told us that HPE was designing a new system that would come to be called HPE for GreenLake Compute Ops Management. This platform promised to simplify and unify the management of complicated infrastructures while automating several essential tasks. But best of all, HPE wanted Kimley-Horn to be a part of the team working to develop the new product. With our success riding on the new solution, we were very excited about the opportunity to help shape the outcome.

As an engineering consulting firm, we like to think about solutions. We knew exactly what we would need in our new management system. It would have to be customizable so we could tailor it to our specific needs both now and in the future. We wanted intuitive dashboards to easily monitor the entire system at a glance and drill down into any problems. And, perhaps most importantly, the new solution had to be designed to streamline our operation.

Based on what we knew about HPE GreenLake for Compute Ops Management, we believed this would be the answer to our problems. Our greatest difficulty was transferring those huge data files to our branch offices for updates, and HPE GreenLake for Compute Ops Management solved this problem with a novel approach. Instead of pulling down the entire file, we could schedule updates and the system would download only the necessary data for that specific update.

Overnight, the server update process went from about four hours to 45 minutes. That is a considerable improvement in terms of the operational impact of downtime. For a company that is always busy and constantly growing, the ability to update servers in under an hour is a big bonus. It made IT look better and eased tensions with staff in the regional offices.

Valued Partnerships Help Us Move Forward with Technology

Kimley-Horn is all about building relationships with our clients, and it is nice to know that HPE feels the same way. Our organizations continue to collaborate and work toward creating an even better solution. New functionalities like alerting and detailed dashboards will provide even more ongoing information about servers' overall health.

While we are excited about new features and the impact on our company, our work on HPE GreenLake for Compute Ops Management represents much more than that. We are a fairly small company, and we are far from HPE's biggest client. Still, I appreciate that HPE values our feedback and responds to our needs with just as much enthusiasm as they would to a larger customer. In a world of constant changes, that attitude gives me the confidence to believe that they can always keep us at the forefront of technology.

Data system integrity is the top priority for my IT team, but the culture and happiness of our staff is a close second. HPE GreenLake for Compute Ops Management promises to restore the work/life balance of our IT team. Now they will not need to spend so much time completing updates during evenings and weekends. Our workers also have more confidence in the system, including its reliability and stability. That means we don't have to work so hard to convince regional staff to schedule maintenance.

We are still on the road to a finished product, but I’m thrilled to have HPE as our partner. We remain convinced that they are on our side and willing to address our worst pain points. We might be a small customer for HPE, but we trust they will hear and listen to us. In the end, their solutions will help us manage our servers easier, faster, and with greater reliability.