How Hyper-Convergence Simplified Our IT—and Gave My Team Back More Family Time

Working in IT can feel like an on-call profession—even if it’s not in your job description. Server upgrades, infrastructure changes, routine maintenance, and even dreaded scheduled downtime are all high priorities. But they can't be done during normal business hours or you'll disrupt other critical activities.

The end result is having to come into work on nights and weekends after everyone else has already gone home. Since IT is mission-critical at many organizations, this sacrifice is a must for many of us. But that doesn’t mean it has to affect your personal life. 

Joining Forces

My career in IT has taken me down a lot of different paths, but the past eight years have been particularly interesting. I work for two different companies right now. The first is OneWorld Community Health Centers, a medical clinic where I'm an IT Director. The second is Heartland Community Health Network, where I started in 2013. I'm the IT Director there, too.

Despite the fact that they're two different companies, they're very closely related. Both organizations share a few staff and even a data center.

At Heartland, I'm lucky enough to work with an incredibly talented group of people. We host the medical records for our Federally Qualified Health Center (FQHC) members who reside in Nebraska, Iowa, and Texas. We also do custom Electronic Health Record (EHR) development. This includes developing and delivering reporting for our members and even includes EHR training.

Fourteen years ago, the government had an idea that would help FQHCs—like OneWorld—stand together. They hoped we could pool our resources by way of a single network that would get everyone up and running on an EHR. That's ultimately how Heartland came to fruition. Ten years ago, we banded together with several other FQHC organizations, started the Heartland Network, and helped move everyone out of the world of paper and into the digital era.

A Hodgepodge of Systems

With the shift to take everyone electronic, this meant Heartland needed to invest in a data center to run everything. But as we rapidly expanded over the following 7–8 years, the number of members utilizing our data center—and the data they brought—grew with it. Unfortunately, keeping up with this expansion was difficult. 

Being a nonprofit with a limited budget, we had to grow our infrastructure in an incremental way—adding solutions as we found the budget. But developing your infrastructure in that way doesn’t lend itself to long-term thinking. All you think about is how to make the solution work right now. We had different consoles, servers, and storage area networks. As you can imagine, this hodgepodge of systems was a nightmare to manage.  

With our ad hoc growth, we’d acquired three separate installations of VMware Essentials Plus. But that had a three-host maximum. If we needed to expand compute, we just couldn’t afford to go out and buy VMware Enterprise and scrap what we had already invested in. All we could afford was to add another cluster, so we purchased one. It was a constant juggling act to maintain consistent versions of hypervisors. 

After-hours maintenance was another issue. Maintenance windows were massive because of how long it took to patch and reboot the systems. The multiple SANs and the time required to maintain firmware added up to an enormous amount of after-hours maintenance for me and my team. 

For our customers, we’re lucky that none were 24-hour facilities. This allowed us to perform maintenance overnight. But where things could get complicated was if an obstetrician, for example, has a mother in the ER who’s ready to give birth. In that situation, access to her chart is vital information. If we’re performing maintenance during that time, the doctor wouldn’t have access to our EHR. That wasn’t a frequent occurrence, but it occasionally happened.

Even if they weren’t impacted by maintenance times, our users still faced issues. Login times to our EHR system were wildly inconsistent. It all depended on the particular Citrix server they landed on or how much I/O was hitting the SAN. With the important work they’re doing, our users required a more consistent experience. 

For our team, how could we try to remedy all of these situations? Well, when a problem would come up, we’d need to investigate. But on the management side, my team didn’t have a centralized place to see our environment’s performance. We had no way to quickly and efficiently home in on the true origin of an issue. We jumped into different management consoles, trying to figure out the root cause of a problem. We needed a solution that could collapse everything into a single pane for us to view. Our staff is too small—and our responsibilities too large—to search everywhere. 

We knew the only way to get us out of this mess would be a full hardware refresh of our aging infrastructure. 

Growth from an Initial Cluster

We spent some time searching for the right vendor to simplify Heartland’s sprawling infrastructure, and the solution to our problems presented itself at a "lunch and learn" hosted by a local vendor. Among other topics, they talked about Nutanix. 

What immediately drew us to Nutanix’s offering was how easily we could expand our infrastructure. Our nonprofit nature wasn’t going to change, which meant we’d still need to grow one step at a time. But we had to find a way to grow incrementally without making a mess like we did last time. 

Nutanix showed how we could start with our initial cluster and then just add nodes as needed. We loved that. And it’s a proven concept. They told us how large companies like Google, Amazon, and Facebook built their architecture in the same way. We’re nowhere near their size, but we knew they approached the problem in the right way. 

This path would allow us to make small investments every year as we added new members. We no longer would have to worry about a massive capital outlay to bring in new compute, SANs, and networking equipment to add more capacity. We’d be able to just add a node that matched my workload needs. 

Nutanix clearly had the solution to simplify our IT into a single view while also letting us grow without breaking the bank. We were sold. 

Simplicity with No Strings Attached

With Nutanix, simplicity was our top priority, and we noticed the results almost immediately. Two weeks after we began our implementation, I had to go on paternity leave because my wife delivered three weeks sooner than we expected. 

When I returned from paternity leave, I checked with my team and asked how things were going. They said, “It's done. We got everything migrated from ESXi to AHV using Nutanix Xtract." I came back at the end of paternity leave to a fully implemented solution. That's how easy things were, and that was just the beginning. 

The very first update I did for Nutanix happened with my newborn child in my arms. I had my son in my hands, feeding him his bottle, and I just clicked "Upgrade." Things had only been in production for about a month, and what used to be a nightmare process was now so simple that I could devote all of my attention to my son instead of a computer screen. I could never have done something like that while upgrading a traditional three-tier infrastructure.

In general, our lives are dramatically simpler now. By moving to Nutanix, we’ve been able to reduce the complexity of our Citrix server provisioning. Because of the integration with Nutanix and Citrix, we were able to move from Citrix Provisioning Services to the simpler Citrix Machine Creation Services. For example, whenever we have to make a major upgrade to a driver, we don’t have to complete a complicated set of reverse imaging to make a change to the driver stack. This saves our team a bunch of time—and they love that. Today, they love working in the infrastructure. There’s excitement and hope again. Any time we’re at a user group meeting or a health care conference, we can’t stop talking about how much easier Nutanix has made our lives. 

Tom, our Systems Administrator, states, "Managing our Nutanix environment is pretty hands off. Their one-click upgrades are non-disruptive. The reduction in infrastructure maintenance has allowed me to focus on improving my IT skills, and on things that matter to the company and our users." 

For the members we support, there’s a massive impact. Our maintenance windows have shrunk considerably for two reasons: The first is because of data locality, and the second is because we’re running on Flash storage. Patches install quickly and reboots are often completed in 15–20 seconds. This means when we are patching hosts and clusters, our end users don’t even notice. 

And those login issues we had before? They’re gone now. Running on Flash means a consistent user experience all day, every day.

We would love to tell you that the phones blew up and our emails overflowed with enthusiastic people saying, "Everything is amazing, you rock!" But that’s not what happened. Nor should it have, really, because when IT is firing on all cylinders, things are so effortless that you don't even realize what goes into it. You only get emails when problems arise, and our inboxes have never been quieter.

Feedback from the Top

Still, we did get recognition from our executives, who were overwhelmed by how fast and efficient our EHR has become. It was great to hear from them, because folks in IT often don’t hear praise.

I'd say the amount of time we're spending on after-hours maintenance alone has been reduced by at least 50%—just because of the faster infrastructure. We're easily saving three hours per week, per admin, because of how well everything performs. 

All of that time and those resources can now be invested back into the company. I want to take us in the direction of infrastructure as code (IaC), which begins with more automation. My team will develop scripts in a platform like Nutanix Calm with PowerShell. This will help automate how we build out of the infrastructure, so whenever we want to make a change, we'll deploy new environments from the existing code.

Nutanix makes all of that automation possible with their APIs and PowerShell integration. Today, when we add a new customer, we need to build out new SQL environment and dozens of other servers for them. But if we build out a set of scripts, suddenly, our quality and consistency go up. And security can improve because we can build that right into the script. It’s exciting just to think about! 

The work we do at Heartland is important, and it’s certainly more than just a job. But, outside of this, I have a family—like many others in my field. If you’ll excuse me, I think I’ll get back to them.