Building a New Data Center and Going All in with Nutanix

The healthcare industry has undergone rapid transformations over the last decade, making it difficult for many companies to keep up with the changes. Large healthcare providers, like Moda Health, have had to prioritize spending to keep their complex environments secured. This has meant we’ve postponed some areas that might otherwise have been considered higher priorities to meet regulations changes, establish policies and procedures, and maintain relationships with vendors.

I’ve been in IT for nearly three decades, but I am a recent addition to Moda Health, having joined the company in 2019. My arrival occurred not long after the financial floodgates opened to start finally prioritizing changes to the IT infrastructure that were long overdue. Hired to be the Senior Network Systems Administrator, I had a lot of work ahead of me, particularly as they had already started projects.

Then the pandemic hit. In addition to needing to manage the client end, Moda Health needed a practical solution to allow the majority of our staff to work remotely. Having put off upgrading the infrastructure for a few years, the IT department found itself with a lot of unplanned work.

It was not an enviable position, but ultimately we navigated these uncertain times. But let's not get ahead of ourselves. I'll start back in 2019.

The Starting Point

Until 2019, Moda Health has maintained three data centers in Portland, Bend and Milwaukie, Oregon. Knowing that we needed another data center, our company had approved $2 million to upgrade the IT infrastructure. 

As often happens when an area is finally given the funding needed, there were many ideas about what should be done, and soon the IT department realized that they had bitten off more than they could chew. They had spent the money to acquire the new technology, but it was impossible with the resources to do everything all at one time. With 225 hosts and around 4,500 VMs with 1.5 petabytes of data, they had to stop and prioritize what they were going to do.

It quickly became apparent they were going to need help, and not too long after that, Moda brought me on board. However, Moda Health also needed to find the right vendor to help guide us through the necessary upgrades to our systems. For every need, there were at least five different vendors. Finding the right partner was as much of a challenge as completing the upgrades, so finding a vendor that could assist for a majority (or all) of the changes was a definite preference.

The New Data Center

In addition to upgrading our current systems, it has been clear that Moda needed a fourth data center, something that they had not started before my arrival. In the healthcare industry, the amount of downtime companies are allowed is strictly regulated. Between October and March, we can be heavily fined for extended downtime for things like outages, making it difficult to schedule regular maintenance within office hours. 

The Portland data center was the largest, but it had no backup generators in the event of an outage. With contracts that don’t allow for any downtime, it was risky to not have a backup generator to ensure the system remained online. But trying to retrofit the building to add the generators would be far too costly. Establishing a fourth center would be more cost-efficient and could offer backups that would simplify maintenance.

There were a lot of requirements for establishing a new data center, including picking a location. With the primary location in Portland, we decided that we’d locate the fourth center in the suburbs of the city, Hillsboro, Oregon. Once that was established, the Moda team began looking for vendors who could provide the technical requirements. 

The IT team wanted the new center to have a single pane of glass that would make it easier to troubleshoot issues, instead of contending with ineffective finger-pointing. A streamlined HCI environment would simplify maintenance and make it considerably easier to manage than the old infrastructure in the Portland data center.

Our management considered both VMware and Dell HCI solutions, but the team eventually opted to use Nutanix. With Nutanix, we could easily perform maintenance and upgrades within regular business hours. We’d have to include some legacy components in the new data center, but we moved anything that wasn’t legacy to Nutanix.

All of these decisions and the project had began before my arrival, but I was there for the second half of the creation of the new data center. In March 2020, we were planning to begin the migration project, and it was originally scheduled to be done by the end of the year. We had established a schedule that would minimize downtime and were preparing to move forward when everything changed.

The Wrench in the Form of the Pandemic

As our IT team began the project ramp up, the pandemic hit, completely upending the plan. The focus went from the migration to rushing to allow staff to work remotely. At the time, only about 5–10% of our staff worked remotely. My team spent between four and six weeks building up the company’s virtual desktop infrastructure (VDI) to go from 75–100 remote workers to an average load of 650–700 users. 

This was a project that had not even been considered before 2020. Originally, we worked in a Citrix virtual environment. As the pressure mounted to improve the system, we began to run the virtual environment on Nutanix. It was a logical switch since we were using it elsewhere and we already knew a lot about what functionality it had to offer.

Even though we could meet the unexpected needs of 2020, it pushed back the migration project, especially as we had to continue everything in a live environment. Given what we had just finished, the team didn’t think we had any other way to accomplish migration without investing money in licensing with VMware or investing in another platform. 

Nutanix offered a solution with Nutanix AOS Data Protection. This solution allowed us to internally relocate our bare metal instances from the VMware traditional host to a Nutanix cluster. There was minimal downtime and robust data protection as we pushed data and verified it before cutting over to the other data center.

A Targeted Approach to Reduce Maintenance and Keep the System Online

Though we are still working through the migration, we have already seen a reduction in how long we have to spend on the systems' maintenance and upkeep. After the update to Nutanix, we found that we no longer have to schedule the regular maintenance for our hardware, firmware, and virtualization operating systems outside of normal business hours. 

To the IT team, this has essentially given us our weekends back because we no longer have to schedule the maintenance outside of regular work hours. Even though IT professionals are always on call, we mostly only have to respond to emergencies instead of doing our regular work outside of business hours. 

The IT team has made daily use of Prism Element and Prism Central. I’ve even become the go-to guy for Citrix VDI and Nutanix. I am incredibly curious about Nutanix Flow once we get their new switches and are able to use VXLAN, though that is still a ways in the future. Being able to do global disaster failover automatically has been one of Moda’s long-term goals, and we are laying the framework using Nutanix Metro Availability to bring that to fruition. We aren’t quite there yet, but we are already thinking about it because of our successes in other areas of managing maintenance while keeping the system online.

The Benefits of Training and Certification

One of the reasons I’ve become the go-to guy for Nutanix is because Nutanix offers training and certification. During the training, three of our seven-member team learned about features that we weren’t even aware were an option. As one of those trainees, I got to see a whole new side to what Nutanix offers. These features have helped us better customize projects to meet Moda’s needs. 

However, it was really Nutanix’s customer support that has impressed me. In my 27 years in IT, I’ve never seen anyone go above and beyond the way Nutanix has, and the rest of my team echoes this sentiment. Where many companies have customer support that will remain on the line to complete the initial troubleshooting, Nutanix’s support technicians will help with the entire process.

By the end of 2020, we’ve migrated roughly 85% of our environment to Nutanix. If I had to do it all over again, I suspect that we would include even more Nutanix hardware because it would be that much easier to streamline the maintenance. Not only does it reduce the finger-pointing that usually occurs when multiple vendors are employed and it is necessary to troubleshoot a problem. Nutanix just works. That kind of confidence is priceless.