Manage a Complex Environment with a Small Team—and Have Time to Spare

For every company that implements considerable changes to their network all at once, many other companies adopt big changes in phases. Sometimes a phased approach is preferred due to budgetary or resource restrictions. Other times, it’s because a product’s evolution leads a company to reevaluate how they do everything—and how they can do better.

At the Joint Industry Board (JIB), we have taken a phased approach to network changes, and as a result, we have become more efficient and more effective in our operations. 

The JIB is a private, not-for-profit company that acts as the administrative arm of the International Brotherhood of Electrical Workers Local Union No. 3 in New York City. We manage all the funds from union dues and provide benefits such as 401(k), education, and insurance and medical plans. It’s like a multifaceted financial and wealth management company for the union. 

On Our Way to Greater Efficiency, a New Challenge Reared Its Head

We used to be a big IBM Z Series Mainframe shop with a team of 12–15 COBOL programmers. It would take forever to get stats or information about the network because all of the mainframe datasets were flat files. It would take our programmers a week to write a query and test it just to get a report. 

Over time, we eventually moved to distributed systems. We began to build a physical Windows environment, and within four years, we had 60–70 physical servers. By synchronizing data in the mainframe to a SQL or ANSI compliant SQL database, someone writing an SQL query could take that week’s worth of COBOL programming down to a few hours’ work.

We were making good headway, but we encountered a new challenge a few years later: building a training facility. Becoming a journeyman electrician out of high school is a process that takes somewhere between four and five years. The JIB bought an old warehouse to convert into a convocation school for new electricians, and we had a year to do it. 

A New Solution for a New Problem

I came to know the JIB during my time at a consulting firm, where they were one of my clients. The JIB hired me in 2007, and I began my tenure here as IT generalist, managing all the physical servers and working on the help desk. As the company grew, so did I—I worked on network and system administration and eventually became the help desk manager. Today, I manage all of our Windows and Linux infrastructure. I also support our mainframe team, so I have my hands in both the network and server environment. We recently hired someone to help us build out our infosec team, and I work closely with them on infrastructure compliance. 

Creating a new warehouse training facility was a different kind of challenge than any we’d faced before. The warehouse was only 10 miles away, but 10 miles in New York City is quite far. I needed a solution to outfit, support, and remotely manage this 170-server-desktop training environment. That’s when our IT vendor, CDW, suggested I take a look at Nutanix.

I am a huge proponent of open-source technologies, and we use a lot of them at JIB. Nutanix uses the Kernal-based Virtual Machine (KVM) at the heart of its AHV hypervisor, which is right up my alley. Nutanix comes with AHV, so I didn’t have to buy a third-party hypervisor. That’s thousands of dollars in licensing fees I didn’t have to spend just to get started. 

Our needs were very basic for this training environment, and I found Nutanix simple and easy to use. Prism Central lets you manage your entire environment through a single console with a neat, clean interface. Prism also integrates with our other tools, like Solar Winds, to provide a holistic view of our environment. 

With all the checks in the “pro” column, I thought, “Let’s give Nutanix a try.” 

I was somewhat skeptical when it came to setting up our environment. Even though we built everything from scratch, it was a lot—a new Nutanix Citrix environment to support 170 clients. I thought for sure we would need two days. We started at 9 a.m. on day one and were finished by lunchtime. That was an eye-opener and a good sign of things to come.

The Evolution of Nutanix Led to a New Phase for Our Environment

Our first environment got off to a smooth start. In parallel, we moved to VMware, performing a 70:1 consolidation from physical machines to virtual machines based on Dell hardware. A few years later, it came time to replace that VMware environment and Nutanix was the first thing that came to mind. 

I’d had such a great experience with Nutanix, not only in the deployment of the first cluster but also in their mind-blowing rate of new AHV features released over the next several years. You put in a feature request, and six months later, it appears. That kind of response is unheard of among their competitors. 

We started with a straightforward use case, but the growth and maturity of the hypervisor have been outstanding. That’s why we went back to Nutanix to invest in a second environment—a production cluster—and a mirror image of it in our DR facility. The production environment was a little more complex than our first cluster, but we had full-stack support from Nutanix. I’ve worked with quite a few vendors over the years, and their support is by far one of the most pleasant experiences in IT.

Self-Healing, Self-Service, Backup, and Security

As we got deeper into our Nutanix relationship, we started to adopt a lot of lot of Nutanix’s automation and alert tools. If a machine doesn’t have enough CPU, for example, there are a lot of self-healing elements within Nutanix to rectify the issue. I’ll get an automatic email that identifies the issue and another email shortly after that states the resolution of the problem. Once configured properly, tasks like adding RAM, adjusting CPU values, and adding more resources are all done automatically. This automatic allocation is a big help, especially with some of our database applications that don’t require a lot of resources unless they’re in use. Nutanix manages all of that for me. 

We also use the built-in self-service portal, which allows us to build out environments for our development teams. Typically, you don’t give users access to servers, but our development team might occasionally need to do a reboot or something else that requires a little more control. The self-service portal does just that. It gives developers access to machines and a management interface to see what they need, without having to build it—and without having the power to do anything destructive. 

Nutanix also makes the backup and restore process much easier. Our old backup method was shipping tape: We had guys driving out to our DR facility to drop off our tape backups. We began to use Veeam, but we didn’t have a good replication link. Once we got the Nutanix cluster, we installed a gigabit site-to-site link that we now use to replicate. Today, asynchronous DR syncs all our production machines nightly to our DR facility. That’s great for testing because we can spin up a test environment that mirrors yesterday’s data in the blink of an eye. It also means that if we experience a disaster, I can restore environments in half a day, rather than the two days’ work with Veeam. 

Security is a high priority for us, too, so the fact that Nutanix follows the STIG guidelines and handles all the necessary patching is outstanding. Some vendors require that we do all the work to update patches and address security vulnerabilities, which places a higher burden our team. Nutanix removes that burden, even after deployment. I don’t have to worry, and I don’t have to manage it, which results in tremendous time savings. 

That time that Nutanix saves my three-person team allows us to focus on other infrastructure projects or even helping get our non-technical teams up to speed. Many of our departments have good technology in place but might not use it as efficiently as they could. Now we have the time to mentor them, which makes JIB more efficient in the long run. 

Even More Capabilities for Us to Explore

I want to continue increasing our technical efficiencies and putting more products and apps onto our recently expanded cluster. We’re starting to implement Nutanix Flow for microsegmentation. One of our goals for Q1 2022 is to start segmenting network traffic, such as separating the accounting department and their servers from the medical department and the medical resources they use. Flow will allow us to establish better governance around this traffic, which will enhance our security posture and regulatory compliance. We also plan to create some public-facing websites that will allow JIB members to access their records. Whenever we have records exposed to the outside world, we need to ensure our systems are as secure as possible. Flow will help us do that. 

Eventually, we want to get rid of our mainframe and move entirely to Nutanix. Once we do that, we want to start exploring Nutanix Xi Cloud to integrate a cloud space within our Nutanix environment.

Nutanix allows our small team to efficiently manage this huge environment encompassing 20 different departments, each with unique needs. For anyone considering moving to Nutanix, there’s no need to be hesitant. Anyone can trial Nutanix for 24 hours, spin up a test environment and see what it can do. I was one of Nutanix’s first 5,000 customers, and they’ve constantly expanded their feature set since then. It’s ever-evolving, and it’s been an incredible ride so far.