The Town of Mansfield’s Unexpected Journey into Hyperconvergence

Sometimes, life presents us with opportunities we didn’t ask for. At the Town of Mansfield, we were forced to upgrade to hyperconvergence at a time when we weren’t expecting it. It was a surprise, but in hindsight we’re glad we made the switch.

Mansfield is a local government located a 40-minute drive southwest of Boston. Our state, Massachusetts, places very specific requirements and procedures for most operations at the local-government level—including IT environments. Our network spreads over seven sites, with the town hall at the center, and everyone else connecting through us. We support all local departments, with the exception of area schools. Otherwise, it’s all on our team of three: IT engineer Doug Collette, systems administrator Roberto Mendoza, and me, director of IT.

A Tale as Old as Our Infrastructure

We started experiencing problems two years ago as a result of our aging infrastructure. Doug and Roberto spent a lot of time taking care of our VMware ESX hosts and making sure they operated adequately, because we had over 30 virtual servers on this system.

Our SANs were iSCSI and it was complicated to create and expand iSCSI volumes. In particular, we always had to fight capacity issues. We rarely kept more than 10 or 20 gigabytes of free space on any virtual drive because we were trying to minimize our storage footprint, which has always been one of the most critical items for us.
 

Storage is expensive and not easily upgradable. When we ran out of capacity on our LeftHand SAN, we replaced it with StoreVirtual. But then we started to run out of capacity there, too. Managing storage was always an issue for us. If we were running low on storage space somewhere, we had to log into the StoreVirtual management center and reconfigure the storage independently of the virtual environment. It was fragmented.
 

In addition, our IT team works regular business hours, but Public Safety, one of the departments we support, works around the clock. When we are done for the day, our network equipment must continue working because Public Safety is a critical function. Whether LeftHand or StoreVirtual, we’ve always used storage technology with mirroring between sites and automatic failover. This automatic failover capability has kept our systems up on several occasions when we lost a storage site overnight.

The Perfect Opportunity for Hyperconvergence

Our StoreVirtual SAN was running at over 90% capacity and needed to be expanded. Unfortunately, at that time HPE had made a decision to discontinue StoreVirtual and proposed us moving to a hyperconverged environment. That came as a surprise to us, because we had been using these appliances for only five years and now we would have to change our entire environment. The good news was that we were already acquainted with hyperconvergence. 

We had discussed hyperconvergence several years before when it started becoming popular even among mid-size businesses, but our current infrastructure was meeting our needs and we couldn’t justify an expensive “rip and replace” strategy. Not being able to expand our StoreVirtual SAN technology changed our minds.

Our host servers were eight years old and no longer supported on VMware 6.5 and our SANs were almost out of capacity, so we justified this being the perfect time to make the move to hyperconvergence. Moving to a new technology like hyperconvergence would require a forklift upgrade and since we needed to replace the servers and storage anyway, this was an opportunity too good to pass up.

When we searched for solutions, we looked at Nutanix, Dell VxRail, Pivot3, vSAN, StorMagic, and HPE SimpliVity. Our number one priority was that the solution has automatic failover with built-in backup, because of how we support Public Safety. HPE SimpliVity was the only product on the market that satisfied these criteria.
 

Doug was attracted to HPE SimpliVity’s deduplication capability while Roberto, as a system admin, found HPE SimpliVity’s ability to easily manage different sites from a single interface appealing, as well as the built-in backup capabilities. As Roberto pointed out, we already had a backup infrastructure, but with HPE SimpliVity we could do it all through one interface. To us, that was the biggest selling point. Roberto loved how a one-terabyte backup would come online in a matter of seconds. With this system we would have a very robust backup system in place.

Although it was only slightly more expensive than the lowest priced competitor, HPE SimpliVity outnumbered the other choices in features. 

In hindsight, Roberto supposes we probably could have done the installation ourselves, but in the end, it was a plus that we brought in an HPE-certified technician to help with the migration. When we spoke about it recently, to Doug’s point, it was sold as a turnkey project, both hardware and installation services. HPE wanted to guarantee that HPE SimpliVity went in smoothly without lots of issues, and it turned out to be a flawless process. 

A Faster System with More Granular Backups

The big thing we remember from our early weeks with HPE SimpliVity is Doug coming to our group meetings so excited to report our deduplication ratio. In December of 2018 it was 200:1. Now it’s around 285:1. We’re only at 40% usage of our HPE SimpliVity storage because of that ratio. Clearly, HPE SimpliVity is an incredibly efficient system and our experience proves it.
 

The tremendous dedupe we get with HPE SimpliVity means we can keep adding to our system and our storage footprint only slightly increases. It also makes everything faster—and this lightning fast speed is something everyone notices.
 

To give you an example of what we mean, shortly before the migration a person in the treasurer’s office complained that every time she ran payroll, it took longer and longer. Our SQL Server was getting bogged down. But after the switch to SimpliVity, the person who usually complained about the sluggishness of the system said, “This thing is great! It went right through.” She noticed a huge improvement in performance running on HPE SimpliVity’s all-flash storage versus our old spinning disc.

With our old system, a full backup of all servers took between seven and eight hours, so we used to only do nightly differential backups. When we first started with HPE SimpliVity, we continued with that schedule and observed each backup took approximately one minute. When we increased the backup frequency to every four hours, we noticed almost no change in the amount of storage required. We now backup even smaller time frame, meaning if we ever run into a problem, our window is only an hour or so old at a maximum. HPE SimpliVity has given us much more granularity in our backups, thus improving our service levels.

To clue you in on what recovering a large backup looks like in action, we’ll relay an incident that happened not long ago. We recently had a network issue that resulted in corruption of our primary SQL Server around 9:30 AM. We were unable to quickly resolve the server problem so we restored our 8:15 a.m. SimpliVity backup and it took about 40 seconds to restore this 950 GB server. With our old backup system, recovering a near-terabyte server would have taken somewhere in the range of five hours and we would have lost over half a day. With HPE SimpliVity, we were up and running again in under a minute.”

With Everything Easy and Stable, We Can Refocus

Our day-to-day work is more structured now. As the system admin, Roberto used to spend his whole time working with the servers. “But with HPE SimpliVity, I can actually work on helping people with their problems,” he says now. “I can plan better knowing I can work on the servers whenever is convenient for me. And I have peace of mind knowing I won’t have to rush to troubleshoot a problem.”
 

The unified, single pane of glass to control both the virtual environment and the storage is a strong selling point for Doug: “In the nine months since we’ve had the system, we haven’t had to increase our storage allocation and I expect it will be a while yet. But when the time comes, HPE SimpliVity makes it much simpler. We can expand storage right in the vSphere interface instead of logging into a separate system. I don’t at all miss having to deal with iSCSI volumes.”
 

One of the features we use is what HPE SimpliVity calls a stretched cluster, which is the synchronous mirroring of the storage between two nodes. It’s an extremely robust self-healing system. We needed automatic failover to satisfy our Public Safety requirements, but as mentioned earlier in this article, the other vendors either couldn’t do it or they needed bolt-on software that wasn’t synchronous. With HPE SimpliVity, if one node goes down, the VMs automatically spin back up on the alternate node. This is crucial for us since we don’t have to worry that the system might go down in the middle of the night.
 

HPE SimpliVity gives us stability and allows us to plan our next steps. Now we can focus on cloud backup and disaster recovery. In the past, we wanted to prioritize these upgrades, but we never had the time or resources to even think about it. HPE SimpliVity lets us move forward, because we’re no longer constantly putting out fires.

New equipment can be financially expensive, but when you’re a team of three people, trying to make do with what you have can be costly in terms of your time. HPE SimpliVity means we can refocus our attention. Hyperconvergence quickly became a standard for us and hopefully to many others in the local-government IT world.