My Hyperconvergence Journey Had Its Bumps: Here’s What You Can Learn From It

Sometimes, things can go horribly wrong when you're upgrading your IT infrastructure. Despite a rigorous RFP process, months and planning, and dozens of meetings between your vendor and your engineers, you can find yourself sidelined by a simple oversight or circumstances beyond your control.

But a shaky start doesn't have to end in catastrophe. As we discovered here at Frandsen Financial Corporation, you can learn from your mistakes. After setting off on the wrong foot with our hyperconvergence journey, we course-corrected, and the eventual outcome was a system that far surpassed our initial goals and expectations. In this story, I’ll tell you how we turned it around. 

Today’s Banks Are Technology Providers

Frandsen feels like a small-town bank, but is, in fact, a large regional financial institution with $1.7 billion in assets. We operate in small towns and mid-size communities and take pride in offering highly personalized services to our customers.

In many ways, we're a technology provider. The majority of our younger customers do everything online and rarely come into our brick-and-mortar locations. Still, we strive to offer a personal approach, even when they're banking on their mobile smartphones, tablets, and computers.

To stand out from the competition, we strive for zero downtime. Chase and Western Union are big enough to withstand their services being offline for a few hours. But our customers expect better because we make an effort to build individual relationships. It's part of our value proposition.

Our challenge is to provide IT infrastructure that is flexible enough to give our branches autonomy, sufficiently robust to give our customers access to our services, and secure enough to meet regulatory and business requirements. We are always looking to do better and to improve the customer experience, be it online or at our physical locations.

Time for an Upgrade

At the beginning of 2016, we realized our infrastructure was holding us back. We were running on a Cisco FlexPod environment that was always in need of upgrades. We were constantly adding blades and storage shelves. The process was supposed to be plug-and-play but wasn't really. Every time we needed new hardware, we had to do a lot of engineering and patching.

Increasing capacity was too slow and complicated. IT couldn't keep up with customer demand, and we couldn't adequately support any mergers or acquisitions.

We were also struggling with disaster recovery. We had an offsite backup data center, but it was mostly storage with very little compute. Our data was protected, but we were unable to restore it without adding new hardware to our DR data center. We had a disaster recovery plan in place, but we never tested it due to this limitation.

Thankfully, we never experienced a system-wide failure because it would have taken days to get everything back online with this previous configuration. 

Starting From Scratch with Hyperconvergence

At the time—and to this day—CDW was our solutions provider. We looked at swapping out our blades and our storage shelves with newer versions of the same technology, but when we drilled down to the minutiae, it seemed like a losing proposition. So I asked, "What if we started from scratch? What would you recommend?"

The answer was hyperconvergence, and it opened up a whole new conversation. I wanted to know what it was and what it could do for us. After a little more digging, the advantages of hyperconverged infrastructure were clear: It was easy to deploy, easy to administer, easy to expand, and it was very fast.

We sent out an RFP and talked to four vendors: Nutanix, Cisco, Dell (VMware), and SimpliVity (this was before the HPE acquisition). As this was happening, we were dealing with performance issues because one of our switches was failing. It took us seven months to resolve the problem, so we decided we were through with that level of complexity. We were through with RAID 6 and wanted a RAID 10 solution that combined mirroring and striping.

Seeking My Peers’ Opinions

Whenever a new technology comes along, you never know if it’s just a gimmick. Hyperconvergence could have easily been one of those things that overpromised and underdelivered, but my fears were quashed when I started to talk to other professionals in my field.

A few years ago, I formed a peer group of CIOs who work for banks our size here in the Twin Cities. We meet once a quarter to talk shop. I was the only one in the bunch who wasn't adopting some form of hyperconvergence. I sat down with a good friend who had just transitioned his bank to a full-blown Nutanix production environment, and he confirmed the benefits of hyperconverged architecture.

I learned a lot from him, but diverged in one big way: When it came time to make our choice, we went with SimpliVity because it was easier to set up and offered a true RAID 10 environment.

Roadblocks to Our Success

In the spring of 2016, we had a roadmap, and we were ready to embark on our hyperconvergence journey. That's when things started to fall apart. Let me be clear here: It wasn't a full-blown catastrophe. We hit a couple of roadblocks and made a couple of mistakes. Getting back on track took some effort, but we managed to pull everything together, and then some.

The first of our challenges was organizational. We signed our contract the week HPE acquired SimpliVity. We had negotiated and engineered a solution with their team as it was being absorbed into a bigger entity. Naturally, there was some confusion about roles and responsibilities.

Our next hurdle was the weather. A massive rainstorm hit Houston in April of 2016, and the factories and warehouses supplying our gear were flooded. HPE sourced parts from their Denver warehouse, and the components they delivered didn't quite match what we'd ordered.

We hadn't even started to rack up our new data center, and we were already pulling our hair out. We wondered what we had gotten into. Were we going to stay with HPE SimpliVity or would we move to our second choice?

Fortunately, we sat down with HPE and ironed everything out. I have to single out my personal friend, Dan Miller, who went to bat for us. He's HPE's field territory manager for Minneapolis-St. Paul, and he knew how hard Frandsen, SimpliVity, and CDW had worked to build a solution.

The flood and the buyout forced us to delay our rollout by a couple of months. Everything was supposed to be in place for September, but we finally powered up our new data center in November 2016. That's when we realized we had undersized our system.

A Good Partner Makes Things Right

It may have been a by-product of the HPE acquisition, but we mapped out our infrastructure as the merger was taking place, and in the process, we missed including a massive file share on our old network. We had ordered only 18 terabytes of storage when we needed 25 or more. 

Once again, HPE came through and cut us a deal on the expansion nodes we needed to correct this oversight. At this point, the new reality hit me. SimpliVity had been fully absorbed into HPE's sales and engineering hierarchy, and the company was throwing its entire weight behind the new division. They were doing everything in their power to make things right.

We went from two production nodes and a single DR node to four nodes in each of our data centers at a substantial discount. Talk about standing behind your product. Here again, I have to give credit where it's due. I also received tremendous help from Charlie Roberts and Russ Campbell, our account executive and our data center solutions architect at CDW.

So Simple, a CIO Can Use It

I mentioned earlier that one of the reasons we chose HPE SimpliVity was its ease of use. A crucial feature that sold us on the platform was its block-level backups, which allow quick-and-easy restores. 

In July of 2019, we were upgrading our phone system while our system administrator was away fishing in Northern Wisconsin. During the process, we lost the folder that contained the employee profile pictures used by our desktop telephone client.

I reached out to my desktop administrator, who was the most senior IT staff member on duty that day, and we went through the backups together. We were able to find and restore the folder within fifteen minutes, and the upgrade went forward without a hitch. HPE SimpliVity is so simple even a CIO like myself can use it. I’m only partially joking when I make that claim.

Speed vs. Value

A few weeks after that, we took another step on our HPE SimpliVity journey. The company announced new hybrid modules that incorporated SSDs and traditional mechanical hard drives. The main advantage of these new modules is cost, but their introduction also presented an opportunity to once again expand the capacity of our production environment.

We decided to move our four flash-based HPE SimpliVity nodes from our DR data center to our production data center, thus doubling our compute and storage capacity. The logic behind this is simple: Speed is not the primary function of our DR data center. In the event of a system-wide failure, the use of mechanical hard drives will have a negligible impact on recovery time.

Think of it this way: Flash-based HPE SimpliVity nodes are the Ferraris of the computing world. They get you where you're going fast, whereas the hybrid nodes are like Mack trucks that haul your stuff to wherever you need it to be.

Let's face it—you can never have too much capacity. We originally budgeted for 23 terabytes of storage in our production environment but ended up needing 30. Recently, we had to move a seven-terabyte database within our HPE SimpliVity infrastructure. To do this, we used 21 free terabytes because we had three instances going at the same time as we fine-tuned and finalized the migration.

Now, we have 60 flash-based terabytes at our disposal, and our production environment is operating at 25% capacity. This setup leaves us with the space we need to test things and move them around. We also have plenty of room to grow over the next two or three years, which means that Frandsen can pursue a merger and acquisition strategy without having to worry that we lack the tools to integrate the IT infrastructure of any financial institutions we might absorb. 

The Final Result

To further stabilize our infrastructure, we moved our primary data center offsite to a colocation facility in South Metro Minneapolis. We save on cooling and heating, and we have redundant power and data lines. Thanks to this new setup and our DR data center, I can sleep easy knowing we can handle any crisis.

When I look back at the past two years, I am proud of what Frandsen's IT team has accomplished. We replaced our aging infrastructure with a cutting-edge hyperconverged solution, and we literally weathered a storm as we built up and rightsized our HPE SimpliVity implementation.

The moral of this story is that people make or break technology. We could have walked away at any time, but we had confidence in our partners. Everything could have fallen apart were it not for the dedication and hard work of my team and our colleagues at CDW and HPE SimpliVity.

My team is now free to add value to the company. Instead of managing our infrastructure, my system administrator is spending more time bolstering security and documenting procedures. We are no longer focused on the minutiae. Instead, we are looking at the big picture, and we are building new applications for our internal users and our customers.

Thanks to HPE SimpliVity, Frandsen can continue to give our customers the personal service they've come to expect as we continue to grow.