How HCI Helps Our Credit Union Compete with the Major Credit Unions and Banks

Central One is a federal credit union serving four communities in Central Massachusetts and growing. Our footprint is small, but our impact is big, and we strive to give our members the same level and quality of service as the major banks.

We're all about metrics. Every month, we mail out member satisfaction surveys and our board crunches the numbers, so we know exactly how we're doing. I'm happy to say that our members consistently rate us at 97–98%, which is a testament to the hard work of our frontline staff and the success of our quality initiatives. 

World-Class Service and a Decade of Growth

We strive for world-class member experience, and that is part of why I took this job. I've worked at three other credit unions because I believe that co-operative finance offers my fellow citizens the best value for their money. Central One takes that approach one step further by listening to our members and providing them with financial products and services that address their day-to-day needs and also allow them to dream and plan for the future.

In the last ten years, we've grown mostly through word of mouth. We take care of our members, and they are eager to share their stories with the wider community. As we expand our customer base, we can offer new and better services, and this fuels a cycle of continuous growth. 

When I came to Central One in 2013, we were struggling with our IT infrastructure. We were experiencing problems with speed and capacity, and we lacked a proper disaster recovery (DR) setup. We had two-and-a-half racks of gear and several towers, and were running 32 aging physical servers that were prone to overheating.

Things got hot in the summer, and we had to crank the air conditioners to lower the data center temperature below 80°F. Every August, without fail, our primary A/C units would blow, and we had to power up the auxiliary unit in the back. It was not a good feeling.

Our servers were six years old and past their usable life. Our primary data center was our only data center, and as a result, our infrastructure did not comply with NCUA (National Credit Union Association) guidelines for information security. Don't get me wrong, we were replicating our core data, but we weren't following best practices or using the most effective technologies.

Another issue was scalability. We run Oracle here, and each new database required us to spin up a new server to host it. Adding new physical machines took too much time and money and hampered our ability to expand the scope of our services and run an acceptable DR solution.

Hyperconvergence Was the Right Solution

I'd already worked with the typical EMC infrastructure and had refreshed my previous employer's Dell EMC VNX installation. I knew hyperconvergence was the right fit for Central One, but what was it (in 2013)? I needed to do my homework before I could present a potential solution to our board.

I checked out EMC's latest offerings, but when I told them I was using Oracle, they froze and then threw more hardware at it. I also looked at Nutanix, and considered a hybrid solution that combined their storage with traditional hosts. But nothing met our specific needs and both solutions added a rack of equipment.

There was also a question of budget. Central One's board had already allocated the year's funding for infrastructure upgrades, and I had to find a solution that fit those financial constraints. The only platform that met all our needs and didn't really exceed our budget was HPE SimpliVity. The company was a relative newcomer headquartered in Westborough—one of the towns where we do business—and the installer was also a member of our local Corridor 9 Chamber of Commerce.

I met with the board in March. Our original plan was to upgrade our primary data center in 2013 and to tackle the creation of a DR site the following year, because this was all we could afford with other solutions. But HPE SimpliVity sweetened the deal by offering to build our disaster recovery data center solution for just an additional $10,000 above budget. That was more than enough to sway our directors, so they gave me an immediate go-ahead. 

A Leap of Faith and Screaming Machines

It was a leap of faith for me and Central One. We were HPE SimpliVity's 19th install (right around there), and possibly their first production environment. It was a whole new ballgame for everyone involved. We didn't have any reference calls we could make. The only people I spoke to were the delivery guys and the installers. They assured us that everything would work, and it did.

We started with a standard two-plus-one setup. We installed two HPE SimpliVity nodes at our primary data center and a third node at our DR site. By the time we'd virtualized all our older servers and transferred our databases to the new hyperconverged infrastructure, we were using half the physical space at our production site and way less energy.

In less than 2 years, we'd doubled the number of VMs from 30 to 60. We could spin up the needed VMs in minutes, and this allowed us to meet our increased needs for capacity as Central One's membership grew.

Our systems started screaming, but we hit an immediate bottleneck. HPE SimpliVity (I refer to them as OmniCubes) talk to each other over 10-gigabit connections, and our switches weren't even one gigabit. We engineered a new internal switch stack and kept going from there.

The Aftermath of a Cryptolocker Attack

I was bullish on VDI in 2014. As I was venturing down that path, I had asked the board to approve the purchase of two more HPE SimpliVity nodes, even though there was no IT budget for them. The response shocked me: "Why didn't you ask for more?"

I believe they were happy with the improvements from the original decision, and they were especially pleased with the disaster recovery tests from the previous year that had allowed us to meet NCUA standards or they just wanted to get rid of me. Either way, as a result, they green-lit the extra hardware. 

Some of our frontline employees were still using Windows XP machines, and I felt that moving to virtual desktops would better secure our endpoints. Unfortunately, I dragged my feet on implementing this project, and by August 2014 we paid the price. A frontline employee accessed a local sports site through one of our XP boxes, and clicked on an ad that unleashed a cryptolocker attack.

We didn't know what hit us. Transactions were still going through the branches because those systems were unaffected, but our back-office applications were down. We thought it was a software issue, so our applications team took over. That was at 3:15 in the afternoon.

I was sure we were in good hands, so I left to coach a baseball game. When I got back that evening, my people were still dealing with the problem, and so we dug deeper. It was taking a while to copy a duplicate file set of data from another branch to one of our Production servers. (Old School mentality—You're my boy Blue")

This took what seemed forever, so I asked my network manager to restore the target server from the backup we'd made at noon with HPE SimpliVity, which took one minute. (The New Way of Thinking)

The same thing started happening doing it the old way.  As we dove in again and found a file labeled "cryptolocker," which we traced back to the XP machine at one of our branches. We then pulled the plug on the affected server and that desktop and restored the file share backup from noon. 

It took us a while to figure it out (four hours), but once we did, we were up and running twenty minutes later. Thanks to HPE SimpliVity, we only lost three reports that were generated between noon and 3:15 PM when the cryptolocker hit. Needless to say, we upgraded everything to Windows 7 within a week or so and renewed our efforts to move our front office to VDI.

Taking a Second Look at HPE SimpliVity

By 2019, we had four HPE SimpliVity nodes in our production environment and another two at our DR data center, but it was time to take a second look at our options. Our first set of HPE SimpliVity nodes was nearing end of life. We were also running out of storage at our DR data center due to backups of Production and running Test environments there. 

We looked at the usual suspects again, but every other solution required third-party add-ons to do the same work as our HPE SimpliVity. We would have paid more for hardware, software, and training to start over.

Right now, I can teach my IT people (and some non-IT) everything they need to know about HPE SimpliVity and VMware in a couple of days. I don't have to send them off for six weeks of offsite training.

I can also set up multiple test environments in minutes to make sure upgrades and new applications run smoothly. I'm building 30-server bubbles that are exact reproductions of our production environment in under 20 minutes, utilizing an HPE partner's application, and if things go wrong when we try something new, I can wipe them and start over until we get everything right.

We even hired a full-time training manager for our frontline workers. She uses virtual machines in an HPE SimpliVity test environment to train staff on how to handle real-time transactions in a full-blown sandboxed replica of our front-office environment.

In short, HPE SimpliVity gives us access to the same tools as the country's biggest credit unions and banks, but on a smaller scale. As a result, we can continue to innovate and to always up the level of service and the quality of financial products we can offer our members. 

An Evolving Partnership

We started with SimpliVity before HPE acquired it. There were a couple of hiccups during the transition, but our business relationship has always been a partnership. It never felt transactional. HPE and our third-party supplier have worked together over the years to provide consistent and timely support. They are approachable, and I'm sure they give us the same quality care HPE SimpliVity offers our Wall Street competitors.

At the same time, HPE is continuously innovating, and their post-acquisition SimpliVity products continue to astound me. I'm especially eager to start working with InfoSight. HPE's new predictive analytics platform will allow us to take a deeper dive into our infrastructure and to better support and secure our network. I think InfoSight will allow me to flex some IT muscle and be more proactive than we have been in the past. We're not quite there yet and are still working on completing our latest round of infrastructure projects, but HPE InfoSight is definitely on our radar.

In the end, Central One is providing outstanding service to our members. HPE SimpliVity is helping us achieve this goal. It helps keep us flexible and adaptable, which is exactly what our members need from us. A satisfaction score of 98% is great, but it means we’ve still got 2% more to go.