How to Find Peace of Mind with Your IT Infrastructure

If you are old enough to remember black and white TV, you may have a romanticized view of the dairy industry. You probably think fondly about chipper milkmen driving white trucks through quiet, small-town streets. Like the mailmen of the dairy industry, the workers would drop a few bottles of fresh milk at every door. They would then pick up the empty bottles left by the family of the home.

While not as popularized in the media, this hometown view of milk also had a production side. Most dairy farms during that era were family operations. Every day, the members of that family would rise each morning and begin milking cows and then process the milk. Most communities could name the family that ran the local dairy. Of course, that time has changed.

There are still family businesses, although an increasing number of operations are run by large corporations. Milk is now primarily distributed to institutions, businesses, and retail operations instead of family homes. Even modern family operations have become complicated structures managing multiple silos within the administrative and product areas. In today's milk farms, it is normal to find a hyperconvergence infrastructure next door to milk pasteurization and bottling processes. But then, maybe I've gotten ahead of myself.

A Company Rich with History

Founded in 1932, Producers Dairy has provided dairy products through three generations of the Shehadey family. The company was initially founded by four businessmen during the Great Depression population rush to California. It was purchased by Larry Shehadey in 1949, and ownership has been passed through the family since that time.

Through the leadership of the Shehadey family, the company has experienced a number of triumphs over the years. During the 1950s, the company was one of the first to move to a 100% refrigerated fleet. This gave the company an opportunity to extend transportation routes and therefore increase production. They were also one of the first to switch to paper cartons in the 1960s. This dedication to constant improvement is a big reason why they have survived to modern times, unlike many other dairy farms from the central valley California region.

While having this rich legacy can be helpful, for many organizations it can be a hindrance. Doing things a certain way for decades sometimes leads to reluctance when change is needed. Unfortunately, even when small changes are made piecemeal for decades, it can also lead to a need to completely revamp the system. That migration to a completely different infrastructure and design is scary for any organization.

This is where I came in. Prior to Producers, I spent a number of years working for Fortune 500 companies which had a global presence and impact. Since my background is in both hardware design and software development, I enjoyed helping multiple clients create architecture solutions, and manage analysis and integration. The challenge of scaling and integrating was part of my core competency. After relocating from the east coast of the US to California, I was looking for a change.

Producers Dairy was a name that resonated in the city we now called home. So when I saw an opportunity, I was curious to see how I could help. Producers had an antiquated hodgepodge of both hardware and software stack cobbled together. Each piece worked for one purpose when it was purchased, but was never maintained or pruned to continue working with the constant change that the nature of the business brought. They knew the setup was holding them back. They needed IT expertise to re-position their technology to support their operational dreams.

I knew nothing about milk, but I loved that they were family-owned with a history of integrity and purpose. Together we decided we could help each other, and three years ago I embarked on my new voyage in the dairy industry.

The Modern Challenge

If you haven't been following the milk industry, there are some things you should know. The US dairy industry is still the largest in the world, but milk drinkers consume half as much as they did in 1975. As you might imagine, this has significantly reduced the bottom line for many companies. As a result, several large milk-producing operations have already filed for bankruptcy in just the last few months.

That unfortunate trend meant our company could no longer afford to keep operating so inefficiently. Adaptation requires a nimble operation, with the ability to analyze performance at multiple processing points. Producers had a number of remote distribution sites that were completely unconnected to the central administration. We also had two processing plants which were depending on being on sync for efficient operations. Getting weekly reports about production, deliveries, or financial transactions from remote sites just doesn't work in 2020. We needed a way to streamline operations across plants, optimized milk production and delivery schedules.

Milk processing is also very different from other industries in the need for tight turnarounds. Dairy has expiration dates. If you sell cars or plumbing equipment, you can potentially survive disruptions that even last days. You can stack material in a warehouse and begin processing deliveries when service resumes. With the dairy industry, however, once you milk a cow, you have to get that product moving through your process. Any delays can quickly begin adding up to potentially millions in lost product.

The combination of old technologies and tight deadlines meant missed weekends and loss of value product. Many IT professionals understand being on call. But the constant worry of being called in at any time can quickly become a lifestyle where you are never far from work. While we loved my job, we wanted to make it more reliable and efficient. My goal was to make sure our teams got their weekends back and we operate with high efficiency and throughput.

Fortunately, the owners at Producers were on board with the change. They knew we needed to do things differently. They knew where they were, and they knew where they wanted to be. They just needed help to get their technology to that point. 

Making the Change to HCI

I started at Producers Dairy with a 25-year-old AS/400 system and a traditional Dell stack which was dangerously close to end of life. They were all running at close to 80% capacity in both computations and storage in our data center. It was almost impossible at that point to do any kind of updates because we were already running near capacity. It became part of my regular job to focus on remaining calm. If people knew how close I was to freaking out about a potential total system crash, it could panic others. In addition, the technology systems in each of the distribution sites operated separately, using whatever internet was available in that area. There were multiple projects already in flight to streamline our technology stack and our network infrastructure. 

We started researching a number of options, including products from every hardware and software company available. We knew that whatever change we made had to work with minimal disruptions. This was not going to be a piecemeal solution. We were adamant that every piece of equipment or software we bring in needed to work for Producers Dairy as a whole. We also knew that whatever technology we wanted to purchase would have to definitely be worth the required investment. After all, Producers is in the dairy business, not the technology business.

Through our partner McMillan Consulting, we eventually landed with HPE SimpliVity. After many technical discussions and sizing exercises, they suggested that we attend an HPE Insiders event in San Jose. Communicating with people already using the systems in their businesses helped us to understand how things would work outside the proposals on paper. These groups convinced us to try HPE SimpliVity, which is a hyperconverged platform merging storage, data management, and VM management. 

We also loved HPE InfoSight, which would help us gain visibility into our virtualized infrastructure. We had used Nimble for years so we were already familiar with the value it added through its dashboards. InfoSight would give us a quick snapshot of any problems in our systems and help us isolate issues. We'd always wanted that level of granularity with our server infrastructure and we'd finally get it with InfoSight. Even though InfoSight was not offered at the time of our implementations, it definitely helped to know the road map and transparency from the HPE engineers during the Insider events.

In all honesty, our biggest reservation about shifting to HPE SimpliVity was belief. Between the sales pitch and the HPE Insiders event, the promises made by the system were nearly unbelievable. We found ourselves embracing the change, but ultimately waiting for the catch.

70 VMs, Zero Downtime

They say that every day you can learn something new. I certainly learned something from the migration process. Based in part due to the functionality promises, I expected the migration process to be long and pain-inducing. Although I kept my professional calm, I know the owners at Producers were also nervous about the impact the transition downtime would have on operations. I learned that updating to new technology doesn't have to be a complicated process.

We were able to transfer nearly 70 virtual machines to HPE SimpliVity with zero downtime. Many of our staff were still waiting for the warning messages and involuntary break times when they discovered we were already finished. We also discarded literally stacks of old servers and other equipment by consolidating footprint to HPE.

With HPE SimpliVity, we're running backups every hour and still taking up the same amount of space as if I took a backup every day with the old system. And we can now test out version upgrades in minutes. Previously, we had to schedule something like that, but now cloning a VM is just a click. We almost didn't believe it at first. We thought, "You can't clone a 500GB VM in 12 seconds. Something isn't right." But it was. 

The HPE SimpliVity system unified our data storage, processing, problem-solving, and server functions into one easy to use interface. We had unprecedented levels of access to performance and statistical information at our fingertips. We were able to monitor operations at our distribution sites within one screen.

Most importantly, everything worked exactly as promised. Our topology is a stretch cluster of two extra-large nodes for corporate and two small nodes for our plant operations, with HPE Composable Fabric on top. This stretch cluster runs about half a mile. Up next, we are looking for an offsite disaster recovery option that will provide us with more redundancy. 

Embracing Calm

Once the dust settled, we began to understand the impact of the change. Not the data measures—but the everyday impact. Leaving for work on a Friday evening with your storage and processing servers all pushing 85% was a nerve-wracking experience. 

How can you really enjoy your weekend knowing that your systems are so close to failure? HPE SimpliVity's virtual system and AI make managing storage and processor allocations nearly automatic.

I no longer consider it an official part of my job to remain calm when problems arise. I am still calm, but now I am okay because I know I can analyze any problems and develop a solution. I can quickly review historical performance charts and discover the point of disruption. The scalability of the virtual storage also means I don't have to worry about a sudden influx of data or a program update crashing my system. But best of all, I can finally enjoy my weekends.

History is an important pillar for a company like ours to rely on. Many people love and know our name. But that history doesn’t need to stay in our way when it comes to technology. Thankfully, our infrastructure has moved into the future.