Turning the Retail Apocalypse into a Retail Boom with Advanced Analytics

We are living in a time of commerce disruption. Online giants like Amazon have changed the way people shop. One-click ordering and next-day delivery make it easy for people to purchase the products they need from the comfort of their home, their office, or using their smartphones when they're out and about.

The shift to e-commerce has made it harder for brick-and-mortar merchants to compete—and many traditional businesses have folded—but the retail apocalypse is not inevitable. There is something to be said for presence. Shoppers still find reassurance in the ability to see and touch a product and in having face-to-face conversations with a knowledgeable sales associate. User reviews and chatbots are no substitute for being there.

The World’s Leading Customer-Tracking Solution

ShopperTrak is the world's leading provider of retail customer-tracking solutions. In fact, we invented the business. Our company was founded 20 years ago by a group of engineers who invented a device that counts the number of people who are entering and exiting a store.

You've probably seen one of our devices at the places where you shop. They look like cameras that are facing down and are mounted at every store entrance. Using this monitor as our foundation, we started building a retail analytics service that provides business intelligence to some 2,000 companies. Our current installed base consists of 150,000 of these sensors that track the habits of some 40 billion shoppers around the world.

To increase our data gathering capabilities, we have also started using Wi-Fi sensors at retail locations to track the movements of shoppers who are carrying Wi-Fi enabled devices like smartphones and tablets. To maintain compliance with privacy legislation like the GDPR in Europe, and California's Electronic Communications Privacy Act, we don't collect any personally identifiable information (PII).

In most cases, we install 10 to 14 devices of both types to cover an entire store. As you can imagine, this generates a lot of data, some of it useful, but much of it not. The question is this: How do we separate valuable data from the noise, and how do we then convert it into actionable insights?

Adding Data Analytics to Our Customer-Tracking Solution

The solution is adding a data analytics layer to our existing combination of perimeter and interior sensors, and to our ShopperTrak Retail Analytics Platform (STAN). To help with the BI of our platform, we chose Qlik Sense for its easy integration with our Oracle, MongoDB, and BigQuery databases. We then authored a series of Python scripts to extract clean data from these sources, and used Qlik's NPrinting app to generate and distribute data visualization reports to our customers.

Qlik Sense and NPrinting have empowered ShopperTrak to go beyond static performance reports that only provide information about customers' movements and their interaction with in-store reps. We can now use this data to generate dynamic reports and to supply our customers with solutions to business problems that are hampering their sales.

Qlik Sense and NPrinting have also greatly accelerated our customer onboarding and report generation processes. In the past, it would take four to six months to add a new account, create a new report, and generate its initial output. The process was slow and laborious because we did everything manually with Excel scripts.

Now, we're up and running in three to four weeks, and we can continue to refine our reports with extensions. Qlik Sense supports complex calculations, as well as flexible access and aggregation levels. These features mean that we can onboard clients with anywhere from 10 to 11,000 stores with equal ease.

Using Traffic Analytics to Boost Sales Per Shopper

Let's look at a couple of ways we've used Qlik Sense to generate dynamic reports that provide actionable insights—including business practice and labor model improvement recommendations—for our retail customers.

The first is a Quarter-to-Date Sales-per-Shopper (SPS) report we generated using a combination of sales and traffic data for one of our customers’ stores. When we crunched the numbers at this location, we determined that the SPS of $12.53 was $7.68 below what it should have been. This figure was based on recorded conversion rates and average dollar sales (ADS).

We then looked at customer traffic figures and saw that the busiest four-hour periods on any given week were Saturday from 3:00 p.m. to 7:00 p.m. and Sunday from 2:00 p.m. to 6:00 p.m. The total traffic during these eight hours accounted for 21.7% of the customer volume in a seven-day period.

To increase conversions and ADS during these periods—and therefore overall SPS—we recommended that our client staff these times with their best sprinters, i.e. those sales associates who can serve the most customers at once.

We also determined that the two lowest four-hour traffic periods (Sunday 6:00 p.m. to 10:00 p.m. and Tuesday 9:00 a.m. to 1:00 p.m.) account for only 2.2% of weekly traffic, and that these were the best times to schedule administrative tasks, restocking, and maintenance. A further recommendation was scheduling marathoners—sales associates who can ask open-ended questions and who can generate upsells—during these hours to increase ADS.

As you can see, these insights are far more valuable than simple traffic numbers.

Crunching the Numbers to Transform Interior Analytics

The second way we used Qlik Sense was for another client to create an interior analytics report that provides store layout suggestions and labor model improvement recommendations. Over a year, we tracked customer movements in 15 stores that were open 24/7. Monitoring all this activity generated some 5 million records every single day, and therefore required vast processing power to yield detailed insights.

Using Qlik, we were able to filter information collected by our Wi-Fi sensors, and then separate it into customer and sales associate data. We then subdivided each store into 12x12-foot zones to create heat maps which show where customer-associate interactions occurred. Next, we calculated dwell times and analyzed shopper behavior by determining sequences of similar readings and by comparing current values with the previous one. Finally, we aggregated this data to create dwell time figures for individual departments in every store.

To perform these calculations with maximum efficiency, we split the data set into smaller batches that are processed one by one in a loop. We also perform calculations using an incremental load, so we only have to re-process the current month every time we load new calculations.

By avoiding repeat calculations, we are able to process huge data sets on a modestly powerful server in a few hours. The resulting Qlik app uses only 80mb to generate a 20-page report that aggregates 500 million records into heat maps, customer behavior records, and other actionable data. This report is far more sophisticated and has far more impact than anything we could have done with Excel or using sensor data alone.

The Best Is Yet to Come

Deploying Qlik Sense has allowed ShopperTrak to bring new services to our clients, and this is good for our bottom line. More importantly, Qlik Sense’s scalability, easy integrations, and powerful associative engine allow us to build solutions quickly and efficiently. We can go from the research and prototype phase to a client-facing solution faster than ever before, and we have new clients signing up for advanced analytics solutions every day.

In a few months, we'll be prototyping a video-tracking solution to better collect customer traffic data. This product is still in the developmental stage because we are looking for ways to ensure that customers' privacy is protected. We are also looking to boost our predictive analytics offerings and to create a product that tracks the impact external events like film festivals, rock concerts, and athletic competitions have on retail sales.

Thanks to Qlik Sense, ShopperTrak is helping to transform the retail landscape. We are no longer looking at a retail apocalypse, but instead are firmly focused on creating a retail boom.