Set It, Build It, Deliver It: How Condé Nast Built an Enterprise BI Platform

Condé Nast is a global media company and home to some of the world’s most iconic brands. While many people are familiar with Vogue, Vanity Fair, The New Yorker, and Wired, they probably don’t give much thought to the powerful in-house data team that delivers business intelligence throughout the company.

Comprising more than 15 Qlik developers, UI designers, and administrators based in New York and Chennai, Condé Nast’s BI team supports almost every business function of the company, from sales and finance to audience development and consumer marketing. 

As the Associate Director of Business Intelligence at Condé Nast, I’m the lead on our enterprise BI platform. I arrived at the company as a BI manager in 2018, bringing more than 10 years of experience in the data field. I used this knowledge to help develop a philosophy and strategy to address our data challenges.

Making a Global Organization More Data-Driven

Aligning with many different teams across the world is a big undertaking for any global organization, but we faced other challenges surrounding the data platform itself. 

Data accessibility—ensuring data can be accessed at the right time, in the right shape and form—is critical for any data-driven organization. Nevertheless, we had data accessibility limitations resulting from our platform and the way our data was siloed. Data wasn’t available to users in a timely manner, and even when it was available, it was not presented in a way our users found useful.

Our second challenge was poor data quality. As a company grows and amasses more data, the quality often gets worse. We had issues with data completeness and accuracy, so users did not trust our BI solutions. 

Our third problem was that the platform offered a poor user experience. Due to a high volume of data and bad implementations, the platform was sluggish. The technology we used at that time wasn’t interactive, so it wasn’t engaging. 

All of these challenges compounded, resulting in a lack of organizational adoption.  

We overcame these challenges by developing a simple yet powerful threefold philosophy: 

• Choosing the right technology and building the correct platform with enough firepower to handle the enterprise BI need 
• Connecting the dots between data and user requirements with correct design patterns and thinking outside the box to cater to versatile use cases 
• Delivering data insights in the shape and form that end users can understand, via their preferred medium 

Set It Right: Selecting Technology and Designing Architecture

For any BI platform, the tools and technologies used define the realm of possibilities. The first step, Set It Right, is about choosing the right technology and architecture. We chose Qlik Sense for its associative in-memory analytics engines as well as its functionality for the non-technical user. Qlik Sense allows a layperson to build a simple dashboard and developers to build complicated solutions using its open APIs. In a nutshell, it gave us endless possibilities. After that, adding Qlik NPrinting for reporting was an easy choice. Now, we’re also entering into Qlik Cloud Services. 

After choosing the technology, it’s imperative that we get the architecture right to support the needs of the BI platform. At Condé Nast, we didn’t use the traditional separate node implementation for different environments, instead opting for a multi-node architecture with logical separation between environments like development, quality assurance (QA), and production. We separated our developer nodes from our consumer nodes, so the user experience on the consumer side would not be impacted by any ongoing development work or data loads. 

Even with the right technology and architecture, no tool can provide every functionality needed to build the perfect BI platform. This is why it’s important to keep an eye on flexibility to mold a platform to your needs. That’s where Qlik Sense excelled for us, enabling us to augment our BI platform to fill gaps in a few key areas: Parallel processing, automatic trigger for data refresh, and pre-caching. 

By default, Qlik Sense runs the data load on one node, which might be limiting in certain big data use cases. However, Qlik’s open APIs provide a way out, allowing us to run our data refresh in parallel and enable multi-node processing. We used Qlik Sense to generate multiple apps for a single data load script and routed them to multiple nodes to run in parallel. This not only helped us reduce the ingestion time on big data, it also helped us to balance these bigger loads on our multiple nodes. 

The second way we augmented our BI platform using Qlik Sense’s open APIs was to ensure dashboards are refreshed as soon as the data is available. This not only removes data uncertainty, it makes operations easy, since you no longer have to monitor for data delays or schedule jobs at fixed times. To make this possible, we built a custom solution to automatically trigger a Qlik data refresh as soon as the underlying data is ready. We embedded the trigger package within the data pipeline scheduler and leveraged the Qlik QRS API to integrate with the Qlik data loads. 

Lastly, we pre-cached bigger apps to boost performance. We wrote custom Selenium scripts to keep the cache warm which ensured the first user accessing the app gets same performance as any other user following him.

Build It Right: Connecting Users with Data and Eliminating Silos

This brings us to the second pillar of our strategy: Build It Right. Our four-step recipe for BI solutions connects the dots between data and user needs. Then we use automations and accelerators to build it fast. 

The first step is to build for consistency. We built a centralized data layer for every business function, which ensures consistency and standardization of metrics across all dashboards and also helped us to overcome those data silos. 

The second ingredient is building user trust by getting the data quality right. Once you get the data in, it’s imperative the data is correct and complete. End users will only trust the BI solutions if they receive high-quality data every time, so we built a validation mechanism into our implementation. We then took it to the next level by defining permissions to allow super users to validate and reload our data from the user interface (UI). 

The third ingredient is re-usability and accelerators. When you cater to an entire enterprise, the time to market matters a lot. To accelerate what we develop, we built an internal library of reusable subroutines and design patterns, so our developers don’t have to reinvent the wheel every time. This not only helped with time to market, but also with standardizing our build and giving us central control on our implementations. 

The final ingredient to build it right is outside-the-box thinking. Building It Right hinges on thinking about how we utilize resources for a few complicated scenarios. One example would be building a dashboard to represent insights for Condé Nast’s user segmentation. Being a large media company, we had more than 1,800 pre-built base user segments. Building over 1,800+ dashboards through conventional means was unfeasible; it would have been impossible to maintain, and would take a huge amount of my team’s resources. That led us to wonder whether we could do this another way. 

Again, Qlik’s open APIs came to our rescue. Using the session app concept to generate app in-memory based on the user request, we came up with an ingenious solution to generate dashboards on the fly. When the user selects a segment from the selection screen, it kick-starts a process to generate a segment performance app. The generated session app dashboard will be used by our React app to render the insights. The session app is powered by a template app, which reads data from our QVD layer which has been prepared offline.

Of course, we love automation, and we have built automation into even the simple things, like switching data connections based on the stream an app belongs to. If an app belongs to the published stream, it automatically points to production. We’ve also standardized the incremental load by writing a reusable subroutine. Doing these basic things in an automated manner not only helps us avoid human error, it also helps us standardize implementation across the platform. 

Deliver It Right: Making Data Accessible to Everyone 

This brings us to our last pillar: Deliver It Right. The number of dashboards grew very quickly, and the challenge became how to organize them and make them accessible. To overcome these challenges, we built a custom homepage that acts as a one-stop shop for users to access any and all BI solutions. We arranged the homepage menu based on business function, providing users an intuitive way to find what they’re looking for and explore other possibilities on the platform. In addition to business function, the homepage menu is also based on Qlik access level, showing users different options depending on permissions. 

Building dashboards to cater to the audience is imperative to achieve adoption. We categorized our users into three broad groups: analysts, super-users/contributors, and executives. 

The analyst-level dashboard has the highest granularity of data to support detailed exploration. The super-user level has less granularity of data, however we promote an environment where users can self-serve their ad hoc needs. Our executive-level apps present very clean top-line numbers, representing the pulse of the business. Executive apps often connect different functions of the business to present a consolidated picture to the executive. 

Comparing an analyst and executive app for the same business function, the analyst app is data heavy and highlights different trends, whereas the executive app is data light and displays high-level numbers. We do, however, always connect our executive apps to the detailed apps, in case the user wants to dive further down into the data. And the executive apps are powered from the analyst apps to ensure consistency of metrics at every level. We also ensure that all apps are mobile friendly, in addition to desktop.

Delivering an Intuitive User Experience

How we bring our data story to the audience makes a world of difference. We’ve pushed the boundaries of our visualization by using React to build custom UIs. We also get the most out of Qlik’s visualization tools to enhance data storytelling. 

At the end of the day, we believe in building an intuitive user experience to ensure that we can share insights legibly and easily with the end user. By putting our threefold philosophy in action, we overcame our challenges to build the Condé Nast enterprise BI platform, a standardized, easily managed platform that provides quality insights our users trust.

Want to learn more? Check out my QlikWorld Online 2021 breakout "Conde Nast's Enterprise BI Platform" here.