Going Beyond A/B Testing at Tableau: Driving Personalized Experiences and Conversion Performance

As the world’s leading analytics platform, Tableau’s mission is to help people see and understand data. When I joined Tableau as Vice President of Marketing in 2019, marketing’s primary funnel-related focus was on driving lead volumes. We have shifted that approach to maximizing pipeline and deal size, while also optimizing the self-service experience to support smaller deals.

A meaningful portion of our transactions at Tableau are small, whether because a large account is adding a few licenses or a smaller company is becoming a new customer. These sales are important to serve our customers and further our mission. While they do not meaningfully contribute to our revenue, they can require a relatively heavy lift from our sales team. To both better serve our customers and enable our sales team to focus on larger deals and customers, we have invested in building out our eCommerce capabilities.

Platform Limitations Reduced eCommerce

Placing an emphasis on self-service came with its own challenges, though. To start, the experience for our eCommerce site was reflective of how we thought about our products, not how potential clients intuitively wanted to buy them. Additionally, only a subset of our products was available online, and existing customers could not add to their deployments. Given these limitations, we did not actively promote eCommerce on our website in case it turned into a suboptimal experience for users. 

To address this, we had to focus on three areas: 

1. Enhancing the baseline functionality 
2. Improving the user experience
3. Increasing traffic from our overall website to our eCommerce engine

For #2 and #3, we needed to ramp up our capabilities in testing and personalization.

For testing, a significant challenge came down to people—my team did not have dedicated testing resources. Part of the reason was that it was harder to justify a new full-time headcount for testing than to obtain funding for technology and third-party assistance. On top of that, making changes to our eCommerce experience meant going through our central IT team, who were responsible for the platform. The team is incredible, but they did not have the bandwidth to make constant improvements to the eCommerce user experience. This reality meant we ran very few tests. 

We needed extra help, and I knew just where to find it.

Turning to Machine Learning Website Optimization

I had first heard of Intellimize when I met Guy Yalif, their co-founder and CEO, at a conference several years ago. I wanted to adopt Intellimize at the company where I was working at the time, but the timing was not right.

When I arrived at Tableau and saw that we had some aggressive targets to hit, I thought of Intellimize right away. It seemed like a great opportunity because of the strengths of their platform as well as their professional services. They are very flexible, which made for quick and easy implementation. With Intellimize on board, we were ready to try new things and ramp up the performance of our eCommerce platform.

Plenty of platforms handle A/B testing, and do it well. But where Intellimize differentiates itself is with its machine learning platform that automatically shows every visitor their own personalized web experience. With traditional A/B testing, half of users get Variant A, half get Variant B, and then you declare a “winner” based on the results. 

What Intellimize brings to the table is the understanding that not every visitor to our site is the same. Variant A might be better for 70% of our site visitors, but Variant B is better for the remaining 30%. So if we choose only Variant A, we are losing out on that 30%. Intellimize can recognize and capitalize on this opportunity to show each web visitor a personalized experience that is best for them. Further, the platform can run many simultaneous tests, and will perpetually optimize the site experience as our visitors change over time.

That unique melding of testing and personalization means we are able to constantly run tests to optimize our outcome, while also enabling distinct, relevant experiences. It is a powerful combination. 

Insights from Machine Learning

We are constantly trying to figure out what about Tableau resonates the most with our customers. We frequently have a number of opinions in the company about what should appear on the website and testing provides a way to let data and customers speak.

This situation comes up most in relation to the top banner of our homepage, which makes sense since that banner is the first impression for a decent portion of our site traffic. These differences of opinion have inspired near-religious debates within the company: Should we exclusively talk about the products? Should we talk about how Tableau enables companies to develop a data culture? Should we use it to highlight events or key announcements? With proactive personalization, that debate becomes moot: Message A is better for some users, Message B for others. Message C is better for another group.

This helped us during the initial days of the pandemic, as we wanted to know what concerned our customers the most. Were they in reactive mode, trying to make sense of how the pandemic affected their industry? Or were they already looking forward to rebuilding even stronger? We assumed that people would be looking ahead to the recovery process, but our testing told us that this was not the case for most people for quite some time. As the pandemic progressed, we also saw regional differences: Asia Pacific was much faster to rebound than the U.S. But then, even as large parts of the U.S. locked down, companies began to focus on their reopening strategy. When we acted on these insights with a change to our homepage, we saw 5x the engagement. 

Through other testing, we also discovered that we needed to simplify our message in a way that facilitated self-service. When visitors received a message tailored to their user profile, potential customers felt confident in using self-service because they did not have to ask questions of a salesperson.

Overlaying Intellimize enabled us to have a lot more volume in our testing muscle and make the changes we needed to drive significant growth.

Excellent Support Leads to Greater Velocity

Intellimize is incredibly intuitive, both in its user interface, as well as how easily we can share results with other people. Beyond the platform itself, Intellimize sets itself apart by the strength of its professional services, excellent customer support, and the wealth of knowledge and best practices they bring to the table. Because they work with so many other companies, their thought leadership on expanding eCommerce was a great supplement to what we already had planned. Especially since we did not have a point person responsible for testing, Intellimize’s expertise helped us make progress quickly. 

There are other ways Intellimize helped us ramp our experimentation velocity. Before Intellimize, we used to have multiple meetings just to decide what to test. Now, we have been able to significantly streamline that process. We know we are going to test every reasonable hypothesis to find what works best, which ultimately leads to faster decision-making. Previously, I needed to be much more involved in the process, and getting approvals would take time. Now I receive regular updates while our team works hand in hand with Intellimize, moving through the process much faster, so we all learn more quickly.  

One challenge I have encountered at several companies is the approach toward costs for full-time headcount vs contractors and agencies. Finance departments are normally more comfortable with the latter, but that does not work well for ongoing programs such as testing. Once the contract is done, the team on that project moves on to something else, and the program may falter. 

Also, most testing and personalization technology companies do not provide professional services. Here again, Intellimize met our needs through their fully integrated team that both advises us on tests as well as helps build them out. They have been excellent partners and truly an extension of our team. On top of that, based on the significant positive financial impacts of our testing, we have been able to hire internal testing resources, which has further accelerated our combined impact. 

Intellimize Significantly Improved Our Performance 

When we first started with Intellimize in 2019, our eCommerce experience was subpar, to the point we made it hard to find as we worried about conversion. We decided we had to significantly increase traffic going to eCommerce, which initially dropped our conversion rate. Through accelerated testing with Intellimize, we have gotten better at personalizing and bringing the right kinds of people into our eCommerce funnel, which has both increased traffic and enabled us to double our conversion rate as the eCommerce experience improved. 

We have built upon this success, and so far this year, we have more than doubled eCommerce revenue YoY. As far as the value of Intellimize, we have seen over a 9:1 return on our spend with them.

Ready to Take on New Goals Like We Never Could Before

Today, we know that the vast majority of a potential customer’s research is done before they ever officially reach out to learn about our service. We also know that for many, their preferred method of finding out about our offering is through self-service rather than a salesperson. My macro goal is to ensure that anyone, from a small customer to an enterprise one, can get everything they need through self-service. Of course, salespeople will still be available to support anyone who wants it, but continually improving our self-service experience will benefit everyone. 

Intellimize has significantly improved Tableau’s performance and brought analytical rigor to how we approach our website and eCommerce. It has also significantly improved the speed of change within our organization. Now, we understand our customers better thanks to insights that we did not have in the past. Looking back, I wish that I had the opportunity to use Intellimize sooner. It truly has made such a huge difference to us at Tableau!