Behind the Scenes at Cisco Live: One of the Biggest Technology Events in Europe

Conferences are a big part of professional networking and development. The scene is pretty familiar: a few hundred people attending keynotes and breakout sessions at a hotel or a conference center—all the while making do with incredibly mediocre Wi-Fi. It’s frustrating when you have to travel to an event and can’t rely on connectivity, but most people assume that’s just the way it is at conferences.

Not at Cisco Live. Our premier annual event is the biggest on Cisco’s calendar in EMEAR. At our January 2020 event in Barcelona, we saw more than 18,000 visitors at more than 1,000 sessions over five days. At Cisco Live, we reveal our latest technologies across our entire portfolio including security, networking, data center and collaboration, so we can’t just use just any existing event-space network. Instead, we build it from the ground up.
 

As the lead engineer of the Cisco Live Barcelona Networking Operations Center (NOC), I get to handpick talent from Cisco worldwide to collaborate on these events. There are twelve people on my core team, with an extended team of 60, all to run this one annual event. As we look ahead to the next event, it’s always worthwhile to look back and see how we prepared and managed an event of this scale—and here’s how you can make sure the network at your event surpasses everyone’s wildest expectations.

Dreaming Big with an Event Space

Thinking about these large-scale events begins with evaluating the venue itself. About a year before the event is set to take place, we start talking with the venue about existing infrastructure and hardware. When it comes to Wi-Fi, sometimes the venue already has a Cisco network we can leverage. They might even have already installed Wi-Fi access points on the roof, which is fantastic because my team can quickly install things at eye-level, but installation becomes a bit more difficult the higher you go. 

More often an event space will say, “Yes, of course we have Wi-Fi,” but it will date back to the days when connectivity meant getting a single bar on your (non-smart) phone. In order for our event to run smoothly, we need high-density Wi-Fi with additional location services, and video streaming on top of that. A single access point on the side of a hallway will not be sufficient to provide an optimal experience for several thousand people.

My team works with the Cisco sales teams to decide which upgrades are necessary, but we also embed venue event engineers into our team to enhance collaboration. Even though we may end up upscaling the network for our particular service and data load needs, there’s an opportunity to permanently uplift the venue’s infrastructure. This puts it in a better position to host future events. In turn, those venue engineers often go on to become Cisco advocates.

But Wi-Fi isn’t the only concern. We bring a lot of gear with us to the venue, which means we also have to look at other logistics to make sure all of that equipment can be accommodated. One of the most important—and overlooked—considerations is the HVAC system. It’s important to ask whether or not the existing air conditioning can provide adequate cooling for the amount of heat that our equipment generates. A lot of times it can’t, so we have to find temporary solutions with the on-site HVAC personnel. 

We also have to determine how we get connectivity to all the event rooms. If any rooms or stands will be built on a concrete floor, for example, sometimes we need to be creative when thinking about using trusses to bring connectivity to that area. Another concern might be whether construction is taking place nearby. While the activity might not impact the event itself, their equipment could accidentally cut the fiber line, which can’t be replaced in a few minutes. Logistically, we have to plan for all of the what-ifs, and that includes separating the cabling for internet and power. 

Checking Every Contingency

Security is obviously a big consideration. We need to keep the environment secure for participants, partners, and ourselves, but each group has different needs. Our partners who have stands at the event want the least amount of blocking between services because they want to run demos. If we used a firewall, running the demos would create so many holes that the firewall would become a hindrance more than anything else.

Instead, we keep everything separated and isolated through access lists. We have on-site registration, so using access lists means that our participants’ information is protected. They also can’t access services inside our network, but if you’re a partner running a demo from a stand, we are not blocking any ports. For security, we use Cisco Stealthwatch and Cisco Umbrella, and my security operations team also performs penetration testing and continuous testing on data flows throughout the event to catch any unusual activity. 

Ultimately, everything has to work, no matter what gets thrown our way. The same way we consider contingencies with our cabling, we build a two-legged system for data just in case something goes haywire. We have two data centers on premises. Each has their own connections, they both use NetApp FlexPod, and all the services are connected to both centers. That way if one goes down, we can keep the event running without interruption. 

Consider that 32,000 devices connected to the network at our event in Barcelona, with a peak of 17,500 devices at one time. With a 50% increase in data traffic over the previous year, there was no room for failure. 

In the days leading up to the event, we do all our failover testing. Once we’re sure the network’s stable, then we begin to build the solutions for the venue upon each other like layers of a cake. We start from the basement, then once we get to the ground floor, we lay out miles and miles of cable, building the network at the same time as the stands, booths, and stages are being built. We test, test, test throughout—if a forklift runs over a cable, it doesn’t matter if that cable worked 10 minutes ago. You have to test it again.

The events business is all about speed, and sometimes we happen upon brilliant solutions in our search to make things faster and smoother. We install our network at the same time as the rigging team is installing lights and speakers, so we installed clamps on our access points that are similar to what the lighting team uses. That way the access point can clamp right onto a truss rather than being secured with a zip tie (which isn’t ideal from a health and safety perspective). 

This universal bracket is now being used across the event industry. We discovered a solution to our problem, but it’s an industry-wide issue so we’ve been happy to share our success with others.

The Brass Tacks of Barcelona

Cisco Live 2020, Barcelona set a precedent for user experience. People expect the latest and greatest from us, and we do as much as we can to surpass any limitations posed by the venue’s initial infrastructure. 

For this event, we used 750 access points, 600 of which were Wi-Fi 6, 500 switches with the core of the network all Catalyst 9k Series Switches, and 40 kilometers of network cable. We distributed 50 sensors across the venue to get an insight via Cisco DNA Assurance into the Wi-Fi infrastructure performance from a client perspective rather than an access point perspective. We then repositioned some antennas to get better coverage throughout the space.

The venue had already worked with Cisco to upgrade to Wi-Fi 6 in some of its halls, so our task was to infill areas that didn’t have the upgrade or areas where we knew the density would be high, such as the keynote space. To accommodate the 7,000 people attending the keynote, we installed an additional 60 Cisco Catalyst 9120axe access points and 120 antennas pointed towards the floor. Two other areas where we knew we wanted Wi-Fi 6 were at registration, which is where we did the network onboarding, and in the breakout rooms. 

For compute we also used Cisco HyperFlex, which came in handy for the location services in the Cisco Events mobile app. One of the most valuable aspects to live events is the in-person interaction, but it’s nearly impossible to try and find anyone in a crowded hall. We developed an opt-in solution in the mobile app called Find My Friends. We were lucky to have HyperFlex on hand because the Find My Friends feature requires a lot of compute, which HyperFlex lets us add very easily. It turned out to be a very popular feature and is now standard for Cisco Live shows across the world. 

With Automation, Calm Descends

The funny thing is, we put in a ton of preparation to be ready for the event, but the moment the event actually starts, calm descends. We go into monitoring mode, with about 20 people in the core part of the NOC sitting in front of large screens, monitoring each part of the network. We filter all the information from the network into Webex groups, which are followed by the team members who aren’t in the NOC.

When I say “monitoring mode,” I mean it. I have to force my engineers to sit on their hands during the day, because it’s their natural inclination to want to tweak anything they see running below 100%. Unless something with the network is truly broken, however, we don’t change anything while the event is actively running. This is a dynamic environment, and even small tweaks work differently in a live setting than they do in a lab. Instead, we plan for how we will change things that evening so the network will perform even better the next day. 

Cisco products are designed to get a lot of information out of the gear. Anything we can automate or write a script for, we will. In my honest opinion, automation is the biggest advantage of our products. I see it in the information we obtain from the network, but I also see the benefits during staging. At Cisco Live, just three people configured 450 switches—all with different configurations—in a single day. Without automation, that would’ve been impossible.

A Great Team Creates Great Events

A networking lead must consider appropriate products and solutions, but the most important consideration is the team itself. I don’t assemble the same group of people every year; the team evolves as people themselves evolve. 

I have a lot of extremely smart people working with me, but there is no room for egos. If we need to install 50 more access points before we are done for the day, everyone roll up their sleeves and help with installation, regardless of their given role. Rule number one is we get it done, and we get it done together. 

Enthusiasm and team spirit are a top priority, as is creating a diverse team with various perspectives. A third of my engineers are women, and I actively look for ways to give opportunities to young people. Give people a chance, and you’ll benefit from their development over time. 

We’re serious about our projects, but if we’re serious all the time, then it’s just a job. We have fun too. A feature of our access points is that when set to discovery mode, they cycle through their blue, red, and green LED lights. One of my engineers synced all our access points on the roof, so they could cycle through “party mode” at the press of a button during the evening celebration event. We thought it was pretty funny but were completely surprised when participants started taking pictures of the lights! Is “party mode” useful? Not really. Did it add a little something extra? Absolutely.

On June 2 and 3 you can join the Cisco Live 2020 digital event, for free—from your home.

For more details on the Cisco Live 2020, Barcelona deployment, I invite you to watch this on-demand session featuring the Cisco Live NOC team.