How NextRoll cut costs by 70% switching from EMR to Coiled

On any given day we are participating in around 50 billion auctions on the web. Every time a person visits a website, we have around 100 milliseconds to respond with which ad to show and how much to bid.

Imagine you have thousands of clients, each with a product catalog of around a million products. That's a big cardinality. And what is a product? It's not just an image URL, but it has metadata—price, categories, different tags the customers are interested in. And then we have historical logs of user interaction data telling us what customers have viewed and when.

A typical workflow would consist of writing code on a laptop, refining it, and then tossing it over the wall to productionalize it against a giant dataset. We were working in pure Python locally, but the production environment used PySpark with its own overhead and quirks.

The thing that really slowed us down was debugging during pager duty alerts in the middle of the night. The person on call might have done the prototyping in Python but had little experience with the Spark job that was eventually productionized. This naturally led to frustration.

Data privacy was non-negotiable for us—our customer data couldn't leave our AWS VPC. We discovered that Coiled's approach was to run the workload for us while keeping everything in our environment. We fell in love with that workflow.

When we told our DevOps engineers what we were about to do, they just asked, 'Is there something we can study?' We gave them our Terraform code and they said, 'This looks great. Nothing is leaking. Everything is being done locally in our AWS data center. Go for it.'

We were in love with Coiled's way of spinning off infrastructure. It was just Python decorator magic. We'd say, 'Please give us one VM,' and immediately see it appear in our AWS console. When we said, 'Can we 5x this?' again in real-time, five machines would come up.

Coiled would introspect what's installed in our virtual environment and make it available in the cloud. We were amazed by how little effort it took to make things work at scale.

You are writing native Python code all the way—be it on your Jupyter notebook, in your Python script, or when you put it all over on AWS. Nothing really changes, literally nothing. The only thing that changes is that, by the way, please use more than one machine or use really, really big machines. It forms a really nice triad—an orchestration framework, a framework to scale jobs, and a framework to process large data

There's another feature we discovered quickly—Coiled allows us to implement all the best practices we were used to. Our SREs would be livid if we weren't propagating tags. With Coiled, it was just adding another keyword argument: 'Please use these tags and spot instances where it makes sense.'

Once we switched from AWS EMR to Coiled, we've seen our estimates drop by as much as 70%.

When we were deploying things to production, we had a lot more confidence, because all of a sudden, what was on our local laptop didn't look starkly different from what was actually being deployed.

Reasoning about failures became much easier. Instead of esoteric Spark JVM dumps that you could barely understand, we were working with familiar Python exceptions. Anyone moderately familiar with Python could identify the problem immediately.

When deploying to production, we had much more confidence because what was on our laptop didn't look starkly different from what was being deployed. We could stress test by simply cranking up the data size and number of VMs.

The biggest win was that developer happiness increased dramatically. If my developers are happy, they're prototyping more, experimenting more, and playing with new ideas. When you're doing 100 experiments, inevitably 10 or 15 of them become products.

In the past, we'd be like, 'I have to write this script and then turn it into a Spark job and set up infrastructure? Never mind.' That's absolutely a killer for innovation.

The Coiled UI helps us keep tabs on our billing. At any moment, I can see how my teammates are using Coiled, and there's a column that literally says 'Here's the estimate for how much your workload costs.' In the past, we'd tear our hair out trying to estimate costs.

We pipe our CloudWatch logs to Coiled, which helps us debug new workloads. If we run into an obscure issue, we can quickly call someone from Coiled, and it's easy for them to look up the logs themselves and identify the problem.

We have a service that crawls all of our advertisers' websites, so it's just ginormous. The previous system was just struggling—it could not make head or tail of how to process this. But with Futures and Coiled, it was just this much code, and I was like, 'Wow!'

We're experimenting with using Coiled to run jobs on demand, letting it terminate instances when they're done. That would allow us to scale out exactly as much as needed, when needed, without idle resources.

The most important thing is that developer experience should be delightful. If we are left dealing with infrastructure issues, we can't be prototyping new things or coming up with new ideas. Coiled and Dask allowed us to do just that. Experimentation is money.