Ergodic Punches Above its Weight Class with Coiled

A Small Hedge Fund Taking on Wall Street Giants

Introduction#

Ergodic Investment Group isn't just another hedge fund. They're a startup combining cutting-edge quant trading strategies with fintech innovation, all powered by a team of just three people. Despite their small size, they're competing in a space dominated by giants with substantial infrastructure and dedicated IT departments.

We started out with a software platform that would analyze market data for the impact of passive investing. Going into the venture capital freeze of 2022, we realized we needed to find alternative revenue streams... so we launched a hedge fund arm.

Jim Kennington

Chief Investment Officer, Ergodic

Their computational challenges are particularly demanding. They're not just analyzing historical market data — they're running agent-based simulations, processing options data alongside equities, and exploring vast parameter spaces to optimize trading strategies. Each simulation involves complex event-driven processes that model market behavior at a transactional level.

Before Coiled, the team invested in powerful local hardware (fully loaded workstations) to handle these workloads, but they were hitting hard limitations that constrained their business potential.

The Computational Challenge#

Ergodic's computational needs were extreme: complex event-driven simulations, options data processing, and parameter space exploration that their local workstations simply couldn't handle efficiently.

Inside each simulation, we're running event-driven models when market data hits, updating our predictions, and applying our decision-making logic. Each run models about 10 years of history for one strategy, which takes roughly 10 minutes even after we've optimized our code.

This computational bottleneck had direct business consequences. Most critically, their data processing pipeline took over eight hours to complete:

We were bound by our slowest data vendor, who usually delivers around lunchtime. We could never make trading decisions by market close, forcing us to trade the next day instead.

This T+1 (next-day) didn't meet the standards of the Ergodic team, especially in fast-moving markets. Additionally, when developing trading strategies, the team had to make difficult compromises:

We were throwing darts with one machine by very sparsely sampling because we didn't have the resources to create a full map of the choices. As with any optimization problem, your optimizer is only as good as the amount of space it can sample.

A Python+Rust Technical Stack#

Ergodic's stack includes:

Polars for data processing: "Using polars, we can achieve parallelism that pandas can't because polars can get around Python's global interpreter lock."
Dask for distributed computing: "We've been using Dask from the get-go. We had our own serial evaluators, but when we wanted parallelism, I made sure our trees could compile into Dask trees."
Custom Rust plugins for accounting operations: "You'd think it'd be easy to model how much you owe the exchange. It's really not. They have several rules, and a pure Python implementation would be too slow."
Custom expression frameworks: "We have our own in-house framework with fully bound Python functions, which we compile into Dask-friendly delayed objects."

None of us are experts in system administration or Kubernetes. We're software developers, not IT people.

Finding Coiled: Cloud Computing Without the Complexity#

Rather than building their own cloud infrastructure, Ergodic chose Coiled for its Python-native approach.

I've built distributed computing frameworks myself before Dask and Coiled existed. Many quant finance companies build their own versions of things they don't need to, then spend valuable research time maintaining legacy code. Coiled is a great fit for us because it's Python-native and doesn't require us to learn a new language or framework.

What made Coiled compelling was how it extended their existing workflow to the cloud:

The Coiled package sync tool saves us so much time. We don't wait 30 minutes for Docker images to build between runs. We change code locally and test it immediately on workers.

This capability eliminated the friction of constantly rebuilding Docker images during active development.

The more time you spend tool building, the less time you spend tool using. Coiled allows us to not worry about the middleware.

The Transformation: From 8 Hours to 30 Minutes#

With Coiled, Ergodic implemented a multi-cluster approach that transformed their capabilities:

Parameter exploration clusters: 1000+ workers for parallel simulations

Within one research cycle, we compare a million combinations instead of a few hundred.
Feature engineering clusters: Multiple smaller clusters running concurrently

These are very deeply nested trees with hundreds of thousands of tasks. With Coiled, we're not limited to one scheduler. We can spin up 30-40 clusters simultaneously.

The results were dramatic:

We kicked off a signal with around 200,000 tasks. Locally that would take most of an afternoon. With Coiled's distributed setup, it took maybe 20 minutes.

When they needed custom solutions for accessing Azure NetApp Files, the Coiled team delivered:

Our data access patterns required mounting Azure NetApp Files, which wasn't initially possible with the standard cluster API. Coiled's team responded within a day and delivered everything we needed in less than a week. This kind of responsiveness from a vendor is exceptional.

Business Impact: Same-Day Trading and Better Strategies#

The technical transformation led to dramatic business improvements:

Pre-Coiled, our data processing took eight-plus hours. With Coiled, we get it done in 30 minutes and have trading decisions well before market close.

This acceleration enabled a shift from T+1 (next-day) trading to same-day (T) trading – a significant competitive advantage in markets where timing is critical.

Beyond faster processing, the ability to run vastly more simulations transformed their strategy development:

It speeds up our research cycle dramatically. Instead of getting hints about the right direction, we get clear answers. We've discovered trading options and styles we weren't aware of before. We've picked a different set of parameters that works fundamentally better.

This comprehensive exploration made a real difference when engaging with investors:

Coiled's responsiveness allowed us to address investor questions quickly, which was essential for our upcoming funding round. One investor asked us to try new permutations with datasets that exceeded what we could do locally. We were finalizing the deal, so we needed answers fast.

Punching Above Their Weight Class#

Coiled has enabled Ergodic to compete with established hedge fund giants despite their small team:

We're able to perform at a level that defies our constraints. We're doing simulations on par with what the biggest players in our space are doing... with three people and a small cloud budget.

The ephemeral nature of Coiled clusters helps control costs without sacrificing performance:

Moving fast was our priority. Coiled lets us scale without recurring costs because the clusters spin up on demand and disappear when we're done.

Jim draws a striking comparison:

We analyzed 50 percent of the data volume that ChatGPT processed over the past week. We spent $5,000 with Coiled while they spent millions on data centers.

The Future with Computational Abundance#

Jim and team are pushing on, full steam ahead.

We are a different business post-Coiled. We operate in ways that weren't possible before, asking questions we couldn't ask before. We're no longer bound by technological resources.

Jim Kennington

Chief Investment Officer, Ergodic

This shift from computational scarcity to abundance has opened new possibilities for the team, allowing them to pursue more ambitious approaches:

With the rise of agentic AI trained on massive clusters, we're able to create similar informational networks with Coiled for a fraction of the cost of using OpenAI or other technologies. Our agents are more rules-based than AI, but the computational patterns are similar.

For a small team competing in sophisticated quantitative finance, this computational edge is transformative. With Coiled, Ergodic has proven that a small, focused team with the right tools can punch far above their weight class.