Blog¶

Production ETL

April 8th, 2024

Run large Python jobs on a schedule on the cloud.

$ coiled run \
   --vm-type p4d.24xlarge \
   --region us-west-2 \
   --container huggingface/transformers-pytorch-gpu \
   python fine_tune.py

ML + GPUs

May 1, 2024

Easy access to GPUs.

../_images/coiled-conceptual-diagram.png

Why Coiled?

January 10, 2024

Coiled makes working with TB-scale cloud data really easy.

@coiled.function(
    region="us-east-1",  # Same region as data
    memory="512 GiB",    # Enough memory for your data
)
def process(filename):
    # Everything inside the function stays the same
    ...

Parallel Serverless Functions

September 7, 2023

Easily adapt your Python code to run in parallel on the cloud.

All posts¶

May 14, 2024 DataFrames at Scale Comparison: TPC-H
May 14, 2024 Dask DataFrame is Fast Now
Apr 19, 2024 Dask vs. Spark
Apr 08, 2024 Easy Scalable Production ETL
Feb 05, 2024 One Trillion Row Challenge
Jan 31, 2024 Real-world Grocery Demand Forecasting
Jan 23, 2024 Schedule Python Jobs with Prefect and Coiled
Jan 16, 2024 1BRC in Python with Dask
Dec 21, 2023 Xarray at Large Scale: A Beginner’s Guide
Nov 17, 2023 Process Hundreds of GB of Data in the Cloud with Polars
Nov 01, 2023 Processing Terabyte-Scale NASA Cloud Datasets with Coiled
Oct 10, 2023 How to Run Jupyter Notebooks on a GPU on the Cloud
Oct 06, 2023 Ten Cents Per Terabyte
Oct 05, 2023 TPC-H Benchmarks for Query Optimization with Dask Expressions
Sep 19, 2023 Coiled observability wins: Chunksize
Sep 07, 2023 Parallel Serverless Functions at Scale
Sep 05, 2023 Processing a 250 TB dataset with Coiled, Dask, and Xarray
Sep 01, 2023 Reduce training time for CPU intensive models with scikit-learn and Coiled Functions
Aug 23, 2023 Fine Performance Metrics and Spans
Aug 10, 2023 Data-proximate Computing with Coiled Functions
Aug 09, 2023 Dask, Dagster, and Coiled for Production Analysis at OnlineApp
Aug 07, 2023 Process Hundreds of GB of Data with DuckDB in the Cloud
Aug 04, 2023 High Level Query Optimization in Dask
Aug 01, 2023 Easy Heavyweight Serverless Functions
Jul 24, 2023 How to Train a Neural Network on a GPU in the Cloud with coiled functions
Jun 23, 2023 Dask performance benchmarking put to the test: Fixing a pandas bottleneck
Jun 14, 2023 Coiled notebooks
Jun 05, 2023 Utilizing PyArrow to improve pandas and Dask workflows
May 18, 2023 Distributed printing
May 16, 2023 Observability for Distributed Computing with Dask
May 15, 2023 GIL monitoring in Dask
May 05, 2023 Performance testing at Coiled
May 05, 2023 How well does Dask run on Graviton?
Apr 18, 2023 Upstream testing in Dask
Apr 04, 2023 Burstable vs non-burstable AWS instance types for data engineering workloads
Mar 15, 2023 Shuffling large data at constant memory in Dask
Feb 23, 2023 Just in time Python environments
Jan 17, 2023 How many PEPs does it take to install a package?
Jan 06, 2023 Scaling Hyperparameter Optimization With XGBoost, Optuna, and Dask
Jan 06, 2023 Handling Unexpected AWS IAM Changes
Jan 06, 2023 AWS Cost Explorer Tips and Tricks
Dec 19, 2022 Automated Data Pipelines On Dask With Coiled & Prefect

Data Privacy

DataFrames at Scale Comparison: TPC-H