Process Satellite Images in Parallel

This example demonstrates how to process thousands of satellite images in parallel using cloud computing. By leveraging Coiled Batch, you can easily run non-Python tasks at scale without complex cloud infrastructure setup.

The example processes 3,111 satellite images in just 5 minutes, costing around $0.70 total. You can run it right now with minimal setup. All you need is:

Save this script as reproject.sh and you're ready to go. If you're new to Coiled, this will run for free on our account.

After saving the script, just run one command to execute it on the cloud:

Now let's explore how this works and why it's so efficient.

Geospatial data processing often involves transforming large volumes of satellite imagery. In this example, we need to reproject 3,111 Sentinel-2 satellite images hosted on AWS S3. These are high-resolution images of Earth's surface that need to be converted from their native coordinate system to a standard projection (EPSG:4326).

This task presents several challenges:

1. Volume: Processing 3,111 files serially would take days
2. Location: The data is hosted in AWS, so downloading it locally would be slow and expensive
3. Software Dependencies: We need GDAL, a specialized C++ library for geospatial processing
4. Resource Intensity: Each file requires significant memory and CPU for processing

Running this job serially on a laptop would take over 2 days, and setting up cloud infrastructure manually through AWS Batch, GCP Cloud Run, or Azure Batch is complex and time-consuming.

We'll use Coiled Batch to run our processing on cloud resources that are co-located with the data. The entire hardware configuration is specified using special #COILED comments in our script:

These directives tell Coiled to:

1. Create 100 VMs in the US-West-2 region (same region as the data)
2. Provision each VM with 8 GB of memory
3. Use a ready-made Docker container with GDAL pre-installed
4. Securely forward AWS credentials to access private buckets
5. Run a total of 3,111 tasks distributed across these VMs

This configuration provides an optimal balance between performance and cost - more VMs would increase parallelism but might not be cost-effective for this particular workload.

The key to efficient processing is the COILED_BATCH_TASK_ID environment variable. Coiled automatically sets this variable to a unique value (0, 1, 2, ..., 3110) for each task. We use this ID to determine which file each task should process:

This line tells each task to:

1. List all the .tif files in the bucket
2. Select the file at the position matching the task ID
3. Process only that specific file

Each task then downloads its assigned file, processes it using GDAL's gdalwarp tool, and uploads the result to an output bucket.

The batch job completed processing all 3,111 images in approximately 5 minutes at a total cost of about $0.70. This represents a remarkable 600x speedup compared to the serial approach.

The graph below shows how Coiled rapidly scaled up 100 VMs to handle the processing and then scaled them back down once the work was complete:

This approach provided several key benefits:

• Efficiency: 600x faster than processing serially
• Cost-effectiveness: Only $0.70 for the entire job
• Simplicity: One script and one command to run
• Flexibility: Works with any command-line tool, not just Python

Here are some ways you could extend this example for your own workflows:

• Do different analyses on the data with different GDAL commands
• Run on different datasets by changing the S3 bucket
• Try different container images for other specialized tools