How the National Snow and Ice Data Center processed 200 TB of data for $150

The National Snow and Ice Data Center (NSIDC) serves as one of NASA's 11 Distributed Active Archive Centers (DAACs), responsible for hosting and distributing critical satellite data related to Earth's cryosphere: the frozen parts of our planet including glaciers, ice sheets, and sea ice.

This data is essential for tracking climate change, predicting sea-level rise, and understanding Earth's complex systems. As NASA's Earth-observation fleet has grown more sophisticated, so has the volume of this data. NSIDC now manages over 6 petabytes of information from various missions, from satellites launched in the 1980s to today's advanced instruments.

The center's most data-intensive mission is ICESat-2, a satellite launched in 2018 equipped with a laser altimeter that measures the height of Earth's surface with unprecedented accuracy.

While primarily focused on measuring millimeter-scale changes in ice thickness, the satellite collects data across the entire globe. Scientists now use this data for applications ranging from measuring sea roughness to detecting theoretical rock waves in the Pacific Ocean that were previously impossible to observe.

Like many scientific institutions, NSIDC is transitioning from a traditional archive facility into what NASA calls a "data science enabling center" that not only stores data but helps scientists access and process it efficiently.

This transition creates a gap between scientific expertise and cloud technology. Scientists who have developed expertise in fields like glaciology or remote sensing now need to navigate complex cloud services to perform their analyses.

As a software engineer at NSIDC, Luis Lopez works to bridge these worlds. Half of his role involves creating tools and training that make cloud computing accessible to researchers who need to focus on their scientific work rather than becoming cloud experts.

When searching for solutions, Luis discovered Coiled, which offered a refreshingly simple approach to cloud computing:

For scientists working with specialized data formats and libraries, Coiled's environment synchronization eliminates a major pain point:

Luis has used Coiled for numerous projects at NSIDC. For his most recent project, he worked with JPL on the "It's Live" project. This initiative maps the velocity of glaciers worldwide to understand their flow into oceans, a critical factor in sea level rise predictions.

This project required processing data from multiple satellite missions spanning decades, from Landsat 5 launched in the 1980s to today's Landsat 8, 9, and Sentinel satellites. The heterogeneous nature of the data complicated processing, with approximately 200 terabytes to analyze.

Using Coiled's batch functionality, Luis created an efficient processing pipeline that transformed this computational challenge into a manageable task:

The results were remarkable. With Coiled, the team processed 200 terabytes of data in less than a day:

The cost efficiency was equally impressive:

For scientific institutions like NSIDC, computational efficiency translates directly to research acceleration and more effective use of limited research funds. The most significant impact has been removing barriers that previously limited scientific exploration.

Without Coiled, processing datasets at this scale would require custom infrastructure development or complex cloud configurations that most scientists simply wouldn't attempt:

This difference in effort changes the nature of scientific inquiry. With a simplified path to computation, scientists can ask bigger questions and process more comprehensive datasets:

For budget-conscious scientific institutions, the cost transparency Coiled provides is equally valuable:

This visibility removes the financial uncertainty that often prevents scientists from utilizing cloud resources, allowing them to make informed decisions about the scale of their analyses.

With the success of projects like "It's Live," Luis now envisions creating more accessible data transformation pipelines for the scientific community. He's particularly excited about developing an on-demand Pangeo Forge workflow that would democratize access to transformed satellite data.

These data cubes make complex datasets more accessible to researchers who lack the computational expertise to process raw satellite data. By using Coiled as the computational engine for this transformation, Luis hopes to create a system where more scientists can generate derived data products:

This approach could transform how scientists interact with NASA's data archives, allowing them to focus on analysis and discovery rather than data engineering. For a field where technological barriers often restrict scientific progress, simplifying access to computation could accelerate research across multiple disciplines.

The work Luis is doing at NSIDC represents a successful bridge between scientific expertise and cloud technology, two domains that often struggle to connect effectively. By providing simple, straightforward tools for complex cloud operations, Coiled allows scientists to leverage modern computational capabilities without a steep learning curve.

As climate research becomes increasingly critical for understanding our changing planet, tools that accelerate scientific discovery and make data more accessible play an essential role. For scientists studying the cryosphere, the ability to process petabytes of data efficiently means better models, more accurate predictions, and ultimately more informed climate policies.

The future of scientific computing lies not just in more powerful machines but in more accessible tools that bring computational abundance to experts across disciplines. Through the bridge Luis is building between scientific expertise and cloud computing, he's helping fulfill NASA's vision of not just archiving data, but enabling the science that makes that data valuable.