Coiled Blog
Accelerating Data Science, Python and Open Source
Sort By:
This blog post explains how to create Dask Arrays from Zarr files using the from_zarr function. The from_zarr function reads the data into a Dask array, allowing the user to work with arrays larger than memory using all of the available cores. https://bit.ly/3H0Vag6
Memory spilling is an important feature that makes it possible to run Dask applications that would otherwise run out of memory. When low on memory, Dask moves data from GPU memory to main memory and/or data from main memory to disk automatically. https://bit.ly/3QtkmiE
This blog post explains how to encode categorical variables with @dask_dev and how to prepare categorical variables for #machinelearning algorithms.
Using categorical variables correctly can give you significant performance boosts and speed up your data analyses.
New blog post: Understanding Dask's meta keyword argument✨
Learn:
- What the is meta keyword argument,
- Why does Dask need meta, and
- How to use it effectively.
https://blog.dask.org/2022/08/09/understanding-meta-keyword-argument
Sharing a quick win:
We noticed an increase in worker memory usage in a benchmark a few days ago, and merged a fix yesterday. Thanks, Guido Imperiale!
Learn how Dask handles memory usage in tricky situations: https://distributed.dask.org/en/stable/worker-memory.html#memory-not-released-back-to-the-os
Relevant issue: https://github.com/dask/distributed/issues/6833
Don't forget to register for our webinar tomorrow, August 9th at 10:00 am EST, with @AccureBattery Infrastructure Chapter Lead Dr. Severin Ryberg, and discover how ACCURE manages a large volume of data using Dask & Coiled. https://bit.ly/3Q3Ek3E
