Dask Heartbeat by Coiled: August 2021

The Coiled Team August 26, 2021


Introduction

The Dask community is highly distributed with different teams working independently. This is powerful but sometimes makes it hard for people within the community to see everything that is going on. The Dask Heartbeat by Coiled is a monthly publication intended to centralize and broadcast Dask news over the previous month.  

If you want something added to this list either send an email at info@coiled.io, or tweet and tag @dask_dev and we’ll try to include it. Keep reading for the latest updates.

Dashboard Improvements

Dask’s diagnostic dashboards have been improved significantly thanks to Ian Rose, Jacob Tomlinson, and Naty Clementi. The updates include:

High-Level Graphs

Freyam Mehta, Genevieve Buckley, Jacob Tomlinson, and others are doing exciting work around making task scheduling faster using high-level graphs. You can read more about the overall objectives in Faster Scheduling. As Genevieve writes in High Level Graphs update, there is ongoing work to use a Blockwise high-level graph layer wherever possible, investigate a high-level graph for Dask’s `map_overlap`, and visualize high-level graphs in Jupyter Notebooks.

NumPy histogramming API in dask.array

Doug Davis helped add support for Dask Array equivalents of NumPy’s `histogram2d` and `histogramdd` functions. This feature is available in Dask version 2021.07.1 and above.

Ongoing Improvements to Memory Management and Scheduling

Guido Imperiale has continued working on active memory management and as of version 2021.07.2, the MALLOC_TRIM_THRESHOLD_ environment variable is set automatically on workers. Gabe Joseph from Coiled also continued improving Dask’s memory scheduling by short-circuiting root-ish checks for some group dependencies.

Releases

Over the month of July, both Dask and Distributed versions 2021.07.0, 2021.07.1, and 2021.07.2 were released.

Dask Monthly Community Meeting 

Some highlights from the July Dask community meeting:

Full meeting notes are available here.

You’re All Caught Up On Dask

That’s it. Thanks for reading.

If you’re interested in taking Coiled Cloud for a spin, which provides hosted Dask clusters, docker-less managed software, and one-click deployments, you can do so for free today when you click below.

Try Coiled Cloud