Dask Heartbeat

Matt Rocklin January 14, 2021

Dask Heartbeat by Coiled

, , , , , ,


The Dask community is highly distributed with different teams working independently. This is powerful but sometimes makes it hard for people within the community to see everything that is going on. The Dask Heartbeat by Coiled is a bi-weekly publication intended to centralize and broadcast Dask news over the previous two weeks.  

If you want something added to this list either send an e-mail at info@coiled.io, or tweet and tag @dask_dev and we’ll try to include it.

Dask turns six years old 

We can celebrate Dask’s birthday!  Dask was created a little over six years ago.  

Bugs in latest release

Last month Dask had a major release, 2020.12.0, which included some significant internal changes. Unsurprisingly, some of these changes had negative effects which Dask maintainers are busy trying to resolve now.

Worker loses track of keys

Under a heavy load, the `Worker` can sometimes skip a beat, and lose a key.  This is currently being resolved in https://github.com/dask/distributed/pull/4360 by Florian Jetter (Blue Yonder) and Gil Forsyth (Capital One).

This comes from a large change restructuring the `Worker`s task state, which should result in better long term maintenance.

Data Reading Errors

Data ingestion operations like reading Parquet or CSV files can sometimes result in serialization issues.  This is being resolved by Rick Zamora (NVIDIA) at https://github.com/dask/dask/pull/7042 

JupyterLab 3.0

With the recent JupyterLab 3.0 release, the infrastructure to load extensions has been heavily modified.  This has resulted in a needed refresh of the Dask-JupyterLab extension.  Ian Rose (Coiled) is handling this here: https://github.com/dask/dask-labextension/pull/162

Update: this is done!

Thank you for your patience

There has been a lot of activity over the last few months, which is great to see.  However, this has also resulted in higher-than-typical churn and we thank you for your patience.

Xarray user survey

Xarray, a Dask-related project, is publishing its annual user survey.  If you are an Xarray user then we encourage you to participate here: https://docs.google.com/forms/d/e/1FAIpQLSfhVUao634zgpWP3BdrMPwzCd3WUqRbZZ4Baq_l2shoMhcIlQ/viewform

Chan-Zuckerberg Biohub job posting

Chan Zuckerberg is hiring a bio-imaging scientist to work with Dask on large scale light-sheet microscopy.  Learn more here: https://apply.workable.com/czbiohub/j/F87328FDEA/ 

New User Story: Biomarkers for rare diseases

Markus Schmitt (Data Revenue) added a new story on discovering rare diseases using Dask’s lower-level APIs.  You can read more here:  https://stories.dask.org/en/latest/datarevenue.html

Wrapping Up

That’s it. Thanks for reading all.

If you’re interested in taking Coiled Cloud for a spin, you can do so for free today when you click below.