Dask Heartbeat by Coiled

James Bourbeau • December 17, 2020

Dask, dask-sql, data science, machine learning, SQL, XGBoost

Introduction

The Dask community is highly distributed with different teams working independently. This is powerful but sometimes makes it hard for people within the community to see everything that is going on. The Dask Heartbeat by Coiled is a bi-weekly publication intended to centralize and broadcast Dask news over the previous two weeks.

If you want something added to this list either send an e-mail at info@coiled.io, or tweet and tag @dask_dev and we’ll try to include it.

Dask 2020.12.0 release

Dask and Distributed version 2020.12.0 was released last week. This release contains many updates (it’s the first release in two months). Some highlights include:

Switching to CalVer for the versioning scheme. We plan to write more about the motivations for this next month. The previous release was version 2.30.1, while this version is 2020.12.0.
The scheduler can now receive Dask HighLevelGraphs instead of raw dictionary task graphs. This allows for a much more efficient communication of task graphs from the client to the scheduler. This is currently off by default but is configurable for early adopters with the optimization.fuse.active config value.
Introduction of new HighLevelGraph layer objects including BasicLayer, Blockwise, BlockwiseIO, ShuffleLayer, and more.
Added support for applying custom Layer-level annotations like priority, retries, etc. with the new dask.annotate context manager.

XGBoost 1.3.0 release

The newly released version 1.3.0 of XGBoost contains several updates that improve XGBoost + Dask integration. This is part of the larger effort to migrate the functionality of Dask-XGBoost into the mainline XGBoost codebase.

NVTabular is now built on Dask-CuDF

NVTabular, a library for processing tabular data needed to train and deploy recommender-systems models on GPUs, introduced a new Dask-CuDF backend to support scalable preprocessing. Rick Zamora (NVIDIA) outlines some recent NVTabular developments in this blogpost https://medium.com/rapids-ai/nvtabular-all-in-on-dask-6241b4e9ca19.

NVTabular from @NVIDIAAIDev is now built on @RAPIDSai @dask_dev Dask-CuDF backend that simplifies programming while enabling performant preprocessing that scales for #RecSys pipelines and better integration with the RAPIDS ecosystem. https://t.co/1QVEX2rY5O
— RAPIDS AI (@RAPIDSai) December 11, 2020

Dask-SQL Updates

Nils Braun (Bosch) shares his blogpost using SQL to drive Dask on Kubernetes

Learn how to quickly get started with @dask_dev and dask-sql on @kubernetesio and query your (big) data with Machine Learning from SQL (e.g. @XGBoostProject in this example) https://t.co/nhImS8C5rv
— Nils Braun (@_nilsbraun) December 8, 2020

Also, in other fun Dask/Pandas/SQL news, we discover that the Dask-SQL project also magically works on Pandas.

Did you know that you can use dask-sql to SQL-query "normal" @pandas_dev data frames (additional to @dask_dev data frames, S3, hdfs, az)? Try it out now! https://t.co/yapPcPCZdX pic.twitter.com/fNeQj1isUW
— Nils Braun (@_nilsbraun) December 4, 2020

This is one nice side effect of the close partnership between the two projects.

Stumpy 1.6.0 release

The STUMPY library for time series analysis improves its dask support in its recent release.

🎉 .@stumpy_dev v1.6.0 is now available on .@condaforge and .@pypi with new features for computing a “consensus motif”, a novel distance measure called “MPdist” for clustering, & we improved support/stability for .@dask_dev! 🙏 Let us know what you think: https://t.co/KHO4M8phJG
— stumpy (@stumpy_dev) December 16, 2020

yt integration continues

Maintainers of the popular yt framework for computation and visualization of volumetric data are busy implementing Dask support. A slide deck on their recent progress is below

Was great presenting at @yt_astro's RHytHM workshop today on my work in leveraging @dask_dev within the yt framework! Slides (with notebook links to some prototypes) here: https://t.co/pK4zmoUY7k
— Chris Havlin (@s_i_r_h_c) December 10, 2020

Quansight delivers Dask Webinar

Dhavide Aruliah (Quansight) https://twitter.com/quansightai/status/1334161550504968192

CZI EOSS Grantee Program

Ben Zaitlen (NVIDIA) presented Dask to OSS maintainers and Life Science practitioners at the Chan Zuckerberg Initiative’s Essential Open Source Software for Science gathering.

Video available here (Dask was on Day Three at the end)

Slides available here

We’re also glad to announce that Genevieve Buckley will be joining full time in February as the Dask Life Science fellow (generously funded by the CZI EOSS program). We’ll have a more detailed announcement next month, and are very excited. Genevieve will be the first employee of Dask itself as an organization, rather than one of the supporting companies.

Deploying Jupyter for Dask on ARM on Kubernetes

Holden Karau walks through how to deploy Jupyter Lab/Notebook on ARM on Kubernetes with Dask support in this blogpost https://scalingpythonml.com/2020/12/12/deploying-jupyter-lab-notebook-for-dask-on-arm-on-k8s.html

KNN blogpost by NVIDIA RAPIDS

Scaling KNN has never been faster or easier. Jump in and run the distributed @RAPIDSai cuML and @dask_dev implementation demo designed to experience true #HPC performance at scale on #GPU. https://t.co/DPTZ5wgxQT
— RAPIDS AI (@RAPIDSai) December 9, 2020

Activity at annual AGU conference

The American Geophysical Union runs an annual conference. Dask took this community by storm a couple of years ago with the Pangeo project. This year is no different

So far it looks like nearly every instance of data-proximate computing for CMIP uses @ProjectJupyter and @dask_dev technology. 🔥
— Ryan Abernathey (@rabernat) December 11, 2020

For reference, CMIP is the Climate Model Intercomparison Project. It’s the standard multi-institutional model for climate change and one of the grander humanity-focused projects we see today.

There are many other happenings at this conference, including this announcement from the climpred project

Interested in forecast verification? #AGU20

I pitch climpred (https://t.co/udzlm2Dski dev w/ @rileyxbrady @AndrewDrew1 @KPegion) powered by @dask_dev @xarray_dev @pangeo_data

A110 – Advances in ESM Across Timescales I eLightning https://t.co/m3mcLXnVw3

Friday, 11th 13:39 CET pic.twitter.com/LdtFJr9XA2
— aaronspring (@realaaronspring) December 10, 2020

2i2c is hiring

2i2c is looking to hire an open-source infrastructure engineer to work on cloud infrastructure for research and education using projects like JupyterHub and Dask. For more information, see their job posting at https://2i2c.org/job/osie-pangeo.

Micro-optimizing and refactoring the Distributed Scheduler

John Kirkham (NVIDIA) has continued making micro-optimization of the scheduler as part of a larger effort to boost performance:

And has recently begun to decouple the state machine and networking communication parts of the scheduler.

Dask-Jobqueue 0.7.2 release

See https://jobqueue.dask.org/en/latest/changelog.html for the full list of changes.

Wrapping Up

That’s it. Thanks for reading all.

If you’re interested in taking Coiled Cloud for a spin, you can do so for free today when you click below.

Try Coiled Cloud