Coiled Heartbeat: September 2021

The Coiled Team September 30, 2021

, , ,


Welcome to the very first edition of the Coiled Heartbeat, a monthly publication that will bring you an overview of all the latest features, updates, and bug fixes in Coiled. 

September was a big month for Coiled with lots of great news to share!

Here are the 3 most important updates:

  1. Adaptive scaling allows you to work with auto-scaling Dask clusters that automatically adjusts the number of workers based on a number of scaling heuristics.

How important is the new adaptive scaling capability to you?

View Results

Loading ... Loading ...

2. Coiled now includes an Analytics dashboard so you can track your (team’s) cluster usage statistics, historical costs, performance reports, and detailed task metrics and timing information for each cluster.

How important is the new analytics dashboard to you?

View Results

Loading ... Loading ...

3. Full support for Google Cloud Platform (GCP) is now available for all users, meaning everyone can now configure Coiled to run in their own GCP account and project.

How important is full support for Google Cloud Platform (GCP) to you?

View Results

Loading ... Loading ...

Read more about each of these (and all of the other updates) below!

Adaptive Scaling

You can now instruct your Coiled clusters to automatically scale depending on their workloads using the coiled.Cluster.adapt() method. The adapt method allows you to specify a range between the minimum and the maximum number of workers, as shown in the below code example. Coiled will handle scaling the number of workers in the cluster up or down for you based on a number of scaling heuristics. Learn more about adaptive scaling in Coiled documentation.

import coiled

cluster = coiled.Cluster()
cluster.adapt(minimum=2, maximum=40)

Analytics Dashboard

Coiled now includes an “Analytics” page with an overview of your activity on Coiled, including total compute time, the number of tasks and workers run, and other usage statistics and visualizations. In addition to account-level statistics, you can view detailed activity on a given cluster in your account, including timing data, cost, and profiling information.

Deploy Dask on GCP

Coiled now provides full support for running managed Dask clusters on Google Cloud Platform. You can choose to run clusters within the Coiled multi-tenant GCP environment or in your own GCP account to make use of data access controls, compliance standards, and promotional credits that you already have in place. Read more about setting Coiled up with GCP in our documentation.

We love hearing your ideas!

We have a bunch of product ideas in our pipeline that we’d love to get your feedback on. Below is a selection, you can see the complete list and leave feedback on our Feedback page:

– Ability to create heterogeneous CPU/GPU clusters and route tasks to appropriate workers

– Streaming integration and use cases with services like Kafka, streamz, etc.

– SQL examples and integration

Which of these product ideas would you find most valuable?

View Results

Loading ... Loading ...

We’d love your input on any of these features in development…and we’re always looking for people who enjoy testing and breaking things 🙂 If that’s you, please drop us a line in the Coiled Community Slack channel!

Other Updates

– Deprecated the ECS backend for Coiled clusters and migrated all users AWS EC2 VMs. The EC2 VMs provide a better balance of performance, consistency across cloud providers, explicit control over CPU/RAM/GPU resources, ease of use for working with larger instances and GPUs, and scalability. Read more here.

– Expanded the Azure documentation for users to configure Coiled to run in their own accounts (reach out to us if you are interested in trying it out!).

– Improved documentation for Teams to make for better distinction between Accounts and Teams.

– Added the account kwarg to coiled.create_notebook() to be consistent with similar actions in the Coiled API.

– Removed mention of deprecated region kwargs for coiled.Cluster()

– Improved log messaging.

– Updated CUDA drivers on Dask workers to version 11.2.

For a complete list of updates and bug fixes, see the Release Notes page in our docs.

Get in touch

If you have any questions or other feedback, please do reach out to us via our Coiled Community Slack, by writing to support@coiled.io, or by tweeting at us. We’d love to hear from you!

Thanks for reading! And if you’re interested in trying out Coiled Cloud, which provides hosted Dask clusters, docker-less managed software, and one-click deployments, you can do so for free today when you click below.

Try Coiled Cloud