Coiled Heartbeat: September 2021
• September 30, 2021
Welcome to the very first edition of the Coiled Heartbeat, a monthly publication that will bring you an overview of all the latest features, updates, and bug fixes in Coiled.
September was a big month for Coiled with lots of great news to share!
Here are the 3 most important updates:
- Adaptive scaling allows you to work with auto-scaling Dask clusters that automatically adjusts the number of workers based on a number of scaling heuristics.
2. Coiled now includes an Analytics dashboard so you can track your (team’s) cluster usage statistics, historical costs, performance reports, and detailed task metrics and timing information for each cluster.
3. Full support for Google Cloud Platform (GCP) is now available for all users, meaning everyone can now configure Coiled to run in their own GCP account and project.
Read more about each of these (and all of the other updates) below!
You can now instruct your Coiled clusters to automatically scale depending on their workloads using the coiled.Cluster.adapt() method. The adapt method allows you to specify a range between the minimum and the maximum number of workers, as shown in the below code example. Coiled will handle scaling the number of workers in the cluster up or down for you based on a number of scaling heuristics. Learn more about adaptive scaling in Coiled documentation.
import coiled cluster = coiled.Cluster() cluster.adapt(minimum=2, maximum=40)
Coiled now includes an “Analytics” page with an overview of your activity on Coiled, including total compute time, the number of tasks and workers run, and other usage statistics and visualizations. In addition to account-level statistics, you can view detailed activity on a given cluster in your account, including timing data, cost, and profiling information.
Deploy Dask on GCP
Coiled now provides full support for running managed Dask clusters on Google Cloud Platform. You can choose to run clusters within the Coiled multi-tenant GCP environment or in your own GCP account to make use of data access controls, compliance standards, and promotional credits that you already have in place. Read more about setting Coiled up with GCP in our documentation.
We love hearing your ideas!
We have a bunch of product ideas in our pipeline that we’d love to get your feedback on. Below is a selection, you can see the complete list and leave feedback on our Feedback page:
– Ability to create heterogeneous CPU/GPU clusters and route tasks to appropriate workers
– Streaming integration and use cases with services like Kafka, streamz, etc.
– SQL examples and integration
We’d love your input on any of these features in development…and we’re always looking for people who enjoy testing and breaking things 🙂 If that’s you, please drop us a line in the Coiled Community Slack channel!
– Deprecated the ECS backend for Coiled clusters and migrated all users AWS EC2 VMs. The EC2 VMs provide a better balance of performance, consistency across cloud providers, explicit control over CPU/RAM/GPU resources, ease of use for working with larger instances and GPUs, and scalability. Read more here.
– Expanded the Azure documentation for users to configure Coiled to run in their own accounts (reach out to us if you are interested in trying it out!).
– Improved documentation for Teams to make for better distinction between Accounts and Teams.
– Added the
account kwarg to
coiled.create_notebook() to be consistent with similar actions in the Coiled API.
– Removed mention of deprecated
region kwargs for
– Improved log messaging.
– Updated CUDA drivers on Dask workers to version 11.2.
For a complete list of updates and bug fixes, see the Release Notes page in our docs.
Get in touch
Thanks for reading! And if you’re interested in trying out Coiled Cloud, which provides hosted Dask clusters, docker-less managed software, and one-click deployments, you can do so for free today when you click below.