Data scientists increasingly solve large machine learning and data problems with Python. But historically Python struggled with parallel computing. This led many of us in the community to make Dask, a library for parallel computing and data science for Python. Dask has been a go-to solution for scalability in the Python data science stack for …
I ran across an interesting problem yesterday: A company wanted to serve many Dask computations behind a web API endpoint. This is pretty common whenever people offer computation as a service or data as a service. Today the company uses the single-machine Dask scheduler inside of a web request, but they were curious about moving …
Dask in production: Multi-Scheduler architectures Read More »
AWS computation costs roughly the following today: On Demand Spot CPU hour $0.04 $0.0125 GiB hour $0.0045 $0.0015 On top of that different services charge a premium: Premium AWS EMR 40% AWS SageMaker 40% DataBricks 100% However, when you pre-commit to a large allocation then you can usually negotiate this down, and get …
Scaling Data Science is a Team Sport An increasing number of organizations need to scale data science to larger datasets and larger models. However, deploying distributed data science frameworks in secure enterprise environments can be surprisingly challenging because we need to simultaneously satisfy multiple sets of stakeholders within the organization: data scientists, IT, and management. …
The Unbearable Challenges of Data Science At Scale Read More »
Summary An increasing number of organizations need to scale data science to larger datasets and larger models. However, deploying distributed data science frameworks in secure enterprise environments can be surprisingly challenging because we need to simultaneously satisfy multiple sets of stakeholders within the organization: data scientists, IT, and management. Solving simultaneously for all sides of …
Scaling Data Science is a Team Sport An increasing number of organizations need to scale data science to larger datasets and larger models. However, deploying distributed data science frameworks in secure enterprise environments can be surprisingly challenging because we need to simultaneously satisfy multiple sets of stakeholders within the organization: data scientists, IT, and management. …
Scaling Data Science is a Team Sport An increasing number of organizations need to scale data science to larger datasets and larger models. However, deploying distributed data science frameworks in secure enterprise environments can be surprisingly challenging because we need to simultaneously satisfy multiple sets of stakeholders within the organization: data scientists, IT, and management. …
TL;DR Dask, the open source package for scalable data science, was developed to meet the needs of modern data professionals. This post describes the evolution of the Dask project and how it meets the needs of people working with medium-to-large datasets across industries (such as energy, finance, and the geosciences) and basic research (such as …
Motivation People new to dask often ask “Who uses Dask?” They typically mean one of two questions: Do many people use Dask? Do people in my field use Dask? The answer to both questions is “definitely”. Dask users are numerous and varied. The user community spans a wide range of applications and professions. This post …
Summary Deploying data science and machine learning frameworks to data science teams is made complex by organizational constraints like security, observability, and cost management. This post lays out the challenges that arise when exposing scalable computing to data science teams in large institutions, and the enterprise infrastructure necessary to meet those challenges. We then finish …
Challenges of Scaling Data Science in Enterprise Read More »
This post lays out the different stages of openness in Open Source Software (OSS) and the benefits and costs of each. Motivation “Open Source Software” is a hot term today. As a result, people are reasonably encouraged to open up their software. This is great, but means that the term “open source” can get a …
Last month I announced that I was forming a Dask company. This month I am pleased to announce Coiled Computing, a Dask company. This post outlines what this company will do, and the various choices that I’ve made in its configuration. You may also want to … Visit https://coiled.io Follow @CoiledHQ on twitter What will …
