Training

We train Python developers to operate effectively at scale.

Courses are taught on-site or delivered virtually in small groups of 10-20 people. We mix short lecturettes with hands-on Jupyter notebooks with instructors guiding progress and providing direct help.

Upcoming Virtual Training

Coiled offers prescheduled virtual training where you learn from the experts in a small group setting. Sign up now as space is limited.

Course Catalog

Click on a title below to see details for that class.

This class delivers a brief introduction to the Python language and ecosystem, as well as a look at how enterprise software development practices operate for Python applications.

The purpose of the class is not to replace the amazing, in-depth Python courses and bootcamps offered throughout the Python world, but rather to get a practitioner up and running with the core elements of Python in the minimal possible time.

Fundamentals of Python is recommended for experienced teams new to Python, or institutions where Python may be joining or replacing other technologies for large-scale computation.

Learn Ideas and Gain Skills

  • Read and write Python code using core language constructs
  • Use external packages (libraries) and manage your coding environment
  • Explore notebooks, IDEs, and debuggers
  • Discuss options for type annotation, testing, linting and automated CI/CD (continuous integration / continuous delivery)
  • Learn about the Python community

Duration: one day

Prerequisites

Experience programming in one or more other (non-Python) languages in a large project or enterprise setting

Topics

Introduction

  • Scope, goals, and non-goals for a one-day Python introduction
  • How and why Python came to be so widely used

Core Language Constructs

  • Creating and modifing variables
  • Looping and branching
  • Writing and calling functions
  • Classes and inheritance

Python Packages

  • Creating a package
  • Using someone else’s package
  • Finding libraries: PyPI, pip, and Conda

Python Environments

  • The problem with dependencies
  • Creating environments with Conda
  • Managing your environments

Landscape of Coding and Debugging Tools

  • Text editors and pdb
  • IPython, Jupyter, and JupyterLab
  • PyCharm, Spyder, and VS Code

Testing

  • doctest
  • unittest
  • pytest

Formatting, Linting, and Type Annotations

  • PEP-8
  • Black
  • pylint
  • flake8
  • Overview of Python type hints, mypy, pros and cons

Continuous Integration / Continuous Delivery

  • General approach for Python projects
  • Example with CircleCI

Participating in and Learning from the Python Community

  • PSF (PyCon, PySlackers, Discord)
  • PyData, SciPy

This technical, hands-on overview class offers a brief but precise look at the many aspects of deploying and programming Dask, beginning with fundamentals of parallelism and Dask Dataframe, a model for large-scale tabular data based on extending Pandas. The class also explores visualizing the system using Dask’s real-time, animated dashboards.

Dask Overview is recommended as a standalone class or as the first day of a deeper, multi-day exploration of Dask.

Learn Ideas and Gain Skills

  • How Dask fits into the Python and big data landscape
  • How Dask can help you process more data faster, from a laptop up to a big cluster
  • Get started coding with Dask
  • Analyze data and train ML models with Dask

Duration: one day

Prerequisites

Python, basic level
PyData stack (Pandas, NumPy, scikit-learn), basic level

Topics

Introduction

  • About Dask – what it is, where it came from, what problems it solves
  • Options for setting up and deploying Dask

Parallelize Python Code

  • Fundamentals of parallelism in Python
  • concurrent.futures
  • Dask Delayed, Futures
  • Example: building a parallel Dataframe

Dask Dataframe

  • How Dask Dataframe works
  • Partitions, reading and writing data
  • Pandas-style analytics with Dask Dataframe
  • Applying custom computations to a Dataframe

Dask Array

  • How Dask Array is related to NumPy NDArray
  • Operations on Dask Array

Dask Graphical User Interfaces

  • Monitoring workers, tasks, and memory
  • Principal performance and troubleshooting challenges with big data
  • Using Dask’s dashboards to understand performance
  • Debugging and profiling user code

Machine Learning

  • Scikit-Learn style featurization with Dask
  • Algorithm support and integration
  • Modeling

Best Practices

  • Managing partitions and tasks
  • File formats
  • Caching
  • Integrating with more Python (and non-Python!) tools like xgboost, plotting libraries, and GPUs
  • Q & A

This class module focuses on Dask Dataframe, a simple model for working with tabular data that may be too large to fit in memory or to process on a single machine.

The Dask Dataframe class is recommended for engineers or data scientists who typically work with tabular (row/column) data and related tools, like SQL databases.

Learn Ideas and Gain Skills

  • How Dask Dataframe extends Pandas to larger datasets
  • How to select, filter, transform, and join data
  • Understand performance with partitioning and indexes

Duration: half-day or full day

Prerequisites

  • Python, basic level
  • Pandas and/or SQL, basic level

Topics

Introduction

  • Python and Pandas for tabular data
  • Limitations of Pandas
  • Dask Dataframe model
  • Key similarities/differences compared to Pandas

Core Dataframe APIs and Operations

  • Reading data
  • Selecting records and columns
  • Using indexing to select records
  • Filtering datasets
  • Combining datasets (joins, unions)
  • Custom functions
  • Aggregations and sorting (groupby, sort)
  • Custom aggregation
  • Window (rolling) operations

Data access

  • Read CSV and Parquet data and best practices for performant reading
  • Read JSON and text data with Dask bag
  • Read custom formats with Dask delayed
  • Writing data efficiently for future access

Best Practices

  • General query improvement patterns
  • Minimizing expensive data transfer
  • Launching work and preserving results with “persist”
  • Partition sizing

This class focuses on Dask Array and Xarray, two libraries to analyze large volumes of multidimensional data.

Recommended for engineers or data scientists who typically work with large volumes of imaging, geophysical, or other data that might come in image, HDF5, or NetCDF files.

Learn Ideas and Gain Skills

  • How Dask Array extends NumPy to larger datasets
  • Select, filter, transform, and apply custom computations to data
  • Leverage XArray indexes and labels to simplify your code

Duration: half-day or full day

Prerequisites

  • Python, basic level
  • NumPy, basic level

Topics

Introduction

  • Python and NumPy for multidimensional data
  • Limitations of NumPy
  • Dask Array model, XArray model

Core Array APIs and Operations

  • Loading Data
  • Slicing, processing, and aggregating data with the Numpy API
  • Chunking data for performance
  • Applying Numpy functions in parallel
  • Stacking data from custom sources
  • Integrating with Numba for performance

Intro to XArray

  • Loading data
  • DataArray and Dataset
  • Dimensions and Coordinates
  • Indexes, selecting data
  • Filtering and aggregating
  • Combining data
  • Plotting
  • Writing output

Best Practices

  • Optimal layouts on disk for performance
  • Managing memory and preserving results with “persist”
  • File formats
  • Chunking data

This class module focuses on Dask Bag, a functional-programming pattern for distributed computation over unstructured or heterogeneous data.

Dask Bag is useful for initial processing of unstructured text, large collections of heterogeneous business records which require special processing, images or diagrams, etc. The class focuses on functional style, the Bag API, and best practices.

Learn Ideas and Gain Skills

  • How Dask Bag applies your Python code to large data collections
  • Transforming, filtering, combining, aggregating and matching objects
  • Addressing performance concerns

Duration: half-day or full day

Prerequisites

  • Python, basic to intermediate level
  • Some knowledge of functional programming is helpful but not required

Topics

Introduction

  • Python functional constructs in the standard library
  • Why use a functional model for “big data”?
  • Dask Bag vs. local Python collections

Core Bag APIs and Operations

  • Ingesting data and creating Bags
  • Understanding partitions
  • Transform and project data with map
  • Understanding execution: Compute, Persist and Visualize
  • Filter data
  • Builtin aggregations: math/stats, conditionals, counting, and sorting
  • Aggregate data with group and fold
  • Combine data with zip and join
  • Writing or retrieving output

Best Practices

  • General query improvement patterns
  • Minimizing expensive data transfer
  • Integration with from_delayed, to_delayed, to_dataframe
  • Partition sizing

This class module focuses on using the Dask scheduler to empower custom parallel computation. Dask Delayed and Futures represent lightweight mechanisms for building and running custom task graphs while staying within traditional Python coding patterns. This combination – regular Python code with a powerful distributed scheduler – enables all kinds of industry or discipline-specific workloads to be parallelized for fast, large-scale computation.

Learn Ideas and Gain Skills

  • How to parallelize algorithms with minimal changes to your code
  • Patterns for scheduling compute against data that may not yet be available
  • Maximizing parallelism and minimizing bottlenecks

Duration: one day

Prerequisites

Python, basic to intermediate level

Topics

Introduction

  • Local parallel programming with ThreadPoolExecutor.map
  • Async with submit() and concurrent.futures
  • Generalizing async code with task graphs
  • Scaling task graphs: basic scheduler idea

Building and Running Graphs with Delayed

  • Creating and running Delayed objects
  • Decorator and explicit Delayed
  • Compute and Persist
  • Inspecting dependencies and task graphs programmatically and visually
  • Allowed operations with Delayed
  • What to do about prohibited operations with Delayed, and why
  • Aggregating data with Delayed
  • Best practices for Delayed

Running and Managing Work with Futures

  • Review of Future/Promise model
  • Launching tasks with submit, map
  • Getting results
  • Inspecting Future status
  • Graphs: Composing Futures
  • Managing Future lifetimes and task scheduling: errors, cancel, wait, fire_and_forget, del
  • Dynamic graphs: as_completed

Review and Q&A

  • Comparison: Futures vs Delayed
  • Using scatter
  • Debugging

This class module focuses on understanding and practical usage of the many graphical reports provided by the Dask dashboard: from real-time data transfer information to a full statistical profiler. For each dashboard panel, we discuss the information presented and then how one might employ that information to draw practical conclusions about improving performance or code.

Learn Ideas and Gain Skills

  • What information is presented in Dask’s graphical dashboards
  • How to convert statistical information from dashboards into actionable understanding of your workload

Duration: half day

Prerequisites

  • Python, basic level
  • Dask programming, basic level

Topics

Introduction

  • General challenges to distributed computing
  • Practical obstacles in common applied scenarios
  • Questions to ask when your code runs slowly or throws errors
  • Locating/activating the dashboard web pages or JupyterLab widgets

Dask Dashboard Reports

  • Resource Utilization
  • Workers
  • CPU
  • Memory (total and by key)
  • Bandwidth (by worker and type)
  • Cluster Map
  • Tasks Processing
  • Progress Bar
  • Task Stream
  • Task Graph
  • Profiler

Applying Dashboards to Investigate a Scenario

  • Workflow overview and goal
  • Sample code
  • Investigating failures
  • Investigating speed/slowdown

Review and Q&A

  • Common patterns
  • Best practices

This class addresses the transition from working successfully on a single server or experimenting with a minimal cluster to achieving successful, reliable, repeatable use of larger Dask compute clusters. We focus on a deep dive into all of the critical components in a distributed Dask cluster, how they work together, and how you can configure them to maximize throughput and minimize costs.

Learn Ideas and Gain Skills

  • What components make up a distributed Dask cluster and what purposes they serve
  • How to configure cluster resources to meet your workload needs
  • How to identify problems, debug, and troubleshoot successfully

Duration: one day

Prerequisites

  • Python, basic level
  • Dask programming, basic level

Topics

Introduction

  • About Dask and Coiled Computing: Making scale-out computing easier
  • Simplest distributed cluster: manual setup
  • Changes in transitioning to distributed environment
  • Storage, fast universal memory access, single shared executable
  • Implications for users (devs) and admin (IT)

Distributed Dask: Cast of Characters

  • Client, Scheduler, Nanny, Worker
  • Where these services are located, their relationships and roles
  • Supporting Players: cluster resource manager (e.g., k8s, Coiled Cloud, YARN, etc.)

Basic Operation of Dask Clusters

  • User perspective
  • Creating clusters with helper tools: Cloud Provider, Coiled Cloud, etc.
  • Cluster API
  • Sizing your cluster
  • Scaling your scaling – manual/automatic
  • Admin perspective
  • CLI: dask-scheduler and dask-worker
  • Managing the worker environment
  • Additional admin concerns (security, tagging, and costs)

Tasks

  • Submitting tasks and directing output
  • Scheduling policy
  • Finding your tasks and data (programmatically)
  • Seeing your tasks and data: the Dask Dashboard

Distributed Data

  • Source data via tasks
  • Source data scatter
  • Storing data worker-local
  • Handling output (result) data, direct parallel write vs. gather/result

Resource usage and Resilience

  • Output spill location and resource management
  • Work stealing
  • Loss of processes
  • Loss of storage on workers

Best Practices, Debugging

  • Dashboard information pages
  • Additional GUIs (e.g., profiler)
  • Review of best practices
  • Remote debugging
  • client.run

Use Case Example: Orchestrating Batch ML Scoring

  • Source data on disk
  • ML model
  • Options for inference, pros/cons
  • Supplying dependencies via code or container image
  • Basic workflow
  • Improvements and optimizations (e.g., batch size)

Q&A, Discussion

This class focuses on leveraging Dask for Machine Learning in several different ways: Dask implements a number of distributed algorithms; interoperates with popular Python libraries, and integrates with several external projects (e.g., PyTorch). This module looks at each of the options, as well as the full ML lifecycle, from ingesting data to performing inference.

Learn Ideas and Gain Skills

  • What does Dask offer – and not offer – for machine learning workflows
  • Leveraging Dask for proper out-of-core and/or parallel training
  • Implementing an end-to-end workflow with Dask and other tools

Duration: half-day or full day

Prerequisites

  • Python, basic level
  • Understanding of ML concepts and workflow, basic level
  • Dask programming, basic level

Topics

Introduction

  • What makes scale-out machine learning different and challenging
  • How Dask flexibly approaches distributed ML challenges
  • Using Dask with – not instead of – other tools
  • Optional: Dask’s model for enabling custom algorithms

Dask and scikit-learn

  • Hyperparameter Search
  • Out-of-core non-parallel training (incremental)
  • In-memory parallel training
  • Combining incremental (out-of-core) and parallel training
  • Review of scikit-learn + Dask helper APIs
  • How to match scikit-learn algorithms to Dask options

Data Preparation and Dask’s Algorithms and Integration with XGBoost

  • Ingesting data
  • Feature engineering transformations
  • Model training (GLM, clustering)
  • Pipelines
  • Dask + xgboost
  • Optional: Dask + GPU (overview)

Performing Inference at Scale

  • General patterns for inference
  • Predicting with Dask Futures
  • About ParallelPostFit
  • Optional: Low-latency vs. batch vs. resource-intensive inference patterns

Custom Algorithms (Optional Intro, full-day only)

  • Dask’s task scheduler
  • Mechanisms for distributing, sharing, aggregating, and collecting data
  • Example: implementing a simulation model

Review and Q&A

  • Gotchas and best practices
  • Architecture options for integrating Dask

This class explores the best ways to leverage Dask within enterprise data architectures.

Most enterprises make heavy use of elements core to Dask (e.g., data manipulation and machine learning); activities external to Dask (e.g., using SQL for reporting and data extraction); and activities orthogonal to Dask but still critical to the success of the overall system (e.g., data storage). Moreover, staff and skillsets are often different across these areas.

We explore options and patterns for getting the best out of both Dask and non-Dask elements of the system.

Learn Ideas and Gain Skills

  • Where is Dask great? and Where do we need additional tools?
  • Integrating Dask with JVM-based data processing systems
  • Best practices to allow your data team to excel with the skills they know best

Duration: half-day or full day

Prerequisites

  • Python, basic level
  • JVM/Hadoop/Spark/Kafka ecosystem, basic level
  • Large-scale data storage patterns, basic level
  • Understanding of ML concepts and workflow, basic level

Topics

Introduction

  • About Dask and Coiled Computing: Making scale-out computing easier
  • Dask, the 30,000-foot view: scheduler, APIs, infrastructure support components
  • Photographic negative: what’s not in Dask
  • Overview of an integration flow from data warehouse to reports or ML models

Integrating Data

  • Finding your data: Hive metastore and alternatives
  • Ingesting data: formats and locations
  • Options for SQL access to data
  • Distributed caching
  • Consuming streaming data

Data Processing, ETL, and Feature Engineering

  • Dask support
  • Shuffling and other data transfer
  • External Python integrations
  • Custom functions / business logic
  • Checking compatibility vs. ANSI SQL, SparkSQL, HiveQL
  • Index pros/cons
  • ML Modeling – orientation/overview
  • Implementing custom algorithms

Data Output

  • Output artifacts
  • ML Models
  • Reports for human or business system consumption
  • ETL writes into another datastore
  • Output to a streaming or message-oriented middleware system
  • Transactional/safe writes – present and future

Additional Goals, Challenges, and Opportunities

  • Orchestration
  • Resilient streaming systems
  • ML serving/scoring systems
  • GPU / heterogeneous compute integration
  • Monitoring, management, and debugging interfaces
  • End-user notebook integration
  • Q & A

This class introduces tools for GPU-accelerated computation – including CuPy, RAPIDS, and Numba – and their integration with Dask for usage on large datasets and multiple GPUs.

Learn Ideas and Gain Skills

  • What does Dask offer – and not offer – for machine learning workflows
  • Leveraging Dask for proper out-of-core and/or parallel training
  • Implementing an end-to-end workflow with Dask and other tools

Duration: one day

Prerequisites

  • Python, basic level
  • Understanding of ML concepts and workflow, basic level
  • Dask programming, basic level

Topics

Introduction

  • Lessons from big data tools
  • NumPy and vectorized compute
  • Native code generation
  • Adding hardware acceleration

Patterns for High-Performance Python

  • Understanding Python limitations and escape hatches
  • Concurrency issues and patterns: multithreading, GIL, multiprocess, multinode
  • JIT compilation with Numba
  • PyTorch and CuPy: not just for deep learning
  • Leveraging DL libraries for “regular” ML

NVIDIA RAPIDS

  • Data catalogs and using SQL on the GPU
  • Working with cuDF: CUDA DataFrame
  • A Bit About How GPU Programming Works Under the Hood
  • Numba and Python-to-CUDA compilation
  • cuDF + Numba: Custom Computation from Python
  • CUDA Machine Learning with cuML
  • Interop with PyTorch for Deep Learning and General Optimization
  • cuGraph High-Level CUDA-enabled Graph Operations

RAPIDS and Dask

  • Multi-GPU and multi-node scaling
  • Combining Dask + GPU
  • ML, dimensionality reduction, and graph algorithms on GPU

Architecture and Problem Solving

  • Extending the end-to-end workflow
  • Accelerated networking
  • Visualizations
  • Architecture, patterns, integration
  • Q & A

Sign up for updates