Overview
Distributed mode lets you split a ModularPipelines pipeline across multiple processes or machines. Instead of running every module in a single process, work is divided between a master that orchestrates and one or more workers that execute modules.
Getting Started
This guide walks you through adding distributed execution to an existing ModularPipelines project using Redis as the coordinator.
Configuration
Distributed mode has two layers of configuration: the core DistributedOptions (shared across all coordinator implementations) and coordinator-specific options like RedisDistributedOptions.
Capabilities and Routing
Not every worker can execute every module. Some modules need Docker, others need a specific OS. The capability system controls how modules are routed to the right worker.
CI Example: GitHub Actions
This is a complete example of running a distributed pipeline across GitHub Actions matrix runners using Redis for coordination.
Architecture
This page describes the internal architecture of distributed mode for contributors and advanced users.