Parallel execution (fan-out / fan-in)

The workflow runtime is sequential at the state-machine level — one active state, one transition at a time. The parallel executor kind adds fan-out inside a single executor invocation: N branches run concurrently, the executor aggregates their results, and the runtime sees one slow-but-normal executor call. CPM (“critical path method”) “crashing” is the mental model — parallelizing independent activities compresses wall-clock without changing CPU work or any state-machine invariant.

The shape

executor:
  kind: parallel
  branches:                            # static list OR { for_each, do }
    - { kind: script, subject: build.cargo.release }
    - { kind: mcp,    connection: scip, tool: lookup, args: { symbol: foo } }
    - { kind: workflow, definitionId: critique-agent, input: { ... } }
  join: all                            # all (default) | any | { at_least: K } | { percent: P } | { expression: "<expr>" } | { aggregator: { kind: ... } }
  max_concurrency: 4                   # REQUIRED when branches.len() >= 10
  on_branch_failure: bail              # bail (default) | continue
  total_timeout_ms: 60000              # optional
  max_recursion_depth: 3               # optional, default 3

Each branch is itself an executor config — anything that the gateway can invoke (script, mcp, cli, rest, workflow, even nested parallel). The schema is recursive.

Dynamic branches (one per item)

branches:
  for_each: "$.context.queries"
  where: "$.value != ''"               # optional pre-fan-out filter; falsy elements dropped before branches spawn
  do:
    kind: mcp
    connection: scip
    tool: lookup
    args: { symbol: "$.branch.value" }

$.branch.value and $.branch.index are template substitution markers replaced per branch when the do: template is expanded. The for_each path resolves against $.context, $.workflow.input, or $.arguments. The optional where: is a string predicate evaluated per element — paths $.value and $.index resolve against the element’s {value, index} projection — and elements that evaluate falsy are dropped before any branch spawns (avoiding the “add a state just to filter” antipattern). Surviving branches keep their original source-array index. If the array (or the post-filter result) is empty, the executor vacuously succeeds and emits a parallel.fanout.empty audit event.

Join conditions

The closed shortcuts (all, any, at_least, percent) cover the common cases. aggregator is the general form — any executor-shaped value invoked post-fan-out that consumes the branches map and returns a verdict. expression is sugar for {aggregator: {kind: expression, expr: "<expr>"}}.

Join	Verdict
all (default)	Succeeds iff EVERY branch succeeded. Any failure fails the whole.
any	Succeeds as soon as ONE branch succeeds; siblings cancelled. Useful for race / hedged-request patterns.
{at_least: K}	Succeeds iff K or more branches succeeded. Failures making K unreachable fail early.
{percent: P}	Succeeds iff at least P percent (0..=100) of branches succeeded. Threshold is ceiling-rounded (51% of 3 needs 2, not 1). Vacuous fan-out (n=0) succeeds. Early exit symmetric to at_least.
{expression: "<expr>"}	Sugar for {aggregator: {kind: expression, expr: "<expr>"}}. Evaluates the inline expression against the branches map; truthy means succeeded.
{aggregator: {kind: ...}}	General form. Holds an executor-shaped config (kind: required). kind: expression evaluates inline; every other kind (script, mcp, rest, workflow, cli) dispatches through the executor registry, receiving {branches[], ok_count, ...} as input and returning the verdict.

Failure modes

on_branch_failure	Behavior
bail (default)	First branch failure cancels in-flight siblings; whole executor fails fast. Best for strict matrices where any failure invalidates the run.
continue	Every branch runs regardless of failures. Aggregate verdict is computed per join condition. Best for graded reports (audits, scans, partial-success-acceptable research).

Caution

With continue and non-idempotent branches, partial side effects CAN commit before the aggregate verdict is failed. v1 does not provide compensating transactions — if your branches have side effects, prefer bail OR design branches to be idempotent. The summary.failed_branches in the output lists which branches failed so you can compose follow-up cleanup workflows.

Output shape

{
  "branches": [
    { "ok": true,  "index": 0, "output": { ... } },
    { "ok": false, "index": 1, "error": { "code": "timeout", "message": "..." } },
    { "ok": true,  "index": 2, "output": { ... } }
  ],
  "summary": {
    "n":                      3,
    "ok_count":               2,
    "failed_count":           1,
    "cancelled_count":        0,
    "durationMs":             4321,
    "first_failure_index":    1,
    "max_in_flight_observed": 2,
    "join":                   "all",
    "verdict":                "failed"
  }
}

Workflow output: mappings can use the new [*] array-projection syntax to pluck per-branch fields:

output:
  allPassed: "$.output.summary.verdict"     # "succeeded" | "failed" | "threshold_not_met"
  verdicts:  "$.output.branches[*].ok"      # [true, false, true]
  scores:    "$.output.branches[*].output.score"  # [90, null, 85]

[*] on a non-array path resolves to null (consistent with the existing unresolved-path contract). Multi-level wildcards work ($.context.groups[*].items[*].name returns nested arrays).

Audit trail

Each branch emits its own events linked to the parent transition by correlation_id:

Event	When
parallel.branch.started	Branch begins; payload: branch_index, branch_executor_kind
parallel.branch.completed	Branch returns Ok; payload: branch_index, durationMs
parallel.branch.failed	Branch returns Err; payload: branch_index, durationMs, error
parallel.branch.cancelled	Branch cancelled (any-join success or bail-failure)
parallel.fanout.completed	Aggregate done, before parent transition record; payload: summary
parallel.fanout.empty	Dynamic for_each resolved to []

Combined with the parent transition’s seq and a per-branch branch_seq counter, the three-tuple (seq, branch_index, branch_seq) is the canonical event ordering. Replay tools reconstruct intra-branch order without depending on timestamps.

The “fan-out lives inside one transition” rule

Load-bearing invariant (SPEC §24.5): fan-out happens inside one executor invocation. Branches NEVER touch the WorkflowStore directly. The parent executor returns one ExecuteResult; the runtime does exactly one save_if_version post-aggregation. The transition record is written once, the workflow version bumps once.

This preserves every existing workflow invariant — optimistic locking still works, deterministic chaining still works, audit ordering still works. Multi-active-state workflow execution is explicitly OUT of scope; the constraint keeps the system coherent.

DOS poka-yoke

branches.len() >= 10 without explicit max_concurrency rejects at config-load with INVALID_PARALLEL_CONFIG. The threshold (10) is a guess — operators with strong opinions set max_concurrency explicitly even for smaller fan-outs. The aggregate event’s summary.max_in_flight_observed lets you see whether you ever hit the cap.

MCP transport bottleneck

parallel of kind: mcp branches against the same MCP connection is bounded by that connection’s concurrency. Typical MCP servers serialize on a single stdio connection. If you want true MCP parallelism:

• Configure N separate connections to the SAME MCP server (each branch uses a different connection name), OR
• Use an MCP server that supports concurrent in-flight requests on one connection.

Per-branch durationMs reveals serialization — if every branch’s start time staggers by roughly its predecessor’s duration, the transport is the bottleneck.

Worked example: parallel verification

scripts:
  test.cargo.workspace:    { verb: test,   lifecycle: stable, body: "cargo test --workspace" }
  lint.rust.clippy-strict: { verb: lint,   lifecycle: stable, body: "cargo clippy --workspace --all-targets -- -D warnings" }
  format.rust.check:       { verb: format, lifecycle: stable, body: "cargo fmt --all -- --check" }

workflows:
  pre_merge_gate:
    initialState: verifying
    blackboard:
      allPassed:    { type: boolean }
      failed_count: { type: integer }
    states:
      verifying:
        transitions:
          run:
            target: gated
            actor: deterministic
            executor:
              kind: parallel
              branches:
                - { kind: script, subject: test.cargo.workspace }
                - { kind: script, subject: lint.rust.clippy-strict }
                - { kind: script, subject: format.rust.check }
              join: all
              max_concurrency: 3
              on_branch_failure: continue   # graded report: run all, then decide
            output:
              allPassed:    "$.output.summary.verdict"
              failed_count: "$.output.summary.failed_count"
      gated:
        transitions:
          merge:
            target: done
            guards:
              - { kind: expr, expr: "$.context.allPassed == 'succeeded'" }
          reject:
            target: rejected
            guards:
              - { kind: expr, expr: "$.context.allPassed != 'succeeded'" }
      done:     { terminal: true }
      rejected: { terminal: true }

Three independent checks run concurrently. With on_branch_failure: continue and join: all, every check completes (so you get the full failure report) but the verdict fails if any check failed. The downstream guard reads summary.verdict to gate the merge.

See also

• SPEC §24 — full normative spec
• Skills, workflows, and cognitive architectures — where parallel fits in the layered model
• Script verbs reference — the executor kinds you’ll fan-out