Control concurrency¶
Can this run use less simultaneous work? Add --concurrency 2 to limit it to two scheduled nodes. Leave the setting out when the normal configured limits are appropriate.
Limit one run¶
Continue the Landing/Operations project from selective runs, keeping workspace-development.yml. Cap its Landing Load at two scheduled nodes:
The command returned:
Load
Read catalogue 2.5s
Workspace Parcel Development
Catalogue Warehouse/Catalogue
Targets
Lakehouse/Landing
Build run graph 0.0s
Wait for Spark session 35.9s
→ Load Lakehouse/Landing/Files/Parcel.Events
→ Load Lakehouse/Landing/Files/Parcel.StatusFiles
✓ Load Lakehouse/Landing/Files/Parcel.Events (read 1, +0 ~0 -0 !0) 3.0s
→ Load Lakehouse/Landing/Tables/Parcel.Event
✓ Load Lakehouse/Landing/Files/Parcel.StatusFiles (read 1, +0 ~0 -0 !0) 3.1s
→ Load Lakehouse/Landing/Tables/Parcel.LiveStatus
✓ Load Lakehouse/Landing/Tables/Parcel.LiveStatus (read 0, +0 ~0 -0 !0) 0.2s
✓ Load Lakehouse/Landing/Tables/Parcel.Event (read 3, +0 ~0 -0 !0) 14.4s
→ Load Lakehouse/Landing/Tables/Parcel.CurrentStatus
✓ Load Lakehouse/Landing/Tables/Parcel.CurrentStatus (read 2, +0 ~0 -0 !0) 7.1s
Execute 22.8s
Record what the run did 0.0s
✓ Load 1m09s
load succeeded: Lakehouse/Landing
Load summary
5 succeeded
0 failed
0 blocked
Rows
read 7
inserted 0
updated 0
deleted 0
rejected 0
Workflow: 64a5966c5760456093c5c15975f95fe3
The command returned:
{'requested': ['Lakehouse/Landing'],
'status': 'succeeded',
'dry_run': False,
'fault_tolerant': False,
'reload': False,
'ignore_stability_threshold': False,
'workspace': 'Parcel Development',
'workflow_id': '[run-id]',
'started_at': '2026-10-17T07:28:20.222295+00:00',
'finished_at': '2026-10-17T07:29:38.974467+00:00',
'order': ['load:Lakehouse/Landing/Files/Parcel.Events',
'load:Lakehouse/Landing/Files/Parcel.StatusFiles',
'load:Lakehouse/Landing/Tables/Parcel.Event',
'load:Lakehouse/Landing/Tables/Parcel.CurrentStatus',
'load:Lakehouse/Landing/Tables/Parcel.LiveStatus'],
'edges': [['load:Lakehouse/Landing/Files/Parcel.Events',
'load:Lakehouse/Landing/Tables/Parcel.Event'],
['load:Lakehouse/Landing/Files/Parcel.StatusFiles',
'load:Lakehouse/Landing/Tables/Parcel.LiveStatus'],
['load:Lakehouse/Landing/Tables/Parcel.Event',
'load:Lakehouse/Landing/Tables/Parcel.CurrentStatus']],
'nodes': [{'node_id': 'load:Lakehouse/Landing/Files/Parcel.Events',
'logical_id': 'Lakehouse/Landing/Files/Parcel.Events',
'physical_target': 'Lakehouse/Landing',
'primitive_kind': 'python_folder',
'dispatch_location': 'Lakehouse/Landing/_/Load/Files/Parcel__Events.py',
'status': 'succeeded',
'executed': True,
'started_at': '2026-10-17T07:29:12.358664+00:00',
'finished_at': '2026-10-17T07:29:15.792496+00:00',
'rows': {'succeeded': True,
'rows_read': 1,
'rows_inserted': 0,
'rows_updated': 0,
'rows_deleted': 0,
'rows_rejected': 0,
'error_message': None,
'bookmark_datetime': '2026-10-17T07:29:15.290041+00:00',
'is_static_skip': False,
'is_refusal': False},
'messages': []},
{'node_id': 'load:Lakehouse/Landing/Files/Parcel.StatusFiles',
'logical_id': 'Lakehouse/Landing/Files/Parcel.StatusFiles',
'physical_target': 'Lakehouse/Landing',
'primitive_kind': 'python_folder',
'dispatch_location': 'Lakehouse/Landing/_/Load/Files/Parcel__StatusFiles.py',
'status': 'succeeded',
'executed': True,
'started_at': '2026-10-17T07:29:12.358925+00:00',
'finished_at': '2026-10-17T07:29:15.956293+00:00',
'rows': {'succeeded': True,
'rows_read': 1,
'rows_inserted': 0,
'rows_updated': 0,
'rows_deleted': 0,
'rows_rejected': 0,
'error_message': None,
'bookmark_datetime': '2026-10-17T07:29:15.298153+00:00',
'is_static_skip': False,
'is_refusal': False},
'messages': []},
{'node_id': 'load:Lakehouse/Landing/Tables/Parcel.Event',
'logical_id': 'Lakehouse/Landing/Tables/Parcel.Event',
'physical_target': 'Lakehouse/Landing',
'primitive_kind': 'python_table',
'dispatch_location': 'Lakehouse/Landing/_/Load/Tables/Parcel__Event.py',
'status': 'succeeded',
'executed': True,
'started_at': '2026-10-17T07:29:15.796582+00:00',
'finished_at': '2026-10-17T07:29:31.685316+00:00',
'rows': {'succeeded': True,
'rows_read': 3,
'rows_inserted': 0,
'rows_updated': 0,
'rows_deleted': 0,
'rows_rejected': 0,
'error_message': None,
'bookmark_datetime': '2026-10-17T07:29:15.888932+00:00',
'is_static_skip': False,
'is_refusal': False},
'messages': []},
{'node_id': 'load:Lakehouse/Landing/Tables/Parcel.CurrentStatus',
'logical_id': 'Lakehouse/Landing/Tables/Parcel.CurrentStatus',
'physical_target': 'Lakehouse/Landing',
'primitive_kind': 'python_table',
'dispatch_location': 'Lakehouse/Landing/_/Load/Tables/Parcel__CurrentStatus.py',
'status': 'succeeded',
'executed': True,
'started_at': '2026-10-17T07:29:31.689523+00:00',
'finished_at': '2026-10-17T07:29:38.967878+00:00',
'rows': {'succeeded': True,
'rows_read': 2,
'rows_inserted': 0,
'rows_updated': 0,
'rows_deleted': 0,
'rows_rejected': 0,
'error_message': None,
'bookmark_datetime': '2026-10-17T07:29:31.740421+00:00',
'is_static_skip': False,
'is_refusal': False},
'messages': []},
{'node_id': 'load:Lakehouse/Landing/Tables/Parcel.LiveStatus',
'logical_id': 'Lakehouse/Landing/Tables/Parcel.LiveStatus',
'physical_target': 'Lakehouse/Landing',
'primitive_kind': 'python_table',
'dispatch_location': 'Lakehouse/Landing/_/Load/Tables/Parcel__LiveStatus.py',
'status': 'succeeded',
'executed': True,
'started_at': '2026-10-17T07:29:15.956968+00:00',
'finished_at': '2026-10-17T07:29:16.098389+00:00',
'rows': {'succeeded': True,
'rows_read': 0,
'rows_inserted': 0,
'rows_updated': 0,
'rows_deleted': 0,
'rows_rejected': 0,
'error_message': None,
'bookmark_datetime': '2026-10-17T07:29:16.006092+00:00',
'is_static_skip': False,
'is_refusal': False},
'messages': []}],
'messages': [{'severity': 'info',
'code': 'dependency_external',
'message': 'Lakehouse/Landing/Files/Parcel.StatusFiles reads physical object '
'shortcuts.Parcel__StatusSource directly. Weaver will not use this '
'reference to order loads.',
'detail': None,
'source': 'load_plan'}]}
The dependency chain still ran in order. A cap of two permits two ready nodes; it does not make this chain parallel.
Build and Test accept the same positive-integer option:
weaver build --item Lakehouse/Landing --concurrency 2 --workspace-config workspace-development.yml
weaver test Lakehouse/Landing --concurrency 2 --workspace-config workspace-development.yml
The command returned:
Test
Read catalogue 2.9s
Workspace Parcel Development
Catalogue Warehouse/Catalogue
Targets
Lakehouse/Landing
Build run graph 0.0s
Wait for Spark session 38.9s
→ Test Lakehouse/Landing/Parcel.CurrentStatusMatches
→ Test Lakehouse/Landing/Parcel.EventStatusIsKnown
✓ Test Lakehouse/Landing/Parcel.EventStatusIsKnown 11.2s
✓ Test Lakehouse/Landing/Parcel.CurrentStatusMatches 13.8s
Execute 12.7s
Record what the run did 0.0s
✓ Test 1m02s
test passed
Test summary
2 passed
0 failed
Workflow: cee836b15e044193b9d0f2ed606bb84b
The command returned:
{'status': 'passed',
'nodes': [{'logical_id': 'Lakehouse/Landing/Parcel.CurrentStatusMatches',
'kind': 'test',
'physical_target': 'Lakehouse/Landing',
'primitive_kind': 'python_validation',
'dispatch_location': '',
'status': 'passed',
'executed': True,
'messages': [],
'started_at': '2026-10-17T07:33:45.072515+00:00',
'finished_at': '2026-10-17T07:34:06.118208+00:00',
'missing_count': 0,
'unexpected_count': 0,
'failure_count': 0,
'error_message': None},
{'logical_id': 'Lakehouse/Landing/Parcel.EventStatusIsKnown',
'kind': 'assumption',
'physical_target': 'Lakehouse/Landing',
'primitive_kind': 'python_validation',
'dispatch_location': '',
'status': 'passed',
'executed': True,
'messages': [],
'started_at': '2026-10-17T07:33:45.072770+00:00',
'finished_at': '2026-10-17T07:34:03.468832+00:00',
'violation_count': 0,
'error_message': None}],
'totals': {'planned': 2,
'executed': 2,
'passed': 2,
'failed': 0,
'invalid': 0,
'missing_count': 0,
'unexpected_count': 0,
'violation_count': 0},
'workflow_id': '[run-id]',
'started_at': '2026-10-17T07:32:35.478510+00:00',
'finished_at': '2026-10-17T07:34:06.527737+00:00'}
Here both validations started before either finished. The cap applies across the invocation, not separately to each item. With two Warehouses and a total cap of two, there are still at most two simultaneous scheduled nodes, not four. Dependencies and resource ceilings can keep actual concurrency lower.
The option does not change workspace configuration, retarget a bundle or rewrite its plan. Omission keeps the configured/default behaviour. It does not set Spark capacity, executor count or cluster size, and does not limit every nested HTTP request.
Use the same cap in Python¶
import weaver
result = weaver.build(
'.',
items='Lakehouse/Landing=Lakehouse/Landing',
workspace='Parcel Development',
catalogue='Warehouse/Catalogue',
concurrency=2,
)
if not result.succeeded:
raise RuntimeError(result.errors)
result.report()
In a Fabric notebook, pass the supported project source location and let Weaver discover the notebook context, or reuse an established Session:
import weaver
report = weaver.load(
'Lakehouse/Landing',
workspace='Parcel Development',
catalogue='Warehouse/Catalogue',
concurrency=2,
)
if not report.succeeded:
raise RuntimeError('Inspect the failed nodes in the Load report')
report.report()
Workspace selection and runtime prerequisites still apply. The invocation cap does not provision compute or make otherwise unavailable work runnable.
Source/file-mode Tests are serial by default. Explicit concurrency lets them use the run scheduler, subject to their resource limits. A cap limits scheduled work, not the number of DAX queries a single validation contains.
Tune persistent resource limits only when needed¶
Build limits and run limits are separate. This is a reference illustration of the defaults, not configuration every project must paste:
execution:
build:
warehouse_concurrency: 2
spark_concurrency: 4
onelake_concurrency: 8
shortcut_concurrency: 2
run:
spark_concurrency: 4
warehouse_concurrency: 4
Warehouse limits apply per Warehouse. Spark run lanes can use separate Spark sessions within one Spark application; they are not a newly provisioned cluster for each table. Build's other ceilings govern storage and shortcut work. An invocation cap remains the overall ceiling on top of those limits.
There is no execution.test, target-level parallel_workers setting or --spark-concurrency CLI flag. Use workspace configuration for accepted keys and precedence.
Measure before increasing a limit¶
Use enough independent work for the cap to matter. A single dependency chain will not become parallel merely because the number increases.
Compare a fixed workload, record waits, outcomes and resource usage, and change one setting at a time. More concurrency can increase contention or throttling. The historical F64 benchmark used 24 Spark lanes and 12 per Warehouse; these are not defaults, small-capacity recommendations or hard maxima.