Skip to content

Recover failed work

Recovery is a new operation against the state that remains. Correct the cause, choose the smallest operation that expresses the intended result, and verify the estate afterward. Build, Load, Test and workflows do not roll back completed work when a later step fails.

Use the same project and configuration as the run that failed. The examples use the Parcel estate and workspace-development.yml; if you only have the Warehouse project, use workspace-config.yml instead. Named examples use Parcel.StatusEvent, the Operations Table; Views have no Load of their own.

Start with Health. It shows source data, model refreshes and validations together:

weaver health --workspace-config workspace-development.yml

Choose the recovery

Intended result Operation
Reconcile corrected project source Check, then rerun Build
Run failed, pending or stale data work Rerun Load, normally with --stale for non-Green work
Reconstruct a selected Table from empty Named Load with --reload
Re-evaluate installed validations Rerun Test
Replace development with a fresh source baseline Re-mirror after reviewing the destination plan
Leave selected physical targets or the estate empty Wipe after a dry run

Reconcile corrected source

After correcting source or a Fabric condition:

weaver check
weaver build --workspace-config workspace-development.yml
weaver health --workspace-config workspace-development.yml

Build reads the project and installed estate again and reconciles its selection. It does not resume a failed Build. Completed physical actions from the earlier attempt may still be present, so inspect the new report and Health result.

Rerun failed or stale Load work

After correcting the data, runtime or permission condition, preview and run non-Green loadable work:

weaver load \
  --stale \
  --dry-run \
  --workspace-config workspace-development.yml

weaver load \
  --stale \
  --workspace-config workspace-development.yml

This uses the definitions last installed by Build. Build first if the correction changed project source. Green upstream work is not added merely because a selected descendant reads it.

After correcting the cause, preview a seed and its downstream work:

weaver load Warehouse/Operations --name 'Parcel[.]StatusEvent' --descendants --dry-run --workspace-config workspace-development.yml

Remove only --dry-run if the selection is correct. This is a whole-name regex, not an exact-name shortcut. Expansion stays inside named items, and a descendant's omitted sibling prerequisite is not automatically refreshed. Run part of an estate gives the deterministic example and both directions. This command does not request reconstruction.

Reconstruct a Table

Reload empties each reached selected Table and resets its bookmark/progress before loading it. Use it only when reconstruction from empty is intended:

weaver load Warehouse/Operations --name 'Parcel[.]StatusEvent' --reload --dry-run --workspace-config workspace-development.yml

The command returned:

Terminal output exit 0
Load (dry run)

  Read catalogue                                                                                           2.7s
Workspace  Parcel Development
Catalogue  Warehouse/Catalogue
Targets
  Warehouse/Operations

  Build run graph                                                                                          0.0s
  Execute                                                                                                  0.0s
✓ Load (dry run)                                                                                           2.7s

plan (reload) succeeded: Warehouse/Operations

    validated                load:Warehouse/Operations/Parcel.StatusEvent

Plan
    1 selected

from pathlib import Path

import weaver

project = Path("builtin") / "project"
load_result = weaver.load(
    items=['Warehouse/Operations'],
    names=['Parcel[.]StatusEvent'],
    source=project,
    workspace='Parcel Development',
    catalogue='Warehouse/Catalogue',
    dry_run=True,
    reload=True,
)
load_result.report()

The command returned:

Notebook output
{'requested': ['Warehouse/Operations'],
 'status': 'succeeded',
 'dry_run': True,
 'fault_tolerant': False,
 'reload': True,
 'ignore_stability_threshold': False,
 'workspace': 'Parcel Development',
 'workflow_id': None,
 'started_at': '2026-10-17T07:24:28.071075+00:00',
 'finished_at': '2026-10-17T07:24:29.624782+00:00',
 'order': ['load:Warehouse/Operations/Parcel.StatusEvent'],
 'edges': [],
 'nodes': [{'node_id': 'load:Warehouse/Operations/Parcel.StatusEvent',
            'logical_id': 'Warehouse/Operations/Parcel.StatusEvent',
            'physical_target': 'Warehouse/Operations',
            'primitive_kind': 'warehouse_procedure',
            'dispatch_location': 'Warehouse/Operations/[_].[Load Parcel.StatusEvent]',
            'status': 'validated',
            'executed': False,
            'started_at': None,
            'finished_at': None,
            'rows': None,
            'messages': []}],
 'messages': []}

The preview names exactly the Tables that will be emptied. Then run it:

weaver load Warehouse/Operations --name 'Parcel[.]StatusEvent' --reload --workspace-config workspace-development.yml

The command returned:

Terminal output exit 0
Load

  Read catalogue                                                                                           2.7s
Workspace  Parcel Development
Catalogue  Warehouse/Catalogue
Targets
  Warehouse/Operations

  Build run graph                                                                                          0.0s
→   Load Warehouse/Operations/Parcel.StatusEvent
✓   Load Warehouse/Operations/Parcel.StatusEvent  (read 3, +3 ~0 -0 !0)                                    2.5s
  Execute                                                                                                  3.2s
  Record what the run did                                                                                  0.4s
✓ Load                                                                                                     6.3s

load (reload) succeeded: Warehouse/Operations

Load summary
    1 succeeded
    0 failed
    0 blocked
  Rows
    read                   3
    inserted               3
    updated                0
    deleted                0
    rejected               0

  Workflow: e0d44df3ceb54cb48ac6f0446ab371f7

from pathlib import Path

import weaver

project = Path("builtin") / "project"
load_result = weaver.load(
    items=['Warehouse/Operations'],
    names=['Parcel[.]StatusEvent'],
    source=project,
    workspace='Parcel Development',
    catalogue='Warehouse/Catalogue',
    reload=True,
)
load_result.report()

The command returned:

Notebook output
{'requested': ['Warehouse/Operations'],
 'status': 'succeeded',
 'dry_run': False,
 'fault_tolerant': False,
 'reload': True,
 'ignore_stability_threshold': False,
 'workspace': 'Parcel Development',
 'workflow_id': '[run-id]',
 'started_at': '2026-10-17T07:38:36.570230+00:00',
 'finished_at': '2026-10-17T07:38:41.369991+00:00',
 'order': ['load:Warehouse/Operations/Parcel.StatusEvent'],
 'edges': [],
 'nodes': [{'node_id': 'load:Warehouse/Operations/Parcel.StatusEvent',
            'logical_id': 'Warehouse/Operations/Parcel.StatusEvent',
            'physical_target': 'Warehouse/Operations',
            'primitive_kind': 'warehouse_procedure',
            'dispatch_location': 'Warehouse/Operations/[_].[Load Parcel.StatusEvent]',
            'status': 'succeeded',
            'executed': True,
            'started_at': '2026-10-17T07:38:39.297089+00:00',
            'finished_at': '2026-10-17T07:38:41.365035+00:00',
            'rows': {'succeeded': True,
                     'rows_read': 3,
                     'rows_inserted': 3,
                     'rows_updated': 0,
                     'rows_deleted': 0,
                     'rows_rejected': 0,
                     'error_message': None,
                     'bookmark_datetime': '2026-10-17T07:38:39.569372+00:00',
                     'is_static_skip': False,
                     'is_refusal': False},
            'messages': []}],
 'messages': []}

The three events were inserted into an empty Table (+3), each with a new Row insert datetime, and the Table's bookmark restarted from that Load.

Plain named Reload follows its matching selection without expansion. Ancestors/descendants can deliberately enlarge the reached set and therefore enlarge the destructive scope. Folders cannot be reloaded; stale and reload cannot be combined. If execution fails after the reset or emptying step, the previous Table contents are not restored automatically.

Continue partial work only when its output is usable

--fault-tolerant lets independent work continue and can publish valid rows/files alongside retained rejects. It does not turn failures into success. A descendant may run after an upstream execution failure settles and read whatever state that failure left; invalid or unresolved upstream work still blocks descendants. Review node outcomes before using partial output. Rejecting loads do not establish a clean bookmark boundary.

weaver load Warehouse/Operations --stale --fault-tolerant --workspace-config workspace-development.yml

A delete/update stability-threshold breach still refuses the change without modifying the target. Fault tolerance changes how that refusal is returned, not whether it is applied. If the large change is intended, review it and explicitly waive those thresholds for this invocation:

weaver load Warehouse/Operations --name 'Parcel[.]StatusEvent' --ignore-stability-threshold --workspace-config workspace-development.yml

This flag does not waive null or unique-key checks, change definitions or reset bookmarks. An incremental merge violating an existing unique key is fatal even with fault tolerance. See rejected data and thresholds and Load failure policy before choosing continuation.

Rerun validation

Correct the data, runtime or validation condition. Build edits before catalogue-mode Test; source/file-mode Test can use the edited validation without recording installed history. For the installed route:

weaver test --workspace-config workspace-development.yml
weaver health --workspace-config workspace-development.yml

Test has no rollback step. A failed validation remains the current finding until a later run records another outcome.

Recover a model or Report

Keep this same project. If you added the native and source-backed models, use their existing bindings in workspace-development.yml; do not replace the independent pipelines or native model.

A semantic-model refresh is Load work. Fix its source, connection or refresh condition, then rerun Load. A DAX finding is validation work: correct the model, data or query, then rerun Test. Their installed results appear in the same Health report as Lakehouse and Warehouse work.

A definition or binding change needs Build instead. Select the model and linked Report through their project, after any source-data Build:

weaver build --item PowerBI/Parcel --workspace-config workspace-development.yml
weaver load SemanticModel/Parcel --workspace-config workspace-development.yml
weaver test SemanticModel/Parcel --workspace-config workspace-development.yml
weaver health --workspace-config workspace-development.yml

These commands assume the Parcel model and Report are in the project and configured. Reports have no separate Load or Test; inspect the Report’s binding and rendering after recovering its model. See Deploy models and Reports and Connect a model to source data.

Refresh a mirrored baseline

Re-mirror empties and reconstructs the destination catalogue and every selected destination target, replacing local materialisations in that boundary. The source items remain untouched. Use it only to replace a mirrored development estate with a fresh source baseline. Run it with the mirror configuration from Develop using Mirror, not the estate's own configuration:

weaver mirror --workspace-config workspace-mirror.yml

The command returned:

Terminal output exit 0
Mirror

Workspace              Parcel Development
Source catalogue       Warehouse/Catalogue
Destination catalogue  Warehouse/CatalogueDev
Targets
  Warehouse/Operations → Warehouse/OperationsDev

Empty these targets? This cannot be undone. [y/N] y

Mirror

  Planning the mirror                                                                                      1.5s
→   Wipe Warehouse/CatalogueDev · 1 action
→   Wipe Warehouse/OperationsDev · 1 action
✓   Wipe Warehouse/CatalogueDev · 1 action                                                                 1.2s
→   Warehouse/CatalogueDev · Catalogue: creating schemas · 1 action
✓   Warehouse/CatalogueDev · Catalogue: creating schemas · 1 action                                        0.0s
→   Warehouse/CatalogueDev · Catalogue: building objects · 24 actions
✓   Wipe Warehouse/OperationsDev · 1 action                                                                1.3s
→   Reconstruct Warehouse/OperationsDev · 2 actions
→   Recreate views over other items in Warehouse/OperationsDev · 1 action
✓   Reconstruct Warehouse/OperationsDev · 2 actions                                                        0.2s
✓   Recreate views over other items in Warehouse/OperationsDev · 1 action                                  0.5s
✓   Warehouse/CatalogueDev · Catalogue: building objects · 24 actions                                      5.8s
→   Warehouse/CatalogueDev · Catalogue: updating catalogue definitions · 1 action
✓   Warehouse/CatalogueDev · Catalogue: updating catalogue definitions · 1 action                          4.4s
→   Warehouse/CatalogueDev · Catalogue: finalising catalogue · 1 action
✓   Warehouse/CatalogueDev · Catalogue: finalising catalogue · 1 action                                    0.7s
→   Recreate catalogue views in Warehouse/OperationsDev · 1 action
✓   Recreate catalogue views in Warehouse/OperationsDev · 1 action                                         0.1s
→   Copy procedures and functions into Warehouse/OperationsDev · 1 action
✓   Copy procedures and functions into Warehouse/OperationsDev · 1 action                                  0.3s
→   Warehouse/CatalogueDev · Forking the catalogue and binding items · 1 action
✓   Warehouse/CatalogueDev · Forking the catalogue and binding items · 1 action                            7.9s
  Mirroring                                                                                               20.5s
✓ Mirror                                                                                                  23.4s

Mirror complete
  Catalogue  Warehouse/Catalogue → Warehouse/CatalogueDev
  1 item mirrored
  2 destinations emptied

1
2
3
4
5
6
7
8
9
# Review this plan before running the next cell.
import weaver
mirror_plan = weaver.plan_mirror(
    workspace='Parcel Development',
    catalogue='Warehouse/CatalogueDev',
    mirror='Warehouse/Catalogue',
)
mirror_plan = weaver.check_mirror(mirror_plan)
print(mirror_plan.describe())
1
2
3
import weaver
mirror_result = weaver.mirror(plan=mirror_plan)
mirror_result.report()

The command returned:

Notebook output
{'workspace': 'Parcel Development',
 'source_catalogue': 'Warehouse/Catalogue',
 'destination_catalogue': 'Warehouse/CatalogueDev',
 'wiped': ['Warehouse/CatalogueDev',
           'Warehouse/OperationsDev'],
 'copied': {'SchemaDictionary': 10,
            'FolderDictionary': 3,
            'TableDictionary': 28,
            'ColumnDictionary': 165,
            'KeyDictionary': 26,
            'ForeignKeyDictionary': 0,
            'TestDictionary': 6,
            'SemanticModelTest': 2,
            'Dependency': 12,
            'Shortcut': 20,
            'SemanticModel': 2,
            'SemanticModelTable': 7,
            'SemanticModelMeasure': 5,
            'SemanticModelRelationship': 2,
            'SemanticModelColumn': 19,
            'Installation': 6,
            'Registry': 74,
            'Bookmark': 7,
            'LoadStatus': 10,
            'TestStatus': 6},
 'uncopied': ['Log', 'LoadStatistic'],
 'items': ['Warehouse/Operations'],
 'mirrored': {'Warehouse/Operations': {'source': 'Warehouse/Operations',
                                       'target': 'Warehouse/OperationsDev',
                                       'relations': 2,
                                       'pointers': 0,
                                       'programmables': 5}},
 'status': 'succeeded'}

Mirror shows the source catalogue, destination catalogue and target bindings before it asks. Confirm only if that boundary is correct. It has no dry-run mode; declining leaves the destinations unchanged.

Any object that had become local in OperationsDev is borrowed again, so its development-only changes are gone. Run Health against the mirror configuration, then Build any project changes that should become local again:

weaver health --item Warehouse/Operations --workspace-config workspace-mirror.yml

Wipe only when the desired state is empty

Wipe is not rollback and is not the normal response to a failed Build or Load. It empties selected Lakehouse/Warehouse contents, including unmanaged content, and normally also empties the resolved catalogue even when only one Warehouse is named. No Fabric item is deleted. Use it only when that complete resolved boundary should end empty; Reports cannot be wiped.

To empty this physical target while retaining the catalogue and removing that target’s claims, request --unbind explicitly and inspect the full settled plan:

weaver wipe Warehouse/Operations --unbind --dry-run --workspace-config workspace-development.yml

The command returned:

Terminal output exit 0
Wipe on Parcel Development

Empty
  Warehouse/Operations

Catalogue
  Warehouse/Catalogue   preserved; claims for Warehouse/Operations unbound

Nothing was changed.

import weaver
wipe_result = weaver.wipe(
    targets=['Warehouse/Operations'],
    workspace='Parcel Development',
    catalogue='Warehouse/Catalogue   preserved; claims for Warehouse/Operations unbound',
    unbind=True,
    preserve_data_source=False,
    dry_run=True,
)
wipe_result.report()

The command returned:

Notebook output
{'workspace': 'Parcel Development',
 'plan': {'workspace': 'Parcel Development',
          'targets': [{'target': 'Warehouse/Operations', 'catalogue': False}],
          'catalogue': 'Warehouse/Catalogue',
          'catalogue_action': 'unbind',
          'unbound': ['Warehouse/Operations'],
          'preserve_data_source': False,
          'left': []},
 'items': [{'target': 'Warehouse/Operations',
            'outcome': 'emptied',
            'catalogue': False,
            'unbound': False,
            'counts': {}},
           {'target': 'Warehouse/Catalogue',
            'outcome': 'preserved',
            'catalogue': True,
            'unbound': True,
            'counts': {}}],
 'unbound': None,
 'dry_run': True}

If the plan is the empty state you want, run it without --dry-run and confirm:

weaver wipe Warehouse/Operations --unbind --workspace-config workspace-development.yml

The command returned:

Terminal output exit 0
Wipe on Parcel Development

Empty
  Warehouse/Operations

Catalogue
  Warehouse/Catalogue   preserved; claims for Warehouse/Operations unbound

Empty 1 item(s)? This cannot be undone. [y/N] y

Wipe

→   Wipe Warehouse/Operations · 1 action
✓   Wipe Warehouse/Operations · 1 action                                                                   1.6s
→   Unbind catalogue claims in Warehouse/Catalogue · 1 action
✓   Unbind catalogue claims in Warehouse/Catalogue · 1 action                                              6.1s
  Emptying Warehouse/Operations                                                                            7.8s
✓ Wipe                                                                                                     9.9s

Wipe complete

  Warehouse/Operations        emptied
  Warehouse/Catalogue         catalogue preserved, claims unbound

# Review this plan before running the next cell.
import weaver
wipe_plan = weaver.plan_wipe(
    targets=['Warehouse/Operations'],
    workspace='Parcel Development',
    catalogue='Warehouse/Catalogue   preserved; claims for Warehouse/Operations unbound',
    unbind=True,
    preserve_data_source=False,
)
wipe_plan.report()
1
2
3
import weaver
wipe_result = weaver.wipe(plan=wipe_plan)
wipe_result.report()

The command returned:

Notebook output
{'workspace': 'Parcel Development',
 'plan': {'workspace': 'Parcel Development',
          'targets': [{'target': 'Warehouse/Operations',
                       'catalogue': False}],
          'catalogue': 'Warehouse/Catalogue',
          'catalogue_action': 'unbind',
          'unbound': ['Warehouse/Operations'],
          'preserve_data_source': False,
          'left': []},
 'items': [{'target': 'Warehouse/Operations',
            'outcome': 'emptied',
            'catalogue': False,
            'unbound': False,
            'counts': {}},
           {'target': 'Warehouse/Catalogue',
            'outcome': 'preserved',
            'catalogue': True,
            'unbound': True,
            'counts': {}}],
 'unbound': {'targets': ['Warehouse/Operations'],
             'logical_items': ['Warehouse/Operations'],
             'statements': 17},
 'dry_run': False}

Operations is now an empty Warehouse with no Weaver claims; the catalogue still holds Landing and any other items. To bring it back, Build and Load it as on first deployment:

weaver build --item Warehouse/Operations --workspace-config workspace-development.yml
weaver load Lakehouse/Landing Warehouse/Operations --workspace-config workspace-development.yml

The first Build into a just-emptied Warehouse can fail with a Fabric Snapshot isolation transaction failed error while the service settles its metadata. Rerun the Build; it reconciles from the state that remains.

Wipe target arguments are physical Fabric items, not logical Weaver item identities. Naming no target selects the physical estate recorded by the catalogue and retains the normal catalogue-removal default; preview that full boundary before authorising it. Semantic-model-only named Wipe instead resets the selected model definitions and retains the catalogue while removing their claims; it does not empty their sources or shared connections. Wipe is not transactional: completed or independent mutations remain after failure. Inspect remaining state before retrying uncertain destructive work.

Exact Wipe catalogue handling, unbinding and authorisation rules are in Wipe operation behaviour.

Remove a claim without emptying the item

Unbind removes the catalogue's ownership claims for the resolved physical items; it does not delete those physical items or contents. It is not Wipe, rollback or a way to repair rows by hand. Review the exact scope and confirmation in the Unbind reference.

Verify the result

After any recovery:

  1. inspect the operation report and workflow identifier;
  2. rerun Health with its normal physical inventory check;
  3. rerun Test when recovered data changes validation inputs; and
  4. use read-only catalogue history when the command report is not enough.
weaver health --workspace-config workspace-development.yml
weaver test --workspace-config workspace-development.yml

Health establishes Weaver's installed and operational state. Tests and domain queries establish whether the recovered data meets its expectations. See Fault tolerance for partial-state reasoning and Operation behaviour for exact rerun and destructive boundaries.