A first Lakehouse pipeline with Python¶
Turn a carrier CSV export into a Delta Table using ordinary Python. Weaver manages the files and Table, discovers the dependency from your import and records each Load's result.
Create the project¶
You need Weaver installed and a disposable Fabric workspace on running capacity where you can create Warehouses, Lakehouses and Environments. The example uses Parcel Development; substitute your own.
In the Terminal, press Enter to skip the optional Warehouse, choose 1 to continue, and answer n to publishing the Environment; the next section publishes it.
If you already have the parcel project from the Warehouse task, add Landing to it. From inside parcel:
The Terminal wizard reads the existing bindings, so it only asks you to confirm. Choose 1, then answer n to publishing the Environment.
Your Warehouse files stay as they were. Initialise also adds a separately named Warehouse Table that reads Landing; Connect items and dependencies uses it later.
weaver initialise --workspace "Parcel Development" --project-folder . --lakehouse Landing --example Parcel
The command returned:
Set up a Weaver project.
Project
Project folder .
Workspace Parcel Development
Catalogue Catalogue
Environment Weaver
Lakehouse Landing
Warehouse Operations
Example Parcel
Semantic Model None
1. Continue
2. Change an answer
3. Cancel
Choose [1/2/3]: 1
Publishing a Fabric Environment can take about 5 minutes.
Publish the Environment now? [y/N]: n
Setting up the Weaver project
Checking the workspace 0.5s
Reading the Environment 0.0s
Checking the project files 0.2s
Creating the Lakehouse Landing 2.5s
Writing the project files 0.0s
✓ Setting up the Weaver project 3.1s
Catalogue Catalogue already exists
Environment Weaver already exists
Lakehouse Landing created
Warehouse Operations already exists
Environment publication: deferred
Weaver project ready in [local-path]/parcel.
Next:
cd [local-path]/parcel
Environment publication deferred.
Publish it before the first build.
weaver fabric environment publish --path Environment/Weaver.Environment
weaver workflow full
Or run individually:
weaver build
weaver load
weaver test
weaver health
You can set this up again with:
weaver initialise --workspace 'Parcel Development' --project-folder . --catalogue Catalogue --environment Weaver --lakehouse Landing --warehouse Operations --example Parcel --non-interactive
The command returned:
{'project_folder': '[notebook-resources]/project',
'workspace': 'Parcel Development',
'resources': [{'role': 'Lakehouse', 'name': 'Landing', 'status': 'existing', 'action': None}],
'files': ['Lakehouse/Landing/Files/Parcel__Events.py',
'Lakehouse/Landing/Tables/Parcel.CurrentStatus.sql',
'Lakehouse/Landing/Tables/Parcel__Event.py',
'Lakehouse/Landing/assumptions/Parcel.EventStatusIsKnown.sql',
'Lakehouse/Landing/tests/Parcel.CurrentStatusMatches.sql',
'README.md',
'workflow.yml',
'workspace-config.yml'],
'example': {'generated': True, 'name': 'Parcel'},
'example_added': True,
'environment_publication': 'not requested',
'environment_definition': 'imported',
'dry_run': False,
'next_commands': ['weaver workflow full',
'weaver build',
'weaver load',
'weaver test',
'weaver health']}
Initialise writes the configuration, Environment definition, carrier Folder, Python Event Table, Spark SQL CurrentStatus Table and two validations. Open the generated files as you read; you do not need to copy them.
Inspect the carrier export¶
Open Lakehouse/Landing/Files/Parcel__Events.py:
"""
Folder ID: Parcel.Events
Description: Parcel tracking events as CSV files.
Lineage: A deterministic carrier export generated by this example.
File key: "*.csv"
Incremental: false
"""
from weaver import Folder
EVENTS = """Parcel ID,Event sequence,Status,Depot
P-1001,1,Accepted,North
P-1001,2,In transit,Central
P-1002,1,Delivered,South
"""
class Parcel__Events(Folder):
def read(self):
with self.staging_folder() as staging:
(staging.path / "events.csv").write_text(EVENTS, encoding="utf-8")
return staging
The small CSV gives this first run three known events without an external carrier connection. The Folder stages those bytes; Incremental: false declares the complete set of files.
Inspect the Python Table¶
Open Lakehouse/Landing/Tables/Parcel__Event.py:
"""
Table ID: Parcel.Event
Description: One row per parcel tracking event.
Lineage: The parcel event CSV snapshot managed in this Lakehouse.
Primary key: Parcel ID, Event sequence
Schema:
Parcel ID: string
Event sequence: integer
Status: string
Depot: string
"""
from Files.Parcel__Events import Parcel__Events
from weaver import Table
class Parcel__Event(Table):
def read(self):
source = Parcel__Events(self).spark_path()
return (
self.spark.read.option("header", True)
.csv(source)
.selectExpr(
"`Parcel ID`",
"cast(`Event sequence` as int) as `Event sequence`",
"Status",
"Depot",
)
)
The Table reads the upstream Folder and casts Event sequence to the declared integer type.
Check, build and load¶
The command returned:
Project valid.
Check validates the local declarations and dependency; it does not execute read() or deploy the project.
The Folder and Table run in Fabric Spark, so publish the generated Environment once before the first Build. It adds your installed Weaver version to the Environment's libraries:
Fabric takes about five to seven minutes to publish an Environment. You only repeat this when you upgrade Weaver or change the Environment's libraries; Running Python in Fabric has the details.
Then install the Lakehouse definitions:
The command returned:
Workspace Parcel Development
Catalogue Warehouse/Catalogue
Targets
Lakehouse/Landing
Build
Check target occupancy 0.0s
Resolve semantic data sources 0.0s
Read catalogue 6.3s
Read Warehouse/Catalogue inventory 0.4s
Read Lakehouse/Landing inventory 2.6s
Read target inventories 3.3s
Read semantic sources 0.0s
Prepare bundle 0.1s
Install selection
Lakehouse/Landing
new 15
selected for build 15
Wait for Spark session 36.0s
→ Warehouse/Catalogue · Reconcile runtime state before physical work · 1 action
✓ Warehouse/Catalogue · Reconcile runtime state before physical work · 1 action 2.5s
→ Lakehouse/Landing · Creating schemas · 1 action
→ Lakehouse/Landing · Building objects · 2 actions
→ Lakehouse/Landing · Creating shortcuts · 2 actions
✓ Lakehouse/Landing · Building objects · 2 actions 0.1s
→ Lakehouse/Landing · Installing load and test artefacts · 5 actions
✓ Lakehouse/Landing · Installing load and test artefacts · 5 actions 0.2s
✓ Lakehouse/Landing · Creating schemas · 1 action 3.9s
→ Lakehouse/Landing · Building objects · 1 action
✓ Lakehouse/Landing · Building objects · 1 action 1.1s
→ Lakehouse/Landing · Building objects · 1 action
✓ Lakehouse/Landing · Building objects · 1 action 2.7s
✓ Lakehouse/Landing · Creating shortcuts · 2 actions 17.2s
→ Warehouse/Catalogue · Updating catalogue definitions · 1 action
✓ Warehouse/Catalogue · Updating catalogue definitions · 1 action 5.6s
→ Warehouse/Catalogue · Finalising catalogue · 1 action
✓ Warehouse/Catalogue · Finalising catalogue · 1 action 0.4s
Install 1m09s
✓ Build 1m33s
Installation
15 succeeded
0 failed
0 skipped
Report [local-path]
Bundle f804b536a13adf807f9923fa4b99f4b1ef4c64a9c2ac499787ac0f2c83d76685
The command returned:
{'source': '[local-path]/parcel',
'items': ['Lakehouse/Landing'],
'bundle_id': '[bundle-id]',
'installation': True,
'bundle_path': None,
'status': 'succeeded',
'errors': [],
'actions': {'total': 0, 'succeeded': 0, 'skipped': 0, 'failed': 0},
'report_path': '/tmp/weaver-build-report-0c5c0u7x/install-report.yml'}
Desktop Lakehouse commands start a Fabric Spark session first; the Wait for Spark session line is that start-up, typically around half a minute. Load the Folder and both Tables:
The command returned:
Load
Read catalogue 3.2s
Workspace Parcel Development
Catalogue Warehouse/Catalogue
Targets
Lakehouse/Landing
Build run graph 0.0s
Wait for Spark session 35.9s
→ Load Lakehouse/Landing/Files/Parcel.Events
✓ Load Lakehouse/Landing/Files/Parcel.Events (read 1, +1 ~0 -0 !0) 3.1s
→ Load Lakehouse/Landing/Tables/Parcel.Event
✓ Load Lakehouse/Landing/Tables/Parcel.Event (read 3, +3 ~0 -0 !0) 12.1s
→ Load Lakehouse/Landing/Tables/Parcel.CurrentStatus
✓ Load Lakehouse/Landing/Tables/Parcel.CurrentStatus (read 2, +2 ~0 -0 !0) 11.4s
Execute 25.6s
Record what the run did 0.0s
✓ Load 1m14s
load succeeded: Lakehouse/Landing
Load summary
3 succeeded
0 failed
0 blocked
Rows
read 6
inserted 6
updated 0
deleted 0
rejected 0
Workflow: d3e86d559e894acf8286840edd8a0bbc
The command returned:
{'requested': ['Lakehouse/Landing'],
'status': 'succeeded',
'dry_run': False,
'fault_tolerant': False,
'reload': False,
'ignore_stability_threshold': False,
'workspace': 'Parcel Development',
'workflow_id': '[id]',
'started_at': '2026-10-17T02:26:03.194674+00:00',
'finished_at': '2026-10-17T02:26:22.682167+00:00',
'order': ['load:Lakehouse/Landing/Files/Parcel.Events',
'load:Lakehouse/Landing/Tables/Parcel.Event',
'load:Lakehouse/Landing/Tables/Parcel.CurrentStatus'],
'edges': [['load:Lakehouse/Landing/Files/Parcel.Events',
'load:Lakehouse/Landing/Tables/Parcel.Event'],
['load:Lakehouse/Landing/Tables/Parcel.Event',
'load:Lakehouse/Landing/Tables/Parcel.CurrentStatus']],
'nodes': [{'node_id': 'load:Lakehouse/Landing/Files/Parcel.Events',
'logical_id': 'Lakehouse/Landing/Files/Parcel.Events',
'physical_target': 'Lakehouse/Landing',
'primitive_kind': 'python_folder',
'dispatch_location': 'Lakehouse/Landing/_/Load/Files/Parcel__Events.py',
'status': 'succeeded',
'executed': True,
'started_at': '2026-10-17T02:26:05.097397+00:00',
'finished_at': '2026-10-17T02:26:06.121562+00:00',
'rows': {'succeeded': True,
'rows_read': 1,
'rows_inserted': 0,
'rows_updated': 0,
'rows_deleted': 0,
'rows_rejected': 0,
'error_message': None,
'bookmark_datetime': '2026-10-17T02:26:05.670933+00:00',
'is_static_skip': False,
'is_refusal': False},
'messages': []},
{'node_id': 'load:Lakehouse/Landing/Tables/Parcel.Event',
'logical_id': 'Lakehouse/Landing/Tables/Parcel.Event',
'physical_target': 'Lakehouse/Landing',
'primitive_kind': 'python_table',
'dispatch_location': 'Lakehouse/Landing/_/Load/Tables/Parcel__Event.py',
'status': 'succeeded',
'executed': True,
'started_at': '2026-10-17T02:26:06.125519+00:00',
'finished_at': '2026-10-17T02:26:15.540410+00:00',
'rows': {'succeeded': True,
'rows_read': 3,
'rows_inserted': 0,
'rows_updated': 0,
'rows_deleted': 0,
'rows_rejected': 0,
'error_message': None,
'bookmark_datetime': '2026-10-17T02:26:06.188623+00:00',
'is_static_skip': False,
'is_refusal': False},
'messages': []},
{'node_id': 'load:Lakehouse/Landing/Tables/Parcel.CurrentStatus',
'logical_id': 'Lakehouse/Landing/Tables/Parcel.CurrentStatus',
'physical_target': 'Lakehouse/Landing',
'primitive_kind': 'python_table',
'dispatch_location': 'Lakehouse/Landing/_/Load/Tables/Parcel__CurrentStatus.py',
'status': 'succeeded',
'executed': True,
'started_at': '2026-10-17T02:26:15.540959+00:00',
'finished_at': '2026-10-17T02:26:22.679307+00:00',
'rows': {'succeeded': True,
'rows_read': 2,
'rows_inserted': 0,
'rows_updated': 0,
'rows_deleted': 0,
'rows_rejected': 0,
'error_message': None,
'bookmark_datetime': '2026-10-17T02:26:15.583668+00:00',
'is_static_skip': False,
'is_refusal': False},
'messages': []}],
'messages': []}
The item-wide run loads the carrier Folder, Event and CurrentStatus in order. Its six inserted entries comprise one CSV file, three event rows and two current-status rows.
Run the two generated validations, then read Health:
The command returned:
Test
Read catalogue 2.9s
Workspace Parcel Development
Catalogue Warehouse/Catalogue
Targets
Lakehouse/Landing
Build run graph 0.0s
Wait for Spark session 33.3s
→ Test Lakehouse/Landing/Parcel.CurrentStatusMatches
→ Test Lakehouse/Landing/Parcel.EventStatusIsKnown
✓ Test Lakehouse/Landing/Parcel.EventStatusIsKnown 9.4s
✓ Test Lakehouse/Landing/Parcel.CurrentStatusMatches 11.1s
Execute 10.7s
Record what the run did 0.0s
✓ Test 54.2s
test passed
Test summary
2 passed
0 failed
Workflow: 4294ea0d4ddb47b89c1a03bf2606bc64
The command returned:
{'status': 'passed',
'nodes': [{'logical_id': 'Lakehouse/Landing/Parcel.CurrentStatusMatches',
'kind': 'test',
'physical_target': 'Lakehouse/Landing',
'primitive_kind': 'python_validation',
'dispatch_location': '',
'status': 'passed',
'executed': True,
'messages': [],
'started_at': '2026-10-17T07:25:38.251162+00:00',
'finished_at': '2026-10-17T07:25:52.637740+00:00',
'missing_count': 0,
'unexpected_count': 0,
'failure_count': 0,
'error_message': None},
{'logical_id': 'Lakehouse/Landing/Parcel.EventStatusIsKnown',
'kind': 'assumption',
'physical_target': 'Lakehouse/Landing',
'primitive_kind': 'python_validation',
'dispatch_location': '',
'status': 'passed',
'executed': True,
'messages': [],
'started_at': '2026-10-17T07:25:38.251418+00:00',
'finished_at': '2026-10-17T07:25:50.461669+00:00',
'violation_count': 0,
'error_message': None}],
'totals': {'planned': 2,
'executed': 2,
'passed': 2,
'failed': 0,
'invalid': 0,
'missing_count': 0,
'unexpected_count': 0,
'violation_count': 0},
'workflow_id': '[run-id]',
'started_at': '2026-10-17T07:24:51.457845+00:00',
'finished_at': '2026-10-17T07:25:53.306907+00:00'}
The command returned:
Health
Read catalogue 3.5s
Check installed items 0.2s
Read installed objects 13.8s
✓ Health 17.5s
Weaver Health Green
Load Green
Last load activity 0h 1m ago
3 succeeded
Tests Green
2 succeeded
Build Green
Installed estate consistent (12 objects)
Slowest loads
Lakehouse/Landing/Tables/Parcel.Event 12.1s
Lakehouse/Landing/Tables/Parcel.CurrentStatus 11.4s
Lakehouse/Landing/Files/Parcel.Events 3.1s
Largest changes
Lakehouse/Landing/Tables/Parcel.Event read 3 +3 ~0 -0 !0
Lakehouse/Landing/Tables/Parcel.CurrentStatus read 2 +2 ~0 -0 !0
Lakehouse/Landing/Files/Parcel.Events read 1 +1 ~0 -0 !0
The command returned:
{'format_version': 3,
'generated_at': '2026-10-17T07:26:05.671183+00:00',
'as_of': '2026-10-16T07:26:05.671183+00:00',
'status': 'green',
'targets': ['Lakehouse/Landing'],
'sections': {'load': {'status': 'green',
'subjects': 5,
'counts': {'succeeded': 5},
'findings': []},
'tests': {'status': 'green',
'subjects': 2,
'counts': {'succeeded': 2},
'findings': []},
'build': {'status': 'green', 'subjects': 15, 'counts': {}, 'findings': []}},
'current_load': {'workflow_ids': ['[run-id]'],
'started_at': '2026-10-16T16:53:41.564676+00:00',
'completed_at': '2026-10-16T16:54:05.193602+00:00',
'counts': {'succeeded': 5}},
'load_activity': [{'object_id': 'Lakehouse/Landing/Files/Parcel.Events',
'target': 'Lakehouse/Landing',
'workflow_id': '[run-id]',
'started_at': '2026-10-16T16:53:41.565083+00:00',
'completed_at': '2026-10-16T16:53:44.449771+00:00',
'duration_ms': 2884,
'rows_read': 1,
'rows_inserted': 0,
'rows_updated': 0,
'rows_deleted': 0,
'rows_rejected': 0,
'is_reload': False,
'is_static_skip': False},
{'object_id': 'Lakehouse/Landing/Files/Parcel.StatusFiles',
'target': 'Lakehouse/Landing',
'workflow_id': '[run-id]',
'started_at': '2026-10-16T16:53:41.564676+00:00',
'completed_at': '2026-10-16T16:53:44.607654+00:00',
'duration_ms': 3042,
'rows_read': 1,
'rows_inserted': 0,
'rows_updated': 0,
'rows_deleted': 0,
'rows_rejected': 0,
'is_reload': False,
'is_static_skip': False},
{'object_id': 'Lakehouse/Landing/Tables/Parcel.CurrentStatus',
'target': 'Lakehouse/Landing',
'workflow_id': '[run-id]',
'started_at': '2026-10-16T16:53:58.002926+00:00',
'completed_at': '2026-10-16T16:54:05.193602+00:00',
'duration_ms': 7190,
'rows_read': 2,
'rows_inserted': 0,
'rows_updated': 0,
'rows_deleted': 0,
'rows_rejected': 0,
'is_reload': False,
'is_static_skip': False},
{'object_id': 'Lakehouse/Landing/Tables/Parcel.Event',
'target': 'Lakehouse/Landing',
'workflow_id': '[run-id]',
'started_at': '2026-10-16T16:53:44.450153+00:00',
'completed_at': '2026-10-16T16:53:57.996574+00:00',
'duration_ms': 13546,
'rows_read': 3,
'rows_inserted': 0,
'rows_updated': 0,
'rows_deleted': 0,
'rows_rejected': 0,
'is_reload': False,
'is_static_skip': False},
{'object_id': 'Lakehouse/Landing/Tables/Parcel.LiveStatus',
'target': 'Lakehouse/Landing',
'workflow_id': '[run-id]',
'started_at': '2026-10-16T16:53:44.608108+00:00',
'completed_at': '2026-10-16T16:53:44.729911+00:00',
'duration_ms': 121,
'rows_read': 0,
'rows_inserted': 0,
'rows_updated': 0,
'rows_deleted': 0,
'rows_rejected': 0,
'is_reload': False,
'is_static_skip': False}]}
Inspect the Table¶
In the Fabric portal, open the Landing SQL analytics endpoint and query the Table:
select [Parcel ID], [Event sequence], [Status], [Depot]
from [Parcel].[Event]
order by [Parcel ID], [Event sequence];
The result should contain the three rows in EVENTS. The managed file is Files/Parcel/Events/events.csv.
What Weaver handled¶
The Folder staged the file, and Weaver published it under Files/Parcel/Events. The Python import supplied the Folder → Table dependency, so Load ran the file work first. Build installed the Table and its code; Load applied the three rows and recorded the result. Lineage remains a description, not another dependency declaration.
You now have the event history in a Table. Inspect the Spark SQL CurrentStatus Table to answer the next question: where is each parcel now? Keep the Python Event and its Folder; the next page explains the generated transformation in this same project.
For edits to either declaration, Check and Build again, then use weaver load Lakehouse/Landing --stale for post-Build catch-up. Changing the EVENTS constant is a code change, not an external data-only incremental run. Loading changes supplies that later variation. Exact authored forms are in Folder and Table.