Testing Lakehouse data¶
Check that CurrentStatus returns the expected parcels, and that every event uses a recognised status. These validations live beside the same Folder and Tables; Build installs them, and Test runs them separately from Load.
Continue the same parcel project, with Event and CurrentStatus loaded. If you are starting here, run the Python project first.
Compare CurrentStatus with an independent result¶
Open the generated Lakehouse/Landing/tests/Parcel.CurrentStatusMatches.sql:
/*
Test ID: Parcel.CurrentStatusMatches
Description: Current statuses match the carrier export's expected latest events.
Primary key: Parcel ID
*/
select *
from values
('P-1001', 'In transit', 'Central'),
('P-1002', 'Delivered', 'South')
as expected(`Parcel ID`, `Status`, `Depot`);
select `Parcel ID`, `Status`, `Depot`
from Parcel.CurrentStatus;
The first query is the expected result from the small carrier export; the second reads the installed Table. It does not repeat CurrentStatus's window calculation. Both return Parcel ID, Status and Depot in the same order. Primary key pairs diagnostic rows for a parcel; it does not change the comparison.
Find unknown event statuses¶
Open the generated Lakehouse/Landing/assumptions/Parcel.EventStatusIsKnown.sql:
/*
Assumption ID: Parcel.EventStatusIsKnown
Description: Every parcel event has a recognised status.
*/
select `Parcel ID`, `Event sequence`, `Status`
from Parcel.Event
where `Status` is null
or `Status` not in ('Accepted', 'In transit', 'Delivered');
An Assumption returns the rows that contradict its statement. This one passes only when that query returns no rows. SQL validations infer their managed dependencies from relation references; unlike Spark SQL Tables, they do not require an explicit Dependencies field.
Run both validations¶
Initialise wrote both validations and the first Build installed them, so run them directly:
The command returned:
Test
Read catalogue 2.9s
Workspace Parcel Development
Catalogue Warehouse/Catalogue
Targets
Lakehouse/Landing
Build run graph 0.0s
Wait for Spark session 33.3s
→ Test Lakehouse/Landing/Parcel.CurrentStatusMatches
→ Test Lakehouse/Landing/Parcel.EventStatusIsKnown
✓ Test Lakehouse/Landing/Parcel.EventStatusIsKnown 9.4s
✓ Test Lakehouse/Landing/Parcel.CurrentStatusMatches 11.1s
Execute 10.7s
Record what the run did 0.0s
✓ Test 54.2s
test passed
Test summary
2 passed
0 failed
Workflow: 4294ea0d4ddb47b89c1a03bf2606bc64
The command returned:
{'status': 'passed',
'nodes': [{'logical_id': 'Lakehouse/Landing/Parcel.CurrentStatusMatches',
'kind': 'test',
'physical_target': 'Lakehouse/Landing',
'primitive_kind': 'python_validation',
'dispatch_location': '',
'status': 'passed',
'executed': True,
'messages': [],
'started_at': '2026-10-17T07:25:38.251162+00:00',
'finished_at': '2026-10-17T07:25:52.637740+00:00',
'missing_count': 0,
'unexpected_count': 0,
'failure_count': 0,
'error_message': None},
{'logical_id': 'Lakehouse/Landing/Parcel.EventStatusIsKnown',
'kind': 'assumption',
'physical_target': 'Lakehouse/Landing',
'primitive_kind': 'python_validation',
'dispatch_location': '',
'status': 'passed',
'executed': True,
'messages': [],
'started_at': '2026-10-17T07:25:38.251418+00:00',
'finished_at': '2026-10-17T07:25:50.461669+00:00',
'violation_count': 0,
'error_message': None}],
'totals': {'planned': 2,
'executed': 2,
'passed': 2,
'failed': 0,
'invalid': 0,
'missing_count': 0,
'unexpected_count': 0,
'violation_count': 0},
'workflow_id': '[run-id]',
'started_at': '2026-10-17T07:24:51.457845+00:00',
'finished_at': '2026-10-17T07:25:53.306907+00:00'}
Both validations passed against the original export. Build installs validations but does not run them, and Test does not load their inputs for you. When you add or edit a validation, Check and Build install it before Test.
To run these source files before installing a validation edit:
weaver test Lakehouse/Landing \
--file Lakehouse/Landing/tests/Parcel.CurrentStatusMatches.sql \
--file Lakehouse/Landing/assumptions/Parcel.EventStatusIsKnown.sql
File mode reads the deployed Tables but records no catalogue Test state. Use the installed-mode command above when you want Health to reflect the result.
Optional: diagnose one mismatch¶
In the Test's expected query only, change P-1001's status from In transit to Delivered. Leave the Folder, Event and CurrentStatus unchanged. Check and Build the validation edit, then select that installed Test. --name is a whole-name regular expression, so [.] matches the literal dot:
weaver check
weaver build --item Lakehouse/Landing
weaver test Lakehouse/Landing --name 'Parcel[.]CurrentStatusMatches'
weaver health --item Lakehouse/Landing
The selected Test should fail and return diagnostic rows showing the expected Delivered and actual In transit. Health should show the failed validation. There is no Load here because the data declarations and their intended data have not changed. Restore the expected value, Check and Build again, then rerun the selected Test; it should succeed.
The small fixed export made the expected rows explicit. A changing carrier feed needs processing state as well as tests: Loading changes and bookmarks shows a changing carrier-feed example, where updating or removing a file changes an additional LiveStatus Table without rebuilding its code or replacing this fixed example.
Tests, Assumptions and Test behaviour own the exact comparison, diagnostic and recording rules. Python validations use the same data: a Test implements expected() and actual() returning Spark DataFrames, while an Assumption implements read() returning violations; their complete forms are in those references.