Lakehouse

Write ordinary Python or Spark SQL. Weaver builds the managed files and Tables, orders their dependencies and records what each Load processed.

Start with a Folder feeding a Table. The Python import connects the two; you keep the transformation code rather than writing another scheduler around it.

Then add a Spark SQL transformation, test the data or load changes using a consumer bookmark. The bookmark belongs to the consumer, so two Tables can process the same source at their own pace.

For existing data, use Shortcuts. Running Python in Fabric covers the matching runtime. When items need shared configuration or operations, Coordinate your estate.