Web
2.2.a Push
Ingestion Jobs
Metadata Driven Framework For Enterprise Data Process
ADS
3.3. Modelingand Analysis
4 Run Outcome Jobs
1.2 OrchestrationConfiguation
2 Run Ingestion Jobs(Skippable)
Visualization
Notebook
Run with parameters
ODS
Validate Table Data By Rules
2.4. Load
Temporary Zone
MDDExecution-Logs Database
Heterogeneouse Data Sources:
DWD
- SQL Server- MySQL- Oracle- PostgreSQL- MongoDB- ODATA/REST API- Excel/CSV/Parquet
2.1 Pull full orincremental data
Landing Zone
Notification
STG
Silver Zone
3.2. Update Detail
Metadata
Analytical Jobs
Archive Zone
AzureData Factory
Copy
3 Run Analytical Jobs
Outcome Jobs
R/W Temp Data
Logs
Databricks Unity Catalog
4.1. Email Notification
3.1 Cleans and Extract Dims and Facts
2.2.b Pull full or incremental data
Email Summary
Data Quality Zone
delta file
Archive/Restore Data
Orchestration
parquet file
Bronze Zone
Recycle/Restore Data
MultiDBMetadata Extractor
2.3. Sink file to lake
DWS
1.1. Extract Metadata
MDDConfiguration Database
Power BIReport Server
Quality Dashboard
Gold Zone
