Found Description
Responsibilities
- Design, develop, test, deploy and maintain ETL/ELT pipelines using Azure Data Factory (ADF) and Databricks.
- Implement scalable data processing and transformation logic in PySpark and SQL on Databricks (Delta Lake / Lakehouse patterns).
- Ingest and integrate data from diverse batch and streaming sources into ADLS Gen2 and Delta Lake.
- Build curated, documented datasets for analytics and ML (support dimensional models where appropriate).
- Implement automated data quality checks, unit/integration tests, and validation gates.
- Monitor pipeline health, implement logging, alerting and dashboards; perform RCA and production incident remediation.
- Optimise pipeline performance and cost cluster sizing, job scheduling, partitioning, caching and query tuning.
- Implement and maintain CI/CD for data jobs and infrastructure (Terraform/ARM/Bicep, Azure DevOps or GitHub Actions).
- Ensure data governan...