Develop & Optimize Data Pipelines Build, test, and maintainETL/ELT data pipelines using AzureDatabricks & Apache Spark (PySpark) .Optimizeperformance and cost-efficiency of Spark jobs.Ensure data quality through validation, monitoring, and alerting mechanisms.Understand cluster types, configuration, and use-case for serverless
Implement Unity Catalog for Data Governance Design and enforceaccess control policies using Unity Catalog.Managedata lineage, auditing, and metadata governance .Enable secure data sharing across teams and external stakeholders.
Integrate with Cloud Data Platforms Work withAzure Data Lake Storage / Azure Blob Storage/ Azure Event Hub to integrate Databricks with cloud-baseddata lakes, data warehouses, and event streams .ImplementDelta Lake for scalable, ACID-compliant storage.
Automate ...
Ready to Apply?
Submit your application for Senior Data Engineer at Toppan Merrill