Found Description
Job DescriptionOverallmso-spacerun:yes> 6– 8 years of relevant experience.Should be flexible to adapt to project requirements needs.Key Responsibilities:Data Pipeline Development: Design, build, and optimize robust, scalable, and efficient ETL/ELT data pipelines using Python and PySpark, primarily within Azure Databricks and Azure Data Factory.Data Ingestion & Processing: Develop and manage processes for ingesting data from various sources (e.g., transactional databases, APIs, streaming sources) and transforming it into clean, usable formats for downstream consumption.Data Quality & Monitoring: Implement comprehensive unit and integration test coverage for data pipelines. Establish and maintain monitoring, alerting, and dashboarding solutions (e.g., Grafana) for data quality, pipeline health, and performance.Cloud Infrastructure Management (OpenShift/Azure): Contribute to the setup, configuration, and maintenance of data-related infrastructu...