Data Engineer – Azure and Databricks
--Yoursoft Alternative--
Data Engineer – Azure and Databricks
Primary Skills: Azure, Databricks, PySpark, Python, SQL, ETL/ELT, Azure Data Factory, Data Lake, Delta Lake, Data Warehouse, APIs, Data Governance
Skill Level: Master
────────────────────────────────────────────────────────
Role Overview
Data Engineer responsible for designing, developing, and maintaining scalable data pipelines and data platforms using Microsoft Azure and Databricks, supporting reporting, analytics, and AI use cases.
────────────────────────────────────────────────────────
Key Responsibilities
- Design and develop scalable ETL/ELT data pipelines using Azure and Databricks.
- Build and optimize Data Lake, Delta Lake, and Data Warehouse solutions.
- Integrate data from multiple enterprise systems using Azure Data Factory, Databricks, APIs, and cloud services.
- Ensure data quality, security, governance, and performance.
- Implement monitoring and data engineering best practices.
- Prepare and transform trusted data products for advanced analytics and Agentic AI use cases.
- Collaborate with data, AI, and business teams to deliver scalable data solutions.
────────────────────────────────────────────────────────
Required Skills
- Strong experience with Microsoft Azure and Databricks.
- Strong Python, PySpark, and SQL skills.
- Experience designing and developing ETL/ELT pipelines.
- Experience with Azure Data Factory.
- Strong knowledge of Data Lake / Delta Lake / Data Warehouse architectures.
- Experience integrating data through APIs and enterprise systems.
- Understanding of data quality, governance, security, and performance optimization.
- Experience working with large-scale data and scalable cloud data platforms.
────────────────────────────────────────────────────────
Nice to Have
- Experience supporting AI / Agentic AI use cases.
- Experience with Azure AI and machine learning platforms.
- Knowledge of data product architecture and modern data engineering practices.
- Experience with CI/CD, DevOps, and infrastructure-as-code.