Data Engineer with Databricks

Data Engineer with Databricks

Skill: Data Engineering

Skill level: Any

Role Overview

We are looking for a Data Engineer to join a multidisciplinary Agile team responsible for developing and enhancing modern data platforms. The ideal candidate has strong experience building and maintaining scalable data pipelines, working with cloud-native technologies, and processing large volumes of data in AWS environments.

The role involves contributing to both the evolution of existing ("brownfield") solutions and the implementation of new data capabilities while ensuring performance, scalability, and data quality.

Key Responsibilities

Data Engineering & Development

• Design, develop, and maintain scalable data pipelines for batch and streaming workloads.
• Develop robust data transformation processes using Python and SQL.
• Build and optimize data ingestion, processing, and storage solutions.
• Ensure data quality, consistency, and reliability across data platforms.

Cloud Data Platform

• Develop and maintain solutions using Databricks on AWS.
• Optimize distributed data processing workloads for performance and scalability.
• Work with cloud-native services to support modern data architectures.

Data Processing & Performance

• Process and analyze large-scale datasets efficiently.
• Optimize SQL queries and data processing jobs.
• Identify and resolve performance bottlenecks within data pipelines.
• Contribute to data architecture improvements to support growing business requirements.

Collaboration & Delivery

• Work within cross-functional Agile teams delivering incremental business value.
• Collaborate with Data Architects, Data Scientists, Software Engineers, and Business Analysts.
• Participate in code reviews, testing, and continuous improvement initiatives.
• Contribute to technical documentation and knowledge sharing.

Required Skills & Experience

• Experience working on brownfield products and within cross-functional Agile teams.
• Strong programming skills in Python.
• Advanced knowledge of SQL.
• Hands-on experience with Databricks.
• Experience working with AWS cloud services.
• Experience processing and managing large-scale datasets.
• Strong understanding of data scalability, performance optimization, and distributed data processing.
• Knowledge of modern data engineering best practices.

Nice to Have

• Experience with CI/CD pipelines for data platforms.
• Knowledge of Apache Spark.
• Experience with Infrastructure as Code (Terraform, CloudFormation).
• Familiarity with data governance and data quality frameworks.
• Exposure to data lake or lakehouse architectures.

Preferred Soft Skills

• Strong analytical and problem-solving skills.
• Excellent collaboration and communication abilities.
• Ability to work independently while contributing effectively within Agile teams.
• Continuous learning mindset and adaptability to new technologies.