Databricks Data Engineer

Job summary

The role involves developing ETL/ELT pipelines and managing data migration projects using Azure Databricks. You will be responsible for optimizing SQL performance and ensuring data reconciliation across complex data environments.

Job details

Proven experience with Azure Databricks, including Unity Catalog and Data Lakehouse Architecture. Strong proficiency in PySpark and Spark SQL. Expertise in ETL/ELT Development. Experience with Azure Data Lake Storage (ADLS). Familiarity with Git and CI/CD practices. SQL Optimization and Performance Tuning skills. Experience in Data Migration and Reconciliation Frameworks. GCP BigQuery Migration Experience is essential.

What you’ll do

The role involves developing ETL/ELT pipelines and managing data migration projects using Azure Databricks. You will be responsible for optimizing SQL performance and ensuring data reconciliation across complex data environments.

Requirements

Candidates must have proven experience with Azure Databricks, Unity Catalog, and Data Lakehouse architecture. Proficiency in PySpark, Spark SQL, and experience with GCP BigQuery migration are essential requirements.

Listed skills

• CI/CD · Preferred
• Git · Preferred

Other relevant skills

Identified from the job description. Confirm important requirements above.

• Azure Databricks
• Unity Catalog
• Data Lakehouse Architecture
• PySpark
• Spark SQL
• ETL/ELT Development
• Azure Data Lake Storage
• Git
• CI/CD
• SQL Optimization
• Performance Tuning
• Data Migration
• Reconciliation Frameworks
• GCP BigQuery

Job areas

• Data & Analytics
• Technology
• Software
• Engineering
• Consulting