Databricks Data Engineer

Job summary

Design and develop scalable ETL/ELT pipelines using Azure Databricks and PySpark while implementing Medallion Architecture. Migrate BigQuery datasets to Delta Lake and develop data validation and quality-check frameworks.

Job details

Key Responsibilities Design and develop scalable ETL/ELT pipelines using Azure Databricks and PySpark. Migrate BigQuery datasets, schemas, tables, and views to Delta Lake. Build and optimize batch and incremental data processing pipelines. Implement Medallion Architecture (Bronze, Silver, Gold) and Delta Lake best practices. Develop data validation, reconciliation, and quality-check frameworks. Support historical data migration, performance tuning, and production deployment. Mandatory Skills Proven experience with Azure Databricks, including Unity Catalog and Data Lakehouse Architecture. Strong proficiency in PySpark and Spark SQL. Expertise in ETL/ELT Development. Experience with Azure Data Lake Storage (ADLS). Familiarity with Git and CI/CD practices. SQL Optimization and Performance Tuning skills. Experience in Data Migration and Reconciliation Frameworks. GCP BigQuery Migration Experience is essential. Preferred Skills Experience with Airflow / Cloud Composer Dataflow. Familiarity with DBT (Data Build Tool) for data transformation. Qualifications The ideal candidate will possess a bachelor’s degree in computer science, Information Technology, or a related field. A minimum of 7-10 years of experience in data engineering, with a strong focus on cloud-based data solutions, is required. Candidates should demonstrate a solid understanding of data architecture principles and best practices.

What you’ll do

Design and develop scalable ETL/ELT pipelines using Azure Databricks and PySpark while implementing Medallion Architecture. Migrate BigQuery datasets to Delta Lake and develop data validation and quality-check frameworks.

Requirements

Requires 7-10 years of data engineering experience with a strong focus on cloud-based solutions and GCP BigQuery migration. A bachelor's degree in Computer Science or a related field is required.

Listed skills

• CI/CD · Preferred
• Git · Preferred

Other relevant skills

Identified from the job description. Confirm important requirements above.

• Azure Databricks
• PySpark
• Spark SQL
• ETL/ELT Development
• Azure Data Lake Storage
• Unity Catalog
• Data Lakehouse Architecture
• GCP BigQuery Migration
• Medallion Architecture
• Delta Lake
• Git
• CI/CD
• SQL Optimization
• Airflow
• Cloud Composer
• DBT

Job areas

• Data & Analytics
• Technology
• Software
• Engineering
• Consulting