Senior Data Engineer –Databricks

Job summary

Develop and implement Databricks data solutions, including pipelines for batch and streaming workloads, while ensuring data quality and efficiency. Collaborate with cross-functional teams to optimize data pipelines and stay current with Databricks features and related technologies.

Job details

Senior Data Engineer –Databricks 3 months Contract with possibility of extension. Work Arrangement - Remote Please find below the details JD: Key Responsibilities: Develop and implement data solutions using Databricks, with hands-on experience in specific Databricks platform features such as Delta Lake, Uniform (Iceberg), Delta Live Tables (Lakeflow Declarative Pipelines), and Unity Catalog. Collaborate with cross-functional teams to design and optimize data pipelines for both batch and streaming data, ensuring data quality and efficiency. Stay updated with emerging technologies, latest Databricks platform features and continuously improve your skills in Databricks and other relevant tools. Mandatory Requirements: 5+ years of data engineering experience, with at least 2 years of hands-on data pipeline design and development experience with Databricks, including specific -platform features like Delta Lake, Uniform (Iceberg), Delta Live Tables (Lakeflow Declarative pipelines), and Unity Catalog. Proficiency in designing and building robust, scalable, YAML configuration driven data pipelines with batch, micro-batch, and streaming data ingestion and processing patterns using tools like Auto Loader demonstrating a strong understanding of modern data engineering practices.Experienced in building complex job/workflow orchestration patterns using either Databricks jobs/workflows or with other orchestration tools like Airflow, dbt etc. Exposure in building robust data quality, pipeline audit/observability solutions with Databricks native features and/or other data quality tools/frameworks like Great Expectations, Collibra, dbt etc. Proficiency in Big Data Platforms: Apache Spark, Presto, Amazon EMR. Experience with Cloud Data Warehouses: Amazon Redshift, Snowflake, Google BigQuery. Strong programming skills using SQL, Stored Procedures and Object-Oriented Programming languages (like Java, Python, PySpark etc.). Familiarity with building DevOPs, CI/CD pipelines using Databricks Asset Bundles with automated validation and testing. Exposure with Infrastructure as Code (IaC) tools like Terraform is a big plus. Familiarity with NoSQL Databases and Container Management Systems. Exposure to AI/ML tools (like mlflow), prompt engineering, and modern data and AI agentic workflows. An ideal candidate will have Databricks Data Engineering Associate and/or Professional certifications completed with multiple Databricks project delivery experience.

What you’ll do

Develop and implement Databricks data solutions, including pipelines for batch and streaming workloads, while ensuring data quality and efficiency. Collaborate with cross-functional teams to optimize data pipelines and stay current with Databricks features and related technologies.

Requirements

Requires 5+ years of data engineering experience, including at least 2 years designing and developing Databricks pipelines using features such as Delta Lake, Delta Live Tables, and Unity Catalog. Candidates should have strong skills in scalable pipeline development, orchestration, data quality, big data platforms, cloud data warehouses, programming, and CI/CD; Databricks certifications and Terraform experience are advantageous.

Listed skills

• SQL · Preferred
• Python · Preferred
• CI/CD · Preferred
• Terraform · Preferred

Other relevant skills

Identified from the job description. Confirm important requirements above.

• Databricks
• Delta Lake
• Apache Iceberg
• Delta Live Tables
• Unity Catalog
• Data Pipeline Design
• Batch And Streaming Data Processing
• Auto Loader
• Workflow Orchestration
• Apache Spark
• SQL
• Python
• PySpark
• CI/CD
• Databricks Asset Bundles
• Terraform

Job areas

• Data & Analytics
• Technology
• Software
• Engineering