Principal Databricks Data Engineer

Job summary

Lead the modernization of legacy Cloudera and enterprise data warehouse platforms into Databricks Lakehouse architectures, building and optimizing large-scale pipelines and layered data platforms. Govern data quality, reconciliation, lineage, security, and reporting capabilities while guiding engineering teams and partnering with finance, risk, analytics, and governance stakeholders.

Job details

**Title: Principal Data Engineer – Databricks** **Key Requirements** * 12–18 years of overall data engineering experience * 8+ years of experience in enterprise Data Warehouse and Data Lake platforms * 5+ years of hands-on experience with Databricks and Spark at scale * Strong experience in modernizing legacy Cloudera platforms (CDH/CDP, Hive, HBase, Impala, Spark) to Databricks Lakehouse * Redesign ingestion, transformation, and consumption patterns from HDFS-based architecture to cloud object storage and Delta Lake * Refactor legacy Hive/Impala logic into PySpark and Spark SQL ELT pipelines * Ensure data reconciliation, audit integrity, and consistency during migration * Design and govern enterprise Data Warehouse and Data Lake/Lakehouse architectures * Implement layered architecture including Raw/Landing, Curated/Conformed, and Semantic/Consumption layers * Modernize traditional EDW platforms into scalable lakehouse architectures * Strong experience in finance and risk data models including General Ledger, Sub-ledger, financial hierarchies, and risk exposure models (credit, liquidity, market risk) * Enable reporting use cases including aggregation, drill-down, and drill-back capabilities * Build and manage semantic/consumption layers for BI, reporting, and analytics * Define business metrics, dimensions, hierarchies, and KPIs * Experience with Databricks SQL, Delta tables, and dbt or similar frameworks * Develop and optimize large-scale data pipelines using PySpark, Spark SQL, and Delta Lake * Implement Medallion architecture (Bronze, Silver, Gold layers) * Optimize workloads using Z-ORDER, OPTIMIZE, caching, and cluster configurations * Implement data governance, data quality frameworks, reconciliation controls, and exception handling * Establish data lineage and metadata management * Ensure data security, access control, and compliance standards * Experience with cloud platforms such as AWS or Azure * Experience with CI/CD pipelines using Git, Terraform, Jenkins, or Azure DevOps * Familiarity with orchestration tools such as Airflow or Databricks Workflows * Experience with dbt is a plus * Act as a technical authority and lead architecture decisions * Mentor and guide senior engineers and establish engineering standards * Strong stakeholder management with finance, risk, analytics, and governance teams * Ability to translate complex data structures into business-ready insights **Nice to Have** * Experience in BFSI, Capital Markets, or regulatory reporting * Exposure to SAP Finance, Oracle Financials, or S/4HANA * Experience supporting AI/ML workloads * Databricks or cloud certifications **Impact** * Lead Cloudera to Databricks transformation initiatives * Shape enterprise finance and risk data platforms * Support regulatory, management, and analytical reporting systems

What you’ll do

Lead the modernization of legacy Cloudera and enterprise data warehouse platforms into Databricks Lakehouse architectures, building and optimizing large-scale pipelines and layered data platforms. Govern data quality, reconciliation, lineage, security, and reporting capabilities while guiding engineering teams and partnering with finance, risk, analytics, and governance stakeholders.

Requirements

Requires 12–18 years of data engineering experience, including 8+ years with enterprise data warehouse and data lake platforms and 5+ years of hands-on Databricks and Spark experience. Candidates should bring expertise in Cloudera modernization, finance and risk data models, cloud platforms, data governance, and technical leadership; CI/CD and orchestration experience are also expected, while dbt experience is a plus.

Listed skills

• Microsoft Azure · Preferred
• SQL · Preferred
• Analytical · Preferred
• Technical · Preferred
• CI/CD · Preferred
• Teams · Preferred
• Data analysis · Preferred
• management · Preferred
• Reporting · Preferred
• Amazon Web Services · Preferred
• Warehouse · Preferred
• Audit · Preferred

Other relevant skills

Identified from the job description. Confirm important requirements above.

• Databricks
• Apache Spark
• PySpark
• Spark SQL
• Delta Lake
• Data Warehouse Architecture
• Data Lakehouse Architecture
• Cloudera Migration
• Financial and Risk Data Modeling
• Data Governance
• Data Quality
• Data Reconciliation
• Databricks SQL
• dbt
• Cloud Platforms
• Technical Leadership

Job areas

• Data & Analytics
• Technology
• Engineering
• Finance & Accounting
• Management & Leadership

More jobs from Zodiac Solutions, Inc