Principal Databricks Data Engineer

Join Our Team as a Principal Databricks Data Engineer and Lead the Future of Enterprise Data Platforms Are you a seasoned data architect with a passion for transforming legacy systems into innovative, scalable lakehouse solutions? We are seeking a visionary Principal Data Engineer specialized in Databricks to spearhead enterprise-wide data modernization initiatives. This is your opportunity to shape the future of finance and risk data platforms, influence strategic decision-making, and mentor the next generation of data engineering talent. What You'll Do Lead complex migrations from traditional Cloudera platforms to modern Databricks Lakehouse architectures, redesigning ingestion, transformation, and consumption patterns. Architect and govern enterprise data warehouses and data lakes, implementing layered structures for optimized analytics. Develop, optimize, and maintain large-scale data pipelines using PySpark, Spark SQL, and Delta Lake, ensuring high performance and reliability. Define and implement business metrics, data models, and semantic layers to support advanced reporting, analytics, and AI/ML workloads. Establish and enforce data governance, security, and compliance standards across cloud environments such as AWS or Azure. Collaborate with cross-functional teams including finance, risk, and analytics to deliver data solutions that drive business insight. Mentor senior engineers, champion best practices, and lead architectural decisions aligned with enterprise standards. Required Skills 12–18 years of overall data engineering experience with a strong emphasis on enterprise Data Warehouse and Data Lake platforms. 8+ years of hands-on expertise in designing and implementing data solutions at scale. Minimum of 5 years working with Databricks and Spark, with proven success in modernization projects. Deep understanding of cloud-based data storage, Delta Lake, and SQL frameworks. Strong experience with modern data ingestion, transformation, and orchestration techniques. Proficiency in data governance, lineage, and security, ensuring compliance and data quality. Proven ability to act as a technical authority and lead cross-functional teams. Nice To Have Skills Experience in BFSI, capital markets, or regulatory reporting environments. Familiarity with SAP Finance, Oracle Financials, or S/4HANA. Exposure to supporting AI and ML workloads. Certifications such as Databricks Certified Data Engineer, cloud platform certifications, or related credentials. Knowledge of orchestration tools like Airflow or Databricks Workflows. Experience with CI/CD pipelines using Git, Terraform, Jenkins, or Azure DevOps. Familiarity with dbt or similar frameworks. Preferred Education And Experience Bachelor’s or Master’s degree in Computer Science, Data Science, or related field. Extensive experience in financial services, risk management, or related sectors. Other Requirements Willingness to collaborate with global teams and manage stakeholder relationships across finance, risk, and analytics functions. Ability to work in a fast-paced, dynamic environment with minimal supervision. Prior experience leading enterprise-wide data transformation initiatives is highly desirable. Take the lead in redefining enterprise data architecture—apply now and become a pivotal part of a transformative journey that impacts some of the most critical financial and risk data environments. Your expertise will drive innovation and deliver measurable business impact!

Lead the migration from legacy Cloudera platforms to Databricks Lakehouse architectures and design scalable enterprise data platforms. Develop high-performance data pipelines and establish data governance and security standards across cloud environments.

Requires 12-18 years of data engineering experience, including at least 5 years specifically with Databricks and Spark. A degree in Computer Science or a related field is preferred, along with expertise in cloud-based data storage and financial services experience.

Identified from the job description. Confirm important requirements above.

• Databricks
• PySpark
• Spark SQL
• Delta Lake
• Data Architecture
• Data Governance
• Cloud Computing
• AWS
• Azure
• Data Pipeline Optimization
• Data Modeling
• ETL
• CI/CD
• Terraform
• Airflow
• dbt

Minimum education

Bachelor’s degree

Minimum experience

10+ years

Apply by

Aug 26, 2026