Technical Team Lead – Databricks & Python

We are seeking a highly experienced Technical Team Lead to provide interim leadership coverage. This role requires hands-on technical leadership, deep expertise in Databricks and Python, and the ability to ensure delivery continuity, production stability, and executive-level communication within a complex environment. This position is 6 months to start, plus multiple long term extensions. The successful candidates will operate as senior practitioners, capable of making real-time architectural, development, and release decisions while proactively identifying risks and ensuring regulatory and production readiness. Key Responsibilities: Provide technical and delivery leadership across the Databricks / Python development team inclusive of forecasting initiatives, managing resource allocations, planning, execution, and communication with leadership Make hands-on contributions to design, development, and review of solutions within Databricks (PySpark, Spark SQL) environments Lead architecture validation, code reviews, and challenge assumptions to ensure scalable and production-ready solutions Own end-to-end integration, testing, and validation within production-adjacent environment Identify and communicate delivery risks, technical gaps, and mitigation strategies to senior leadership in real time Support and guide Agile teams, enabling high-quality delivery under tight timelines in high-pressure environments Skills & Experience: 7-10 years of Data Engineering experience along with 3-5 years of Databricks expertise Advanced experience with PySpark, Spark SQL, and Delta Lake in enterprise-scale environments Proven ability to design, optimize, and troubleshoot large-scale data pipelines Strong experience with performance tuning (partitioning, caching, parallelism) Python development leadership experience Deep experience building scalable systems using Python (FastAPI, Django, or similar frameworks) Experience integrating Databricks workflows into enterprise CI/CD pipelines Hands-on coding capability with ability to review, challenge, and improve production code Experience leading engineering teams in high-stakes delivery environments Strong experience in code reviews, design governance, and mentoring senior engineers Financial Services Domain Expertise Please apply today!

Provide technical and delivery leadership across the Databricks/Python development team while making hands-on contributions to design and development. Identify and communicate delivery risks and support Agile teams to ensure high-quality delivery under tight timelines.

Candidates should have 7-10 years of Data Engineering experience with 3-5 years of Databricks expertise. Advanced skills in PySpark, Spark SQL, and Python development are essential, along with experience in leading engineering teams.

Identified from the job description. Confirm important requirements above.

• Databricks
• Python
• PySpark
• Spark SQL
• Delta Lake
• Data Engineering
• Performance Tuning
• FastAPI
• Django
• CI/CD
• Architecture Validation
• Code Reviews
• Agile
• Mentoring
• Production Readiness
• Risk Management

Minimum experience

10+ years