Applications are currently unavailable for this posting.
Browse current jobs- Employment type
- Contract
- Experience level
- Lead
- Posting language
- English
- Application method
- Apply on LinkedIn (LinkedIn sign-in required)
Job summary
Refactor complex T-SQL procedures into optimized Python and PySpark, using vectorized DataFrame operations and tuning Spark performance. Build Azure Data Factory pipelines to ingest loss data, validate results against existing SQL outputs, and support performance testing and troubleshooting.
Job details
Lead Data Engineer Contract Ontario PySpark & Optimization Role Summary We are looking for a hands-on Lead Data Engineer to support a proof of concept in Azure Databricks. The role focuses on converting complex SQL stored procedures into optimized Python and PySpark code. The objective is to demonstrate that the platform can process 100,000 simulation years in minutes rather than hours, while maintaining accurate results. Key Responsibilities • Refactor complex T-SQL logic, including pricing rules, layer applications, and reinstatements, into Python and PySpark. • Replace row-by-row processing with efficient, vectorized DataFrame operations. • Develop Azure Data Factory pipelines to ingest loss data from Touchstone/RMS into ADLS Gen2. • Optimize Spark jobs through effective partitioning, memory management, and skew handling. • Validate the new solution against existing SQL results to ensure accuracy. • Contribute to performance testing and troubleshooting. Required Skills • Advanced Python skills, including Pandas and PySpark. • Strong T-SQL experience and the ability to analyze complex stored procedures. • Hands-on experience with Azure Databricks, Delta Lake, and Azure Data Factory. • Strong knowledge of distributed computing and Spark optimization. • Experience with partitioning, sharding, broadcast variables, and data-skew management. • Familiarity with CI/CD and version control for data engineering projects. • Desirable Skills • Experience with Monte Carlo simulations or stochastic modelling. • Knowledge of Parquet and Avro file formats. • Experience analyzing SQL Server execution plans. • Background in insurance, risk analytics, or simulation-based modelling.
What you’ll do
Refactor complex T-SQL procedures into optimized Python and PySpark, using vectorized DataFrame operations and tuning Spark performance. Build Azure Data Factory pipelines to ingest loss data, validate results against existing SQL outputs, and support performance testing and troubleshooting.
Requirements
Requires advanced Python, Pandas, and PySpark skills; strong T-SQL experience; and hands-on experience with Azure Databricks, Delta Lake, and Azure Data Factory. Candidates should understand distributed computing and Spark optimization, including partitioning, sharding, broadcast variables, and data-skew management; simulation, insurance, and risk analytics experience is desirable.
Listed skills
- Python · Preferred
- CI/CD · Preferred
Other relevant skills
Identified from the job description. Confirm important requirements above.
- Python
- Pandas
- PySpark
- T-SQL
- Azure Databricks
- Delta Lake
- Azure Data Factory
- Spark Optimization
- Distributed Computing
- Data Partitioning
- Sharding
- Broadcast Variables
- Data Skew Management
- CI/CD
- Version Control
- Monte Carlo Simulation
Job areas
- Data & Analytics
- Technology
- Software
- Engineering
More jobs from Ardent SoftSol Inc.
Senior Data Engineer
- Hybrid
- Toronto, ON
- Posted Nov 17, 2026
Senior Full Stack Developer
- Hybrid
- Montréal, QC
- Posted Nov 17, 2026
Senior Data Engineer
- Hybrid
- Toronto, ON
- Posted Nov 17, 2026
Sr FinOps Architect
- Hybrid
- Toronto, ON
- Posted Nov 16, 2026
