Job Description

Insight Global is seeking a Databricks Developer for a large maritime transportation organization. This role is focused on building scalable data pipelines, optimizing Spark workloads, and developing reliable data products within an Azure cloud environment. The ideal candidate will have strong hands-on experience with Databricks, PySpark, Azure Data Factory, and modern data engineering best practices.

Required Skills & Experience

3+ years of experience in Data Engineering or a related field Strong hands-on experience with Databricks, including: PySpark, Spark SQL, Delta Lake, Databricks notebooks and workflows Experience building ETL/ELT pipelines using Azure Data Factory (ADF) and Databricks Strong SQL and Python development skills Experience integrating and transforming structured and semi-structured data Experience working with REST APIs and JSON-based data ingestion Strong understanding of data modeling, data warehousing, and data quality practices Experience working with data formats such as JSON, Parquet, and Avro Experience with Git, Azure DevOps, GitHub, and CI/CD pipelines Strong troubleshooting and Spark performance optimization skills Excellent communication and collaboration skills Nice-to-Have Skills Experience with Kafka, MQTT, Event Hub, or other streaming technologies Experience working with ADLS and SQL Server Knowledge of data governance and metadata management Azure certifications (DP-203 or related) Experience in transportation, logistics, or enterprise-scale environments Responsibilities Design, develop, and maintain scalable data pipelines using Databricks, PySpark, SQL, and Delta Lake Build and support ETL/ELT processes for structured and semi-structured data sources Integrate data from APIs, databases, cloud storage, and other enterprise systems Develop and optimize Spark workloads through partitioning, caching, broadcast joins, Z-ordering, and other performance tuning techniques Build ingestion frameworks that transform REST API and JSON data into analytics-ready datasets Create and maintain Azure Data Factory pipelines, linked services, datasets, triggers, and orchestration workflows Develop parameterized ADF solutions and automate Databricks notebook execution Ensure data quality through validation, monitoring, and testing frameworks Contribute to data governance, documentation, and operational best practices Collaborate with Data Engineers, Architects, Analysts, and Business Stakeholders to deliver trusted enterprise datasets Support CI/CD processes and version-controlled deployments Develop and maintain both batch and real-time data pipelines leveraging technologies such as Kafka, MQTT, and Azure Event Hub Troubleshoot production issues and continuously improve platform performance, reliability, and scalability Preferred Technologies: Databricks, PySpark, Spark SQL, Delta Lake, Azure Data Factory, Azure Data Lake Storage (ADLS), Python, SQL, Git, Azure DevOps, GitHub Actions, Kafka, Event Hub, MQTT, REST APIs.