Data Engineer

Job summary

Design, build, and maintain scalable batch and streaming data pipelines, integration workflows, warehouses, lakes, and data models. Ensure data quality and security, monitor and troubleshoot pipeline performance, optimize processing and costs, document technical workflows, and collaborate with analytics and software teams.

Job details

Key Responsibilities Design, develop, and maintain scalable data pipelines for batch and streaming data. Extract, transform, and load data from multiple sources using ETL/ELT processes. Develop and optimize SQL queries for data processing and analysis. Build data integration workflows using Python and relevant data engineering tools. Design and maintain data warehouses, data lakes, and data models. Ensure data quality, accuracy, consistency, and reliability. Monitor pipeline performance and troubleshoot data processing issues. Collaborate with Data Analysts, Data Scientists, and software development teams. Implement data validation, monitoring, logging, and error-handling mechanisms. Optimize data storage, processing performance, and infrastructure costs. Document data pipelines, workflows, and technical processes. Follow data security, access control, and governance practices. Required Skills Strong knowledge of SQL, including joins, subqueries, CTEs, and window functions. Programming knowledge in Python. Understanding of ETL/ELT pipelines and data integration concepts. Knowledge of relational databases such as MySQL, PostgreSQL, or SQL Server. Understanding of data warehousing, data modeling, and database design. Familiarity with file formats such as CSV, JSON, and Parquet. Knowledge of data quality checks and pipeline troubleshooting. Understanding of distributed data processing concepts. Preferred Skills Experience with Apache Spark or PySpark. Familiarity with Apache Airflow or other workflow orchestration tools. Knowledge of cloud platforms such as AWS, Microsoft Azure, or Google Cloud Platform. Experience with cloud data warehouses such as Snowflake, BigQuery, or Amazon Redshift. Familiarity with Apache Kafka for streaming data. Knowledge of Databricks and Delta Lake. Understanding of Docker, Git, and CI/CD pipelines. Familiarity with data governance, metadata management, and data security.

What you’ll do

Design, build, and maintain scalable batch and streaming data pipelines, integration workflows, warehouses, lakes, and data models. Ensure data quality and security, monitor and troubleshoot pipeline performance, optimize processing and costs, document technical workflows, and collaborate with analytics and software teams.

Requirements

Requires strong SQL and Python skills, knowledge of ETL/ELT, relational databases, data warehousing, data modeling, common file formats, data quality, and distributed processing. Preferred qualifications include experience with Spark, Airflow, cloud platforms and warehouses, Kafka, Databricks, Delta Lake, Docker, Git, CI/CD, and data governance.

Listed skills

• SQL · Preferred
• Python · Preferred

Other relevant skills

Identified from the job description. Confirm important requirements above.

• SQL
• Python
• ETL/ELT
• Data Pipelines
• Data Integration
• Relational Databases
• Data Warehousing
• Data Modeling
• Data Quality
• Distributed Data Processing
• Apache Spark
• Apache Airflow
• Cloud Platforms
• Cloud Data Warehouses
• Apache Kafka
• Databricks

Job areas

• Data & Analytics
• Technology
• Software

More jobs from Spait Infotech