About the role
Toronto
Job DescriptionJob DescriptionRole OverviewWe are seeking a highly skilled and experienced Senior Data Engineer to join our dynamic team. In this role, you will be responsible for designing, building, and maintaining scalable and reliable data pipelines and infrastructure on AWS. You will play a critical role in enabling our data-driven decision-making processes by ensuring the availability and quality of our data. The ideal candidate will possess a strong background in AWS cloud services, Python, SQL, PySpark, Airflow, and infrastructure as code (CDK). Experience with DevOps practices is a significant plus.Responsibilities Data Pipeline Development Design, develop, and maintain robust and scalable data pipelines using Python, PySpark, and Airflow to ingest, process, and transform large datasets. Cloud Infrastructure (AWS) Architect, build, and manage data infrastructure on AWS using services like S3, EC2, EMR, Redshift, Glue, and Lambda. Infrastructure as Code (CDK) Implement and manage infrastructure as code using AWS CDK to ensure consistency, repeatability, and scalability of our data platform. Database Management Design and optimize database schemas and queries using SQL for efficient data storage and retrieval. Data Quality and Testing Implement comprehensive unit testing strategies to ensure data quality and pipeline reliability. Performance Optimization Identify and resolve performance bottlenecks in data pipelines and infrastructure. Collaboration Work closely with data scientists, analysts, and other engineers to understand data requirements and deliver effective solutions. Documentation Create and maintain thorough documentation of data pipelines, infrastructure, and processes. Monitoring and Alerting Implement monitoring and alerting systems to ensure the health and performance of data pipelines and infrastructure. DevOps (Preferred) Contribute to DevOps practices, including CICD pipelines, automated deployments, and infrastructure monitoring.Qualifications Experience 6-10 years of experience in data engineering or a related field. Programming Languages Strong proficiency in Python and SQL. Big Data Technologies Extensive experience with PySpark for distributed data processing. Workflow Orchestration Proven experience with Airflow for scheduling and managing data pipelines. Cloud Computing (AWS) Deep understanding of AWS cloud services and best practices for data engineering. Infrastructure as Code (CDK) Hands-on experience with AWS CDK for infrastructure provisioning and management. Database Systems Solid understanding of relational and NoSQL databases. Testing Strong understanding and implementation of unit testing strategies. Problem-Solving Excellent analytical and problem-solving skills. Communication Strong communication and collaboration skills. DevOps (Preferred) Knowledge of DevOps principles and practices, including CICD, containerization (Docker), and orchestratio
About Atlantis IT Group
"Atlantis IT Group" is an IT venture that offers end-to-end services and products. The focus of "the Company" is to cater one-stop solutions to small and medium-sized businesses.
"Atlantis" had its first branch in Toronto, Canada, and was established in 1990. The Company has emerged from a small farm to a medium company providing all sorts of solutions, headquartered in Toronto, Canada The Company has had its branches and offices in different countries such as the USA, Canada, and India for the last 32 years.
Atlantis is enlarging its global footprint by offering complementary product and service lines by partnering and strategically working with businesses. The primary specialized area of the Company includes ERP solutions, with cloud-based and on-premise solutions available to meet the requirements of all Industries.
OUR PRODUCTS:
"WORKSWARE ERP" The emergence of WORKSWARE ERP is noticed from a ground level to an upgraded form because of the integrated ERP system, dedicated towards forward-thinking discrete manufacturing enterprises.
"BM-Cloud" BM-Cloud is One of the key offerings of Atlantis consulting as a Cloud-based business management ERP software since 2013. The significant feature of BM-Cloud includes easy usage with the enhancement of simplicity and usability as a complete integrated ERP system. The fundamental modules covered by BM-Cloud are accounting, inventory management, purchasing, sales, and HR/payroll systems. In other words, it is designed primarily to support businesses in the trading and retail sectors and includes all the essential components required for a firm to assist in decision-making.
"BM-POS"
We provide a cloud-based retail solution with all the tools required to manage a retail business, including a sales screen, inventory management, real-time dashboard, employee administration, and other functions that keep retailers in charge of their operations.
About the role
Toronto
Job DescriptionJob DescriptionRole OverviewWe are seeking a highly skilled and experienced Senior Data Engineer to join our dynamic team. In this role, you will be responsible for designing, building, and maintaining scalable and reliable data pipelines and infrastructure on AWS. You will play a critical role in enabling our data-driven decision-making processes by ensuring the availability and quality of our data. The ideal candidate will possess a strong background in AWS cloud services, Python, SQL, PySpark, Airflow, and infrastructure as code (CDK). Experience with DevOps practices is a significant plus.Responsibilities Data Pipeline Development Design, develop, and maintain robust and scalable data pipelines using Python, PySpark, and Airflow to ingest, process, and transform large datasets. Cloud Infrastructure (AWS) Architect, build, and manage data infrastructure on AWS using services like S3, EC2, EMR, Redshift, Glue, and Lambda. Infrastructure as Code (CDK) Implement and manage infrastructure as code using AWS CDK to ensure consistency, repeatability, and scalability of our data platform. Database Management Design and optimize database schemas and queries using SQL for efficient data storage and retrieval. Data Quality and Testing Implement comprehensive unit testing strategies to ensure data quality and pipeline reliability. Performance Optimization Identify and resolve performance bottlenecks in data pipelines and infrastructure. Collaboration Work closely with data scientists, analysts, and other engineers to understand data requirements and deliver effective solutions. Documentation Create and maintain thorough documentation of data pipelines, infrastructure, and processes. Monitoring and Alerting Implement monitoring and alerting systems to ensure the health and performance of data pipelines and infrastructure. DevOps (Preferred) Contribute to DevOps practices, including CICD pipelines, automated deployments, and infrastructure monitoring.Qualifications Experience 6-10 years of experience in data engineering or a related field. Programming Languages Strong proficiency in Python and SQL. Big Data Technologies Extensive experience with PySpark for distributed data processing. Workflow Orchestration Proven experience with Airflow for scheduling and managing data pipelines. Cloud Computing (AWS) Deep understanding of AWS cloud services and best practices for data engineering. Infrastructure as Code (CDK) Hands-on experience with AWS CDK for infrastructure provisioning and management. Database Systems Solid understanding of relational and NoSQL databases. Testing Strong understanding and implementation of unit testing strategies. Problem-Solving Excellent analytical and problem-solving skills. Communication Strong communication and collaboration skills. DevOps (Preferred) Knowledge of DevOps principles and practices, including CICD, containerization (Docker), and orchestratio
About Atlantis IT Group
"Atlantis IT Group" is an IT venture that offers end-to-end services and products. The focus of "the Company" is to cater one-stop solutions to small and medium-sized businesses.
"Atlantis" had its first branch in Toronto, Canada, and was established in 1990. The Company has emerged from a small farm to a medium company providing all sorts of solutions, headquartered in Toronto, Canada The Company has had its branches and offices in different countries such as the USA, Canada, and India for the last 32 years.
Atlantis is enlarging its global footprint by offering complementary product and service lines by partnering and strategically working with businesses. The primary specialized area of the Company includes ERP solutions, with cloud-based and on-premise solutions available to meet the requirements of all Industries.
OUR PRODUCTS:
"WORKSWARE ERP" The emergence of WORKSWARE ERP is noticed from a ground level to an upgraded form because of the integrated ERP system, dedicated towards forward-thinking discrete manufacturing enterprises.
"BM-Cloud" BM-Cloud is One of the key offerings of Atlantis consulting as a Cloud-based business management ERP software since 2013. The significant feature of BM-Cloud includes easy usage with the enhancement of simplicity and usability as a complete integrated ERP system. The fundamental modules covered by BM-Cloud are accounting, inventory management, purchasing, sales, and HR/payroll systems. In other words, it is designed primarily to support businesses in the trading and retail sectors and includes all the essential components required for a firm to assist in decision-making.
"BM-POS"
We provide a cloud-based retail solution with all the tools required to manage a retail business, including a sales screen, inventory management, real-time dashboard, employee administration, and other functions that keep retailers in charge of their operations.