Hadoop Developer

Job summary

Develop and maintain scalable data processing solutions and distributed ETL pipelines using the Hadoop ecosystem. Build Kafka-based ingestion solutions and manage Airflow workflows for data pipeline orchestration.

Job details

Immediate need for a talented Hadoop Developer This is a 12+ months opportunity with long-term potential and is located in Canada/United States (onsite) Please review the job description below and contact me ASAP if you are interested. Job Diva ID: 26-24992 Pay Range: $50 - $60/hour. Employee benefits include, but are not limited to, health insurance (medical, dental, vision), 401(k) plan, and paid sick leave (depending on work location). Key Responsibilities: - Note- Candidates must be willing to relocate and work onsite in the USA from Day 1 Develop and maintain scalable data processing solutions using the Hadoop ecosystem, including HDFS, YARN, Hive, and HBase. Design and develop distributed ETL pipelines for large-volume data processing and transformation. Develop Spark applications using Spark SQL and Spark Streaming for batch and real-time data processing. Build and maintain Kafka-based data ingestion and streaming solutions. Use Sqoop to transfer data between relational databases and Hadoop environments. Develop applications and data processing scripts using Python, Scala, and/or Java. Create Shell scripts to automate data processing and operational tasks. Develop and manage Airflow workflows for data pipeline orchestration and scheduling. Optimize distributed processing jobs for performance, scalability, and reliability. Troubleshoot data pipelines, Spark jobs, Hadoop components, and production data-processing issues. Collaborate with data engineers and technical teams to support data integration and large-scale analytics requirements. Key Requirements and Technology Experience: Must have skills: - Hadoop Ecosystem (HDFS, YARN, Hive, HBase), Spark (SQL, Streaming), Kafka, Sqoop, Python/Scala/Java, Shell scripting, Airflow (workflow orchestration), distributed data processing, ETL pipelines Our client is a leading IT Industry, and we are currently interviewing to fill this and other similar contract positions. If you are interested in this position, please apply online for immediate consideration Pyramid Consulting, Inc. provides equal employment opportunities to all employees and applicants for employment and prohibits discrimination and harassment of any type without regard to race, colour, religion, age, sex, national origin, disability status, genetics, protected veteran status, sexual orientation, gender identity or expression, or any other characteristic protected by federal, state, or local laws. By applying to our jobs, you agree to receive calls, AI-generated calls, text messages, or emails from Pyramid Consulting, Inc. and its affiliates, and contracted partners. Frequency varies for text messages. Message and data rates may apply. Carriers are not liable for delayed or undelivered messages. You can reply STOP to cancel and HELP for help. You can access our privacy policy here.

What you’ll do

Develop and maintain scalable data processing solutions and distributed ETL pipelines using the Hadoop ecosystem. Build Kafka-based ingestion solutions and manage Airflow workflows for data pipeline orchestration.

Requirements

Candidates must have expertise in Hadoop, Spark, Kafka, and Airflow, along with proficiency in Python, Scala, or Java. Experience in distributed data processing and shell scripting is required.

Benefits

• Health insurance • Medical • Dental • Vision • 401(k) plan • Paid sick leave

Listed skills

• Java · Preferred
• Python · Preferred

Other relevant skills

Identified from the job description. Confirm important requirements above.

• Hadoop Ecosystem
• Hdfs
• Yarn
• Hive
• Hbase
• Spark
• Spark Sql
• Spark Streaming
• Kafka
• Sqoop
• Python
• Scala
• Java
• Shell Scripting
• Airflow
• Etl Pipelines

Job areas

• Data & Analytics
• Technology
• Software
• Finance & Accounting
• Consulting