GCP Senior Data Engineer

Job summary

Lead a team of 10 engineers while building and optimizing data pipelines and managing GCP data infrastructure, including BigQuery, Cloud Storage, and Data Fusion. Establish engineering standards and monitoring, optimize performance and costs, and collaborate on self-service data platforms, analytics, machine learning, data quality, and governance.

Job details

Required Skills Leadership: Experience leading data engineering teams onshore and offshore. This is a lead position, not a senior data engineer position. Candidates without leadership experience will not be considered for this role. Technical: Proficiency in SQL, Python, Java/Scala, and distributed systems (Spark, Beam). GCP Expertise: Hands-on with BigQuery, Dataflow, Dataproc, Pub/Sub, Composer (Airflow), and Terraform. DevOps: Familiarity with CI/CD, Git, and infrastructure-as-code. Nice to Have GCP Certifications (e.g., Professional Data Engineer). Experience with real-time processing, Vertex AI, or multi-cloud environments. Key Responsibilities Pipeline Development: Build and optimize ETL/ELT pipelines using Dataflow (Apache Beam), Dataproc (Spark), Pub/Sub, and Cloud Functions. Data Infrastructure: Architect and manage BigQuery, Cloud Storage, and Data Fusion for efficient data ingestion, transformation, and storage. Team Leadership: Lead a team of 10 engineers, review code, establish data engineering standards, CI/CD, and monitoring (e.g., Cloud Monitoring, Logging). Performance & Cost: Optimize pipeline performance, query efficiency, and GCP cost management (e.g., slot reservations, partitioning). Collaboration: Work with data architects, scientists, and analysts to deliver self-service data platforms and support advanced analytics/ML use cases. Data Quality & Governance: Implement data validation, lineage, and metadata management (e.g., Data Catalog, dbt).

What you’ll do

Lead a team of 10 engineers while building and optimizing data pipelines and managing GCP data infrastructure, including BigQuery, Cloud Storage, and Data Fusion. Establish engineering standards and monitoring, optimize performance and costs, and collaborate on self-service data platforms, analytics, machine learning, data quality, and governance.

Requirements

Candidates must have experience leading onshore and offshore data engineering teams; this is a lead role, and leadership experience is mandatory. Required technical expertise includes SQL, Python, Java or Scala, distributed systems, GCP data services, Terraform, and CI/CD; GCP certification and experience with real-time processing, Vertex AI, or multi-cloud environments are preferred.

Listed skills

• SQL · Preferred
• CI/CD · Preferred
• Team Leadership · Preferred
• Java · Preferred
• Terraform · Preferred
• Python · Preferred

Other relevant skills

Identified from the job description. Confirm important requirements above.

• Team Leadership
• SQL
• Python
• Java
• Scala
• Distributed Systems
• Apache Spark
• Apache Beam
• BigQuery
• Dataflow
• Dataproc
• Pub/Sub
• Cloud Composer
• Terraform
• CI/CD
• ETL/ELT

Job areas

• Data & Analytics
• Technology
• Software
• Management & Leadership
• Engineering

More jobs from J&M Group