Staff Software Engineer - Lakehouse

At eBay, we're more than a global ecommerce leader — we’re changing the way the world shops and sells. Our platform empowers millions of buyers and sellers in more than 190 markets around the world. We’re committed to pushing boundaries and leaving our mark as we reinvent the future of ecommerce for enthusiasts. Our customers are our compass, authenticity thrives, bold ideas are welcome, and everyone can bring their unique selves to work — every day. We're in this together, sustaining the future of our customers, our company, and our planet. Join a team of passionate thinkers, innovators, and dreamers — and help us connect people and build communities to create economic opportunity for all. About the Team and the Role: We are part of the Analytics Data Platform Team within eBay, working with a mission towards improving availability, security, performance, tools and other features within Analytics Platform. We are seeking an experienced Software Engineer with a strong background in Spark/Lakehouse table format technologies, responsible to build and optimize our next-generation lakehouse platform. What You Will Accomplish: * Our focus is on enhancing Spark stability/usability/performance and extending Iceberg's capabilities. * Additionally, we are building internal storage and metadata services to power our lakehouse architecture. * Troubleshoot and resolve production issues, ensuring minimal downtime. * Maintain and improve system monitoring, logging, and alerting. What You Will Bring: * 10+ years of hands on experience Building Software with a focus on strong programming skills in either Java or Scala. * Comprehensive knowledge of Apache Spark’s internals, including Catalyst optimizer, Tungsten execution engine, shuffle mechanics, AQE, and memory management. * Solid understanding of file formats like Apache Parquet and table formats such as Apache Iceberg, Delta or Apache Hudi internals, including snapshots, manifests, partition specs, and table metadata. * Familiarity with Linux/Unix systems, including shell scripting and system commands. * Understanding of networking principles, as Hadoop often operates in distributed environments. Overall, the candidate is expected to have analytical skills to tackle complex distributed system challenges and optimize solutions. * Excellent problem solving and analytical skill on both defect and performance issue resolution. Nice to Have: * Contribution to related open source projects like Apache Spark/Iceberg. * Experience in designing and implementing distributed systems. * Good understanding of other Hadoop related projects including HDFS/HBase/Yarn. #LI-Hybrid Additional Details This job posting relates to an existing vacancy within eBay. eBay is an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, national origin, sex, sexual orientation, gender identity, and disability, or other legally protected status. If you have a need that requires accommodation, please contact us at talent@ebay.com [talent@ebay.com]. We will make every effort to respond to your request for accommodation as soon as possible. View our accessibility statement [https://ebayinc.com/accessibility/] to learn more about eBay's commitment to ensuring digital accessibility. We use cookies to enhance your experience and may use AI tools for administrative tasks in the hiring process. To learn how we handle your personal data and use AI responsibly, please visit our Talent Privacy Notice [https://jobs.ebayinc.com/us/en/talent-privacy-notice], Privacy Center [https://www.ebayinc.com/company/privacy-center/], and AI Hiring Guidelines [https://jobs.ebayinc.com/us/en/how-we-hire].

The role involves building and optimizing the next-generation lakehouse platform, focusing on enhancing Spark stability, usability, and performance, and extending Iceberg's capabilities. Responsibilities also include troubleshooting production issues and maintaining system monitoring, logging, and alerting.

Candidates must have over 10 years of hands-on software building experience, focusing on strong programming skills in Java or Scala, coupled with comprehensive knowledge of Apache Spark internals and table formats like Iceberg, Delta, or Hudi. Analytical skills for complex distributed system challenges are essential.

Identified from the job description. Confirm important requirements above.

• Spark
• Lakehouse
• Java
• Scala
• Apache Spark Internals
• Catalyst Optimizer
• Tungsten Execution Engine
• Shuffle Mechanics
• AQE
• Memory Management
• Apache Parquet
• Apache Iceberg
• Delta
• Apache Hudi
• Linux/Unix
• Shell Scripting

• Technology
• Data & Analytics
• Software
• Engineering

Minimum experience

10+ years

Posting language

English

Working hours

40 hours per week

Location requirements

Country, Canada