Data Mining Engineer, Software

Job summary

Build and operate infrastructure for discovering, acquiring, and processing public-sector information at scale from websites, APIs, documents, procurement platforms, and legacy systems. Develop and maintain scraper fleets and reliable data pipelines, including orchestration, monitoring, failure recovery, data-quality checks, and reusable tools for onboarding new sources.

Job details

A fast growing, series A start up is hiring for a Software, Data Mining Engineer to join their growing team. They are focused on building the data and intelligence layer for the public sector. Its platform turns fragmented information from government websites, procurement systems, budgets, meeting records and public documents into actionable intelligence for businesses selling to government. You will be responsible for owning the infrastructure that discovers, acquires and processes public-sector information at scale. What You’ll Do: Build systems for creating, deploying, scheduling, monitoring and maintaining large fleets of scrapers. Acquire information from websites, APIs, PDFs, spreadsheets, procurement platforms and legacy systems. Develop orchestration, concurrency, retries, backfills, change detection and failure recovery. Monitor data quality and identify broken sources or incorrect extraction. Build abstractions that make new sources easier and cheaper to onboard. Relevant experience: Web crawling, scraping or browser automation. Large-scale external data ingestion and aggregation. Distributed systems and reliable data pipelines. Alternative data, search infrastructure, knowledge graphs or document processing. Data quality, monitoring and operational reliability. What We’re Looking For: 2-6+ years of software and data engineering experience Strong software engineering ability, production systems experience and the autonomy to define an approach from a broad business objective Strong experience with Python/Go, ETL, React, Docker, Kuberenetes Experience working with LLMs and ai agents are a plus Bachelors in Computer Science, Mathematics, Statistics or similar

What you’ll do

Build and operate infrastructure for discovering, acquiring, and processing public-sector information at scale from websites, APIs, documents, procurement platforms, and legacy systems. Develop and maintain scraper fleets and reliable data pipelines, including orchestration, monitoring, failure recovery, data-quality checks, and reusable tools for onboarding new sources.

Requirements

The role seeks 2–6+ years of software and data engineering experience, strong software engineering skills, and experience operating production systems with autonomy. Relevant experience includes Python or Go, ETL, React, Docker, Kubernetes, and data ingestion; a bachelor's degree in computer science, mathematics, statistics, or a similar field is listed, while LLM and AI agent experience is a plus.

Listed skills

• Kubernetes · Preferred
• Go · Preferred
• Docker · Preferred
• React · Preferred
• Python · Preferred

Other relevant skills

Identified from the job description. Confirm important requirements above.

• Python
• Go
• Web Crawling
• Web Scraping
• Browser Automation
• Data Ingestion
• ETL
• Distributed Systems
• Data Pipelines
• Docker
• Kubernetes
• React
• Data Quality Monitoring
• Orchestration
• Document Processing
• LLMs and AI Agents

Job areas

• Software
• Technology
• Data & Analytics
• Government & Public Sector

More jobs from arcadia