Who you are

• Have primary experience running PostgreSQL in high-growth, large production environments using both self-managed (VM, Kubernetes with modern PostgreSQL Operators) as well DBaaS services
• Have hands-on experience using data from PostgreSQL internals to design, build and troubleshoot systems
• Have primary experience with infrastructure automation, orchestration and configuration management (Chef, Ansible, Puppet, Terraform)
• Have solid understanding of SQL and PL/pgSQL
• Significant experience working in a Large SaaS distributed Systems production environment
• Share our values, and work in accordance with those values
• Have excellent written and verbal English communication skills, with an urge to collaborate and communicate asynchronously
• Have an urge to document all the things so you don't need to learn the same thing twice, and an urge for delivering quickly and iterating fast
• Have a proactive, go-for-it attitude. When you see something broken, you can't help but fix it
• Solid data modeling and data structure design skills
• Bonus: Solid programming skills as a (former) backend engineer - Preferably with Ruby and/or Go
• Bonus: Experience with Clickhouse, or other modern OLAP database
• Expertise in at least 1 area of SRE work, with general knowledge of all areas
• Capable of mentoring Junior team members
• Contributes small improvements to the GitLab codebase to resolve issues

What the job involves

• Site Reliability Engineers (SREs) are responsible for keeping all user-facing services and other GitLab production systems running smoothly
• SREs are a blend of pragmatic operators and software craftspeople that apply sound engineering principles, operational discipline, and mature automation to our environments and the GitLab codebase
• We specialize in systems, whether it be networking, the Linux kernel, or some more specific interest in scaling, algorithms, or distributed systems
• The Database Operations team’s mission is to build, run, own and evolve the entire lifecycle of the PostgreSQL database engine for GitLab.com
• The team is focused on owning the reliability, scalability, evolution, performance & security of the database engine and its supporting services
• The team should be seeking to build their services on top of Reliability::Foundations services and cloud vendor managed products, where appropriate, to reduce complexity, improve efficiency and deliver new capabilities quicker
• Automating every operational task is a core requirement for this role. For example, package updates, configuration changes across all environments, creating tools for automatic provisioning of user facing services, etc
• Responding to platform emergencies, alerts, and escalations from Customer Support
• Ensure systems exist to manage software life-cycles (e.g. Operating Systems) with a minimum of manual effort
• Develop a fully automated multi-environment observability stack based on the existing SaaS system, and extend it to predict capacity needs based on the usage patterns
• Plan for new service roll-outs, expansion and capacity management of existing services, and work with users to optimize their resource consumption
• Work on database reliability and performance aspects for GitLab.com from within the SRE team as well as work on shipping solutions with the product
• Analyze solutions and implement best practices for our PostgreSQL database clusters and its components
• Work on observability of relevant database metrics and make sure we reach our database objectives
• Work with peer SREs to roll out changes to our production environment and help mitigate database-related production incidents
• OnCall support on rotation with the team
• Provide database expertise to engineering teams (for example through reviews of database migrations, queries and performance optimizations)
• Work on automation of database infrastructure and help engineering succeed by providing self-service tools
• Use the GitLab product to run GitLab.com as a first resort and improve the product as much as possible
• Plan the growth of GitLab's database infrastructure
• Design, build and maintain core database infrastructure components that allow GitLab to scale to support hundreds of thousands of concurrent users
• Support and debug database production issues across services and levels of the stack
• Make monitoring and alerting alert on symptoms and not on outages
• Document every action so your learnings turn into repeatable actions and then into automation
• Review, analyze and implement solutions regarding database administration (e.g., backups, performance tuning)
• Work with Ansible, Terraform, Chef and other tools to build mature automation (automate setup of new replicas or testing and monitoring of backups)
• Implement self-service tools for our engineers using GitLab ChatOps
• Provide technical assistance and support to other teams on database and database-related application design methodologies, system resources, application tuning
• Review database related changes from engineering teams (e.g., database migrations)
• Recommend query and schema changes to optimize the performance of database queries
• Jump on a production incident to mitigate database-related issues on GitLab.com
• Participate actively in the infrastructure design and scalability considerations focusing on data storage aspects
• Make sure we know how to take the next step to scale the database
• Design and develop specifications for future database requirements including enhancements, upgrades, and capacity planning; evaluate alternatives; and make appropriate recommendations
• Identifies projects that result in substantial cost savings or revenue
• Identifies changes for the product architecture from the reliability, performance and availability perspective with a data driven approach
• Proactively work on the efficiency and capacity planning to set clear requirements and reduce the system resources usage to make GitLab cheaper to run for all our customers
• Identify parts of the system that do not scale, provides immediate palliative measures and drives long term resolution of these incidents
• Identify Service Level Indicators (SLIs) that will align the team to meet the availability and latency objectives
• Ability to thrive in a fully remote, asynchronous work environment that places a high emphasis on documentation and written communication
• Develop expertise in a domain and radiate that knowledge
• Participate in blameless RCAs on incidents and outages, looking for answers that will prevent the incident from ever happening again
• Lead Junior SREs by setting the example
• Develop ownership of a major part of the infrastructure
• Trusted to de-escalate conflicts inside the team

Benefits

• We offer benefits to manage your health, wealth, and well-being regardless of location
• Flexibility in schedule to be there for life’s important moments
• Equity compensation & Employee Stock Purchase Plan offered
• Generous Paid Time Off