Site Reliability Engineer
Overview
We are hiring a Site Reliability Engineer (SRE) for a new, fully remote SRE team at a fast-growing healthcare technology company that changes how health systems coordinate and deliver care through secure, modern technology. The team works across engineering, product operations and security to make production systems measurable, observable and recoverable. Its scope covers monitoring and alerting, incident response, availability reporting and resilience testing.
As a mid-level engineer, you build and run monitoring, dashboards, runbooks and automation with guidance from senior engineers. You join the on-call rotation and grow into incident command. You also help build the team's practices from the start, setting the first availability targets and defining how they're measured, making sure the right team gets each alert, and turning incident follow-ups into real fixes. You'll see the results in less on-call noise, faster detection, shorter recovery times, and availability numbers that customers and auditors can trust.
The role overlaps with DevOps and platform engineering. You work in AWS and Kubernetes, write infrastructure as code, and build automation. The difference is that you own production reliability and the incident response practice first. Strong software development skills are a must, because much of the job is replacing manual, recurring operational work with code.
This is a contract-to-hire role. You join as a W-2 employee of Arsenal AI working full-time on the client's team, with a path to a full-time role with the client.
Arsenal AI is a Pittsburgh-based venture studio and AI consulting firm. We build and invest in AI-native companies, and we embed engineering teams directly with our clients to ship production software.
US Citizenship is required for this position.
What You'll Do
Monitoring & Alerting
Own the reliability of monitoring and alerting. Make sure alerts fire, reach the right team, and are themselves monitored.
SLOs & Availability Reporting
Define and maintain SLIs, SLOs and availability targets, and report uptime and incident metrics such as MTTD, MTTR, paging volume, and how many postmortem action items get closed.
Incident Response
Build and run the incident response practice, including severity definitions, the incident commander role, tabletop exercises, postmortems, and tracking action items to closure. Work with Security on external and regulatory reporting.
Alert Quality
Improve alert quality continuously. Broaden error coverage, tune and remove duplicate noisy alerts, and give every paging alert a runbook.
Dashboards & Detection
Build dashboards and detection for system health and saturation across databases, workflow engines, message queues and logging infrastructure.
Recovery Testing
Plan, run, time and document backup and restore, failover, and maintenance-window procedures that meet audited recovery objectives.
Deployment Safety
Improve deployment safety with post-deploy health checks, gates after failed deployments, and automated rollback.
Automation
Write code that reduces operational work, such as internal tools for secret rotation, platform upgrades and recurring compliance tasks, in Go, TypeScript and Pulumi.
Operational Cadence
Turn recurring operational obligations into scheduled, visible practices that keep running during a busy week.
Documentation
Write and maintain runbooks, incident response plans and disaster recovery documentation.
Team Practices
Take part in code reviews, design sessions, all sprint ceremonies, and an on-call rotation.
Required Qualifications
- •Experience: 2–5 years of experience in SRE, production operations, DevOps or platform engineering, including on-call rotations. AWS experience is preferred.
- •Education: A bachelor's degree in computer science, software engineering, computer engineering or a related field, or equivalent practical experience.
- •Software Development: Strong development skills, with hands-on Go and/or TypeScript (the client's primary languages), plus scripting in Bash and/or Python.
- •Observability & Alerting: Grafana/LGTM (Mimir, Loki, Tempo), Prometheus/PromQL or Splunk, including designing alerts people can act on and reducing alert noise.
- •Incident Management: Serving as or supporting an incident commander, applying a severity model, and running blameless postmortems.
- •SLIs & SLOs: Defining them and reporting availability to stakeholders.
- •AWS: Core services, including ECS/Fargate, RDS, EBS, CloudWatch, IAM, Lambda, and SSM/Secrets Manager.
- •Kubernetes: Operating workloads, ideally on AWS EKS, including stateful services.
- •Infrastructure as Code: Ideally Pulumi. Terraform, CDK or CloudFormation are also fine.
- •Linux, Git & GitHub: Linux administration and troubleshooting, plus version control with Git and GitHub.
- •Communication: Strong verbal and written skills, including runbooks and incident documentation that hold up under pressure.
- •Location: Fully remote within the continental U.S., in the Eastern or Central time zone (excluding New York), with high-speed internet.
Desired Qualifications
- •ServiceNow: ServiceNow ITSM/CSM, including major incident and outage records, and assignment group rules.
- •Recovery Testing: Backup and restore, disaster recovery, and failover testing against RTO/RPO targets.
- •Monitoring: Synthetic monitoring, uptime monitoring and public status pages, plus anomaly detection, statistical alerting, and capacity and saturation monitoring.
- •Deployment Safety: Automated rollback, post-deploy verification and deployment gates. Jenkins experience is a plus.
- •On-Call Programs: Building one, including rotations, shadowing, onboarding and runbook coverage.
- •Automation: Automating secret rotation and other recurring procedures with least-privilege access.
- •Compliance: FedRAMP and/or HIPAA environments, including contingency planning and incident reporting.
- •Certifications & Process: AWS certifications (SysOps Administrator, DevOps Engineer, Solutions Architect) and Agile experience.
Eligibility Requirements
This engagement supports a U.S. Department of Veterans Affairs contract. These are federal contract requirements, and candidates must confirm they meet all three before applying.
- •U.S. citizenship.
- •A REAL ID-compliant driver's license and a valid U.S. passport, for federal identity proofing.
- •Successful completion of a background check and drug screening before your start date.
Arsenal AI is an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, national origin, age, disability, or any other legally protected status.
Ready to Apply?
Fill out the form below to submit your application.