Infrastructure Platform Engineer
Remote
Team: Infrastructure / Platform Engineering
$5,000 - $6,100/month USD
Role Overview
As an Infrastructure Engineer, you will own the GPU platform that runs production inference, specifically:
- Cluster architecture
- Deployment reliability
- Observability
- Capacity management
- Incident response mechanisms
Your job is to make the platform predictable and reliable — even as we scale hardware, models, tenants, and traffic patterns. You'll work closely with serving/runtime and gateway teams to ensure the platform enforces the right isolation, exposes the right telemetry, and supports safe changes without downtime. This role blends strong systems intuition with real production discipline: reliable rollouts, clean operational tooling, and fast incident response.
Responsibilities
- Own GPU cluster architecture and operations: Provisioning, node images, driver/runtime lifecycle, GPU plugin/operator lifecycle, and standardized deployment patterns for serving pools and system services.
- Define and maintain the production baseline: Golden node configurations, cluster hardening, upgrade paths, and known good compatibility matrices (drivers, CUDA, runtime, kernel).
- Build reliability into the platform: Includes SLOs/SLIs, alerting quality, runbooks, incident tooling, and postmortems with real follow-through (automation, guardrails, and elimination of repeat incidents).
- Enable safe delivery: Canary deploys, progressive rollouts, rollback paths, and configuration safety (validation, guardrails, change controls, and safe defaults).
- Own fleet health and maintenance workflows: Node draining, GPU quarantining, automated remediation, scheduled maintenance, and safe break-glass procedures with auditability.
- Capacity and utilization: Scheduling constraints, binpacking/fragmentation management, warm pools, autoscaling primitives, and quota enforcement hooks that align with product tiers and fairness goals.
- Observability: Metrics/logs/tracing across gateway, serving, GPU; latency breakdowns, saturation signals, queue depth, GPU memory/compute metrics, and fleet health dashboards that help correlate customer symptoms to root causes.
- Production readiness for heterogeneous environments: Manage differences across hardware generations and evolving server platforms, minimizing reliability risk while improving utilization.
- Security baseline: Secrets management, least-privilege access, audit trails for operator actions, and secure operational workflows.
- Partner with networking: Topology, failure domains, load balancing, and performance-sensitive traffic paths that impact tail latency and availability.
- Build operational tooling: Fleet management, debugging workflows, safe admin actions, capacity tooling, and maintenance automation that reduces MTTR and improves operator efficiency.
- Collaborate across teams: Align rollout plans, health semantics, capacity signals, and failure handling so the entire platform behaves predictably under load.
Requirements
- 5+ years in infrastructure/SRE/platform engineering for production distributed systems.
- Strong Kubernetes experience in production (or equivalent orchestration), with real ops ownership.
- Experience operating GPU clusters or other high-performance compute fleets (or similarly performance-sensitive infrastructure).
- Strong debugging skills across Linux, networking, and distributed systems failure modes.
- Strong operational discipline: automation-first mindset, measurable reliability, careful change management, clear communication during incidents.
- Willing to participate in an on-call rotation for owned systems.
Nice to Have
- Experience with high-throughput gateways/service meshes (e.g., Envoy), OpenTelemetry, and multi-region architectures.
- Experience with Slurm/HPC-style scheduling, RDMA/IB, or performance-sensitive networking.
- Experience building internal developer platforms and golden paths for consistent deploy/rollback workflows.
- Experience managing GPU driver/runtime upgrades safely across a fleet (compatibility testing + staged rollouts).
- Familiarity with observability patterns for latency-sensitive systems (request correlation, sampling strategy, high-cardinality metrics control).
