Senior Network Production Engineer, AI Supercomputing

Own Production Network Health Own the availability, performance, and operational readiness of the MRC Ethernet backend network. Define and operate service-level indicators for packet loss, congestion, link health, path diversity, bandwidth, tail latency, collective performance, and job impact. Detect slow or degraded components before they cause training failures or reduce model FLOPs utilization. Build health models that correlate switch, NIC, host, topology, and application telemetry. Operate the Network at Frontier Scale Participate in on-call rotation and lead the response to high-severity network incidents. Diagnose failures spanning GPUs, NICs, switches, cables, firmware, drivers, network operating systems, MRC, NCCL, Kubernetes, and training workloads. Develop automated mitigation mechanisms, including path avoidance, node quarantine, workload relocation, and safe component remediation. Improve Large-Job Reliability and Performance Work directly with training teams to investigate collective-performance degradation, stalls, timeouts, job restarts, and unexplained MFU loss. Establish network qualification gates for admitting nodes and racks into large training pools. Improve placement and routing policies for jobs spanning racks, rows, and network planes. Build production-quality software and automation for network validation, monitoring, diagnosis, remediation, and fleet-wide change management. Automate firmware, driver, configuration, and network operating-system compliance checks. Establish clear operational ownership and measurable service levels for the MRC backend network. Build end-to-end visibility from physical links and network paths to collective performance and training-job health. Significant experience operating large-scale datacenter or high-performance computing networks in production. Deep understanding of Ethernet, RDMA, RoCEv2, congestion control, routing, load balancing, and lossless or near-lossless network design. Experience debugging complex failures across switches, NICs, hosts, drivers, firmware, and distributed applications. Strong Linux systems knowledge and proficiency in Python, Go, C++, or another systems-oriented programming language. Experience building monitoring, automation, diagnostic, or remediation systems for production infrastructure. Demonstrated ability to lead high-severity incidents and coordinate resolution across multiple engineering teams. Strong analytical skills and the ability to translate telemetry into a defensible root cause. Willingness to participate in a global 24/7 on-call rotation. Experience operating GPU training clusters or other tightly coupled distributed-computing systems. Knowledge of MRC or other multi-rail, multi-plane, multipath Ethernet transports. Experience with NCCL, collective communications, CUDA, GPUDirect RDMA, and distributed training frameworks. Familiarity with ConnectX-class NICs, modern Ethernet switch ASICs, SONiC, SAI, and switch telemetry. Experience with InfiniBand and the operational differences between InfiniBand and Ethernet-based AI fabrics. Familiarity with Kubernetes, Slurm, workload placement, and cluster-health certification. Experience with streaming telemetry, gNMI, Prometheus, Datadog, Kusto, eBPF, or equivalent observability systems. Experience managing fleet-wide firmware, driver, or network operating-system changes. A track record of improving reliability through automation rather than recurring manual intervention.