Site Reliability Engineer, AI & HPC Infrastructure

Palo Alto, CA, United States

Site Reliability Engineer, AI & HPC Infrastructure Tesla's Supercomputing/AI infrastructure team works directly with the high-performance computing and machine learning infrastructure on which our ML algorithms run; this includes virtual simulations, Autopilot hardware, silicon design, and Dojo. With the rapidly-growing need for more data and optimized compute resources, cluster builds are getting larger and increasingly complex. Continued development/automation of deployment, monitoring, self-healing and alerting processes is imperative to the success of our engineering groups. As the scope and impact of our Full-Self-Driving (FSD) & Robotaxi efforts continue to scale, so does the value of this team and its work. As a Site Reliability Engineer, you will be responsible for maintaining and improving our infrastructure to ensure engineering teams across Autopilot/AI & Dojo have the necessary tools and resources to be productive. This includes managing/operating our HPC clusters, monitoring compute/GPU/network metrics, Linux troubleshooting & performance tuning, and collaborating with our Data Center team to coordinate the smooth operation of hundreds of servers & bring up new GPU capacity. Your work will directly facilitate neural network training at scale, streamline FSD development, and enable Dojo to become the most powerful supercomputer to date.

What to Expect Tesla's Supercomputing/AI infrastructure team works directly with the high-performance computing and machine learning infrastructure on which our ML algorithms run; this includes virtual simulations, Autopilot hardware, silicon design, and Dojo. With the rapidly-growing need for more data and optimized compute resources, cluster builds are getting larger and increasingly complex. Continued development/automation of deployment, monitoring, self-healing and alerting processes is imperative to the success of our engineering groups. As the scope and impact of our Full-Self-Driving (FSD) & Robotaxi efforts continue to scale, so does the value of this team and its work. As a Site Reliability Engineer, you will be responsible for maintaining and improving our infrastructure to ensure engineering teams across Autopilot/AI & Dojo have the necessary tools and resources to be productive. This includes managing/operating our HPC clusters, monitoring compute/GPU/network metrics, Linux troubleshooting & performance tuning, and collaborating with our Data Center team to coordinate the smooth operation of hundreds of servers & bring up new GPU capacity. Your work will directly facilitate neural network training at scale, streamline FSD development, and enable Dojo to become the most powerful supercomputer to date.

What You’ll Do Support the AI/ML cluster infrastructure on both GPU and Dojo platforms, focusing on systems automation, configuration management and deployment at scale

Improve our cluster health monitoring and auto-recovery pipeline

Work with users on debugging application performance issues

Work with hardware and storage vendors to tune and optimize our servers, storage and network

Write Ansible playbooks for configuration management

Performance tuning and OS provisioning on Linux systems

Manage HPC clusters, workloads and applications

Automation and systems engineering in Python, Golang or Bash/Shell

Participate in 24x7 on-call rotation

What You’ll Bring Proficiency in Python, Golang and/or Bash

Proficiency with Linux fundamentals and performance optimizations (Ubuntu/RHEL OS)

Demonstrable knowledge of TCP/IP, IPoIB, Linux operating system internals, filesystems, disk/storage technologies and storage protocols

Experience collaborating with network and data center teams for large scale cluster builds

Experience with configuration management software (Ansible, etc.), systems monitoring & alerting (Prometheus, Grafana, Telegraf, Splunk, etc.), and/or administering HPC workload managers

Experience with high-throughput low-latency networks, GPU-based computing systems, and/or high performance storage systems

Experience with Slurm, LSF and storage management of distributed parallel file systems a plus

Bachelor's Degree in Computer Science, Computer Engineering, Electrical Engineering, Physics or proof of exceptional skills in related field

3+ years of additional equivalent experience or evidence of exceptional ability related to the position

Compensation and Benefits Benefits

Along with competitive pay, as a full-time Tesla employee, you are eligible for the following benefits at day 1 of hire:

Aetna PPO and HSA plans > 2 medical plan options with $0 payroll deduction

Family-building, fertility, adoption and surrogacy benefits

Dental (including orthodontic coverage) and vision plans, both have options with a $0 paycheck contribution

Company Paid (Health Savings Account) HSA Contribution when enrolled in the High Deductible Aetna medical plan with HSA

Healthcare and Dependent Care Flexible Spending Accounts (FSA)

LGBTQ+ care concierge services

401(k) with employer match, Employee Stock Purchase Plans, and other financial benefits

Company paid Basic Life, AD&D, short-term and long-term disability insurance

Employee Assistance Program

Sick and Vacation time (Flex time for salary positions), and Paid Holidays

Back-up childcare and parenting support resources

Voluntary benefits to include: critical illness, hospital indemnity, accident insurance, theft & legal services, and pet insurance

Weight Loss and Tobacco Cessation Programs

Tesla Babies program

Commuter benefits

Employee discounts and perks program

Expected Compensation

$104,000 - $348,000/annual salary + cash and stock awards + benefits

Pay offered may vary depending on multiple individualized factors, including market location, job-related knowledge, skills, and experience. The total compensation package for this position may also include other elements dependent on the position offered. Details of participation in these benefit plans will be provided if an employee receives an offer of employment.

Tesla is an Equal Opportunity / Affirmative Action employer committed to diversity in the workplace. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, age, national origin, disability, protected veteran status, gender identity or any other factor protected by applicable federal, state or local laws.

Tesla is also committed to working with and providing reasonable accommodations to individuals with disabilities. Please let your recruiter know if you need an accommodation at any point during the interview process.

Privacy is a top priority for Tesla. We build it into our products and view it as an essential part of our business. To understand more about the data we collect and process as part of your application, please view our Tesla Talent Privacy Notice

#J-18808-Ljbffr