Principal Software Engineer, Distributed Systems

Background

We are hiring a Staff / Principal Site Reliability Engineer focused on distributed systems and platform reliability . This is not a traditional DevOps role.

You will own the architecture and engineering practices that ensure Sigma workflows execute reliably even when individual components fail.

Processes crash. Networks become unavailable. APIs time out. Workers hang. Messages may be delivered more than once. Databases experience contention. Kubernetes pods restart.

Your job is to design Sigma so those conditions are expected, detected, and automatically recovered from.

The fundamental principle is simple:

An accepted Sigma job may fail, but it should never disappear.

You will work closely with engineering leadership to evolve Sigma's execution architecture, improve production reliability, establish observability standards, and eliminate classes of distributed-system failures before they affect customers.

What You'll Own

Design and improve the systems responsible for executing long-running and scheduled infrastructure workflows.

Build patterns for:

Durable workflow execution

Idempotent operations

Automatic retries and exponential backoff

Worker leases and heartbeats

Stuck-job detection and recovery

Workflow timeouts and cancellation

Failure isolation

Dead-letter handling

Checkpointing and workflow resumption

Concurrency control

Distributed locking and lease management

Reconciliation loops

Graceful degradation during dependency failures

Evaluate and implement workflow technologies such as Temporal or equivalent systems where appropriate.

Messaging and Event Infrastructure

Own and improve Sigma's asynchronous execution architecture.

Work extensively with technologies such as:

Kafka

Distributed consumers

Consumer groups

Message delivery semantics

Partitioning

Consumer lag

Backpressure

Retry strategies

Event ordering

Duplicate message handling

Poison-message isolation

Ensure event-processing failures cannot silently cause workflows to stop progressing.

Database Reliability

Help design application patterns that remain reliable under high concurrency.

Deeply understand and troubleshoot:

PostgreSQL transactions

Row and advisory locking

Deadlocks

Lock contention

Transaction isolation

Connection pooling

Long-running transactions

Query performance

Database failover

Schema and migration safety

Partner with application engineers to identify and eliminate patterns that create production contention or reliability risks.

Kubernetes and Platform Reliability

Improve the reliability of Sigma deployments running in containerized environments.

Responsibilities include:

Kubernetes workload architecture

Pod lifecycle and failure recovery

Resource limits and capacity planning

Autoscaling

Health checks

Graceful shutdown

Rolling deployments

High availability

Infrastructure-as-code

Disaster recovery

Backup and restore validation

Design systems that tolerate node, pod, process, and dependency failures without losing customer work.

Observability

Make the state of Sigma understandable in real time.

Build and improve:

Metrics

Distributed tracing

Structured logging

Dashboards

Alerting

Workflow-level telemetry

Kafka consumer monitoring

Database performance monitoring

Worker health monitoring

Queue depth and backlog monitoring

Technologies may include:

OpenTelemetry

Grafana

Prometheus

Sentry

CloudWatch

Engineers should be able to answer questions such as:

Why is this workflow still running?

Which worker owns this task?

When did it last make progress?

Which dependency is causing the delay?

Did this operation execute once or multiple times?

Can the workflow safely resume?

Are scheduled jobs starting when expected?

Reliability Engineering

Help establish engineering practices expected of mature enterprise platforms.

This includes:

Service-level objectives and indicators

Error budgets

Capacity planning

Load testing

Failure-mode testing

Chaos testing

Incident response

Root-cause analysis

Production readiness reviews

Runbooks

Automated recovery

Disaster-recovery testing

Move Sigma from detecting incidents to preventing and automatically recovering from them.

What We're Looking For

You have significant experience operating production distributed systems where reliability matters.

Strong candidates will have deep experience with several of the following:

Kubernetes

Kafka or similar event-streaming systems

PostgreSQL

Distributed systems

Asynchronous worker architectures

Workflow orchestration

Temporal, Cadence, Step Functions, Conductor, or similar systems

Python backend systems

Cloud infrastructure

Infrastructure-as-code

Observability platforms

High-availability architectures

You understand concepts such as:

At-least-once delivery

Idempotency

Leases and fencing tokens

Distributed locks

Leader election

Retry semantics

Backpressure

Eventual consistency

Transaction boundaries

Failure domains

Reconciliation

Split-brain scenarios

Distributed tracing

Most importantly, you have personally diagnosed and improved production systems experiencing real distributed-system failures.

What Success Looks Like

Within your first several months, you will help Sigma establish an architecture where:

Scheduled workflows reliably begin when expected

Accepted work cannot silently disappear

Worker crashes do not result in lost workflows

Infrastructure failures automatically recover where possible

Long-running workflows can safely resume

Duplicate execution is prevented or safely handled

Stuck workflows are automatically detected

Database and messaging contention is visible before becoming an outage

Engineers can trace a workflow from request through every execution step

Customer environments can scale without requiring manual infrastructure babysitting