Senior ML/Platform Engineer, Bailey AI

What This Role Is

Bailey is a clinical AI system that orchestrates LLMs, manages health safety evaluation, and integrates with clinical data to help people navigate their health. Three data scientists build it. Nobody currently owns what happens after something works.

This role owns the path from prototype to production and the infrastructure that tells us whether production is working correctly. You'll build deployment pipelines, evaluation frameworks, monitoring systems, and the observability layer that makes a clinical AI system trustworthy, not just functional.

This is not a role where you maintain someone else's platform. You'll design and build it. The team is small enough that your decisions shape the system's architecture directly. Your output is running systems, not notebooks or papers. You need to understand what you're deploying, not just how to deploy it. Much of this work hasn't been defined yet; the evaluation framework exists as a strategy document, the monitoring system exists as a gap, and you turn those into infrastructure.

What You'll Actually Do

We're looking for a fast ramp here — this role is senior specifically because it operates without a playbook, and that means starting to produce quickly, not spending a quarter getting oriented.

In your first 30 days:

• Deploy and version ML artifacts that currently move to production ad hoc
• Stand up observability for the existing LangGraph agent orchestration (what's happening, how often, how well)
• Instrument the health safety evaluator so its 5-dimension scoring is monitored continuously, not checked manually

Within 3 months:

• Build the evaluation infrastructure described in our product strategy: drift detection, distributional monitoring, safety-weighted quality scoring
• Own CI/CD for ML artifacts and agent configurations
• Harden existing services (conversation persistence, MCP tool orchestration, clinical skills framework) for production reliability
• Get up to speed on Bailey's evaluation documentation and use it to shape how we test and monitor Bailey's behavior going forward, including flagging where traditional ML methods (not just LLM-based approaches) could deliver the same outcome more efficiently

Ongoing:

• Be the person who knows whether the system is healthy, degrading, or broken before users or clinicians notice
• Collaborate with data scientists so their work reaches production without a handoff gap
• Keep testing and monitoring aligned with Bailey's evolving evaluation documentation, and keep surfacing opportunities to swap in traditional ML where it's more efficient than an LLM-based approach
• Evolve the platform as the product grows (voice, memory, multimodal are on the roadmap)

What You Bring

These are real requirements, the things that would be hard to learn on the job at the pace this role demands:

• You've built production ML infrastructure. Not just used it. Deployment pipelines, serving systems, model tracking/registries (MLflow or equivalent), monitoring that pages you at 2am. You know the difference between a model that works in a notebook and one that works in production.
• You're a strong software engineer first. Distributed systems, containerization, event-driven architectures, CI/CD. ML is the domain; engineering is the craft.
• You're comfortable owning something end-to-end. No one will hand you a spec and check your work weekly. You'll identify what needs to exist, propose how to build it, and ship it.
• You work well in small teams. You communicate clearly, you surface problems early, and you're comfortable with the visibility and accountability that comes with a four-person team.

What Would Make You Particularly Effective Here

These aren't requirements. They're things that would shorten your ramp or deepen your impact:

• Experience in healthcare or regulated industries (you understand why audit trails and compliance aren't afterthoughts)
• Background in ML research or applied ML (you can have informed opinions about model behavior, not just model infrastructure)
• Familiarity with AWS ML services like Bedrock and SageMaker, which are part of our current stack
• Experience with LLM orchestration frameworks (LangChain, LangGraph, or similar)
• Exposure to clinical data standards (FHIR, HL7) or health information systems
• Publications, open-source contributions, or side projects that show you go deeper than your day job requires

About the Team

You'll join a small data science team that builds Bailey's AI capabilities: models, integrations, clinical skills, and orchestration. The team works closely with engineering and product. You'll be the first person dedicated to production infrastructure and evaluation systems, which means you'll shape how those concerns are handled from the ground up.

Apply for this Position