LLM Inference Engineer (Mid, Sr, Staff)
Not mentioned in the posting. Hippocratic AI filed 23 H-1B, H-1B1 or E-3 applications for AI roles in FY2025–FY2026, 6 for workers changing employer. Its US visa filings
In Hippocratic AI's visa applications for similar roles: $260K – $350K (the middle half of 5 base wages offered, FY2025–FY2026). Not this posting's range.
ROLE MISSION
As HAI's LLM Inference Engineer, you will own the serving infrastructure that determines whether our breakthrough healthcare AI reaches patients efficiently and reliably. You'll optimize the systems that translate raw model capability into sub-100ms responses—making the difference between conversational experiences that feel natural and those that feel broken. This role exists because inference optimization at scale is where research meets reality: your work directly determines latency, cost, and availability for millions of patient conversations across healthcare systems.
WHAT YOU WILL ACCOMPLISH
Own your first major outcome: By day 90, you will have shipped a measurable improvement to our inference serving stack (reduced latency, improved throughput, or optimized cost per inference), validated the gains across our production deployment scenarios, and established the performance optimization roadmap that will guide infrastructure investment.
Drive lasting impact: At 12 months, you will have designed and deployed advanced serving architectures (disaggregated inference, optimized caching, speculative decoding) that meaningfully improve patient experience and operational efficiency, contributed novel optimization techniques that become part of our core infrastructure, and made our serving stack a durable competitive advantage in healthcare AI deployment.
THE TEAM
You'll work alongside systems engineers, ML researchers, and infrastructure experts who are obsessed with making AI systems fast, reliable, and cost-effective. This is a team that values deep technical rigor, continuous benchmarking, and solving hard systems problems that have real impact on patient experience and business unit economics.
WHAT YOU'LL DO
- Design and implement multi-node serving architectures for distributed LLM inference
- Optimize multi-LoRA serving systems
- Apply advanced quantization techniques (FP4/FP6) to reduce model footprint while preserving quality
- Implement speculative decoding and other latency optimization strategies
- Develop disaggregated serving solutions with optimized caching strategies for prefill and decoding phases
- Continuously benchmark and improve system performance across various deployment scenarios and GPU types
LOCATION REQUIREMENT
We believe the best ideas happen together. This role is based in our Menlo Park, California office, expected to be in office five days a week.
COMPENSATION
Compensation is based on experience, expertise, and level of responsibility. We offer competitive packages that reflect the seniority and scope of the role, along with equity, health insurance, and other benefits.
WHAT YOU BRING
MUST-HAVE:
- Experience optimizing LLM inference systems at scale
- Proven expertise with distributed serving architectures for large language models
- Hands-on experience implementing quantization techniques for transformer models
- Strong understanding of modern inference optimization methods, including:
- Speculative decoding techniques with draft models
- Eagle speculative decoding approaches
- Proficiency in Python and C++
- Experience with CUDA programming and GPU optimization
NICE-TO-HAVE:
- Contributions to open-source inference frameworks such as vLLM, SGLang, or TensorRT-LLM
- Experience with custom CUDA kernels
- Track record of deploying inference systems in production environments
- Deep understanding of performance optimization systems
Show us what you've built: Tell us about an LLM inference or training project that makes you proud! Whether you've optimized inference pipelines to achieve breakthrough performance, designed innovative training techniques, or built systems that scale to billions of parameters - we want to hear your story.
Open source contributor? Even better! If you've contributed to projects like vllm, sglang, lmdeploy or similar LLM optimization frameworks, we'd love to see your PRs. Your contributions to these communities demonstrate exactly the kind of collaborative innovation we value.
Join a team where your expertise won't just be appreciated—it will be celebrated and amplified. Help us shape the future of AI deployment at scale!
References
1. Polaris: A Safety-focused LLM Constellation Architecture for Healthcare, https://arxiv.org/abs/2403.13313
2 https://arxiv.org/abs/2403.133132. Polaris 2: https://www.hippocraticai.com/polaris2
3 https://www.hippocraticai.com/polaris23. Personalized Interactions: https://www.hippocraticai.com/personalized-interactions
4 https://www.hippocraticai.com/personalized-interactions4. Human Touch in AI: https://www.hippocraticai.com/the-human-touch-in-ai
5 https://www.hippocraticai.com/the-human-touch-in-ai5. Empathetic Intelligence: https://www.hippocraticai.com/empathetic-intelligence https://www.hippocraticai.com/empathetic-intelligence6
WHY JOIN HIPPOCRATIC AI
Reinvent healthcare with AI that puts safety first. We’re building the world’s first healthcare‑only, safety‑focused LLM — a breakthrough platform designed to transform patient outcomes at a global scale. This is category creation.
Work with the people shaping the future. Hippocratic AI was co‑founded by CEO Munjal Shah and a team of physicians, hospital leaders, AI pioneers, and researchers from institutions like El Camino Health, Johns Hopkins, Washington University in St. Louis, Stanford, Google, Meta, Microsoft, and NVIDIA.
Backed by the world’s leading healthcare and AI investors. We recently raised a $126M Series C at a $3.5B valuation, led by Avenir Growth, bringing total funding to $404M with participation from CapitalG, General Catalyst, a16z, Kleiner Perkins, Premji Invest, UHS, Cincinnati Children’s, WellSpan Health, John Doerr, Rick Klausner, and others.
Build alongside the best in healthcare and AI. Join experts who’ve spent their careers improving care, advancing science, and building world‑changing technologies — ensuring our platform is powerful, trusted, and truly transformative.
EQUAL OPPORTUNITY
Hippocratic AI is an equal opportunity employer. We do not discriminate on the basis of race, color, religion, national origin, sex, age, disability, sexual orientation, gender identity or expression, genetic information, military or veteran status, or any other characteristic protected by applicable law. We are committed to building a team that reflects the patients we serve. We actively encourage applications from candidates of all backgrounds. If you require accommodations during the hiring process, please contact [email protected].
Please be aware of recruitment scams impersonating Hippocratic AI. All recruiting communication will come from @hippocraticai.com http://hippocraticai.com email addresses. We will never request payment or sensitive personal information during the hiring process.
Apply on Hippocratic AI's site
Attention Head lists this role as published on Hippocratic AI's own job board. Applications go to the employer.
Similar roles at other employers
- Sr. Staff Forward Deployed EngineerDatabricks · Remote - Washington D.C.; Washington, D.C.Nov 21, 2026
- Staff Applied AI EngineerBILL · United States$209K–250K
- Staff AI Agent Engineer – Moveworks | Partner SuccessServiceNow · New York, New York, United States$167K–291K
- Principal AI EngineerGallatin · El Segundo, CA; Washington, D.C.; Austin, TX; San Francisco, CA$240K–310K
- Member of Technical Staff, Core InfrastructureParallel Web Systems · San Francisco; New York, New York; Palo Alto$150K–300K
All staff and principal AI/ML Engineering roles in the United States →
More roles at Hippocratic AI
- Senior Software EngineerMenlo Park, CANov 12, 2026
- Senior Applied Scientist, Reinforcement LearningMenlo Park, CAOct 27, 2026
- Audio Transcription SpecialistMenlo Park, CAOct 22, 2026
- LLM Inference Systems EngineerMenlo Park, CAOct 7, 2026
- HCP Evaluator (PT 10-20 hrs)United StatesOct 2, 2026
- Sep 30, 2026
- Senior Applied ScientistSeattle, WASep 23, 2026
- AI Engineer (Senior, Staff, Senior Staff, Principal)Menlo Park, CAAug 24, 2026