attention head AI jobs Alerts Saved
Jobs › Robots and Pencils

Principal Applied Scientist

Robots and Pencils · US Remote
Apply on Robots and Pencils' site
Tailor my resume
Posted
Nov 12, 2026
Last verified
6 h ago
Level
Staff+
Function
Research
AI roleRemotePythonGoLLMsEvals

We're looking for a Principal Applied Scientist to lead the design and build of an evaluation framework for an enterprise planner-based agent. You will work directly with the client's engineering team to benchmark their agent across planner models, executor LLMs and inference optimization levers. Every proposed change, whether a model upgrade, a harness change or a routing strategy, becomes a controlled, statistically sound experiment with a measurable dollar outcome.

This is a research-heavy, customer-facing role with a significant build component. You will own both the experimental method and the tooling that makes it repeatable: a scenario-driven user simulator, isolated experiment execution, statistical comparison, and automated insights in a locally hosted dashboard the client can adopt as their own. The work runs in three overlapping streams (evaluation pipeline, grading and statistics, dashboard and experiment campaign) and ends in a recommendation the client can act on.

Why This Role Matters

At Robots & Pencils, we design AI systems for a human world. Our name says it all. Robots and pencils means engineering paired with creativity, because every agent we ship has to work for real people in real workflows. That balance is baked into how we operate.

Agents are only as good as the evidence behind them. Today, most teams pick a model or switch on a cost lever on instinct. Here, you replace instinct with data: you build the system that tells the client which planner model, which executor and which levers to roll out, how much quality each change costs or saves, and what it is worth in dollars at production volume. You also show how well the planner carries over from data analytics to tasks. This is a research role with real build accountability, and your work becomes the client's own decision-making tool. It demands people who take ownership, lead with craft, and care deeply about what they put their name on.

What You'll Do

Evaluation Pipeline & Simulation

Grading, Statistics & Insights

Dashboard, Experiments & Readout

Collaboration & Technical Leadership

What You'll Bring

Helpful Extras and Unique Skills

Our salary range is $200,00 - $300,000

Apply on Robots and Pencils' site

Attention Head lists this role as published on Robots and Pencils' own job board. Applications go to the employer.

Not this one? Get roles like it weekly

A job alert: the week's best new AI roles from employers' own job boards, every Monday. Unsubscribe in one click.

Similar roles at other employers

All staff and principal Research roles in the United States →

More roles at Robots and Pencils

All roles at Robots and Pencils →