Founding AI Quality & Release Engineer

True Talent Solutions Partners · Remote, United States

About the Role

This is a founding-level software engineering role focused on building reliable, AI-powered verification and release infrastructure. You will develop the systems that evaluate agent-based products, strengthen CI signals, and help ensure safe, predictable releases.
What Youll Do

Build and maintain test harnesses that use agents to verify product behavior.

Develop evaluation logic, including LLM graders and regression coverage for agentic systems.

Create automated release gates that make readiness visible and prevent unsafe releases.

Own CI health, release pipelines, artifact publication, and deployment automation.

Improve the reliability of staging and demonstration environments.

Investigate quality gaps and incidents, and improve systems to prevent recurring issues.

What Were Looking For

Seven to fifteen years of experience in quality engineering, release engineering, or platform engineering.

Recent hands-on experience using AI-native methods to test or evaluate LLM-based or agentic systems.

Practical experience with LLM APIs, tool calling or MCP, and evaluation methods.

Strong familiarity with GitHub Actions and GitHub-based CI/CD, including container builds and pipeline automation.

Experience with Python, TypeScript, Docker, Kubernetes, and Helm.

Working knowledge of at least one major cloud or data platform, such as AWS, Azure, Databricks, or Snowflake.

A track record of taking independent ownership of engineering systems, ideally in a high-velocity startup or while modernizing established engineering practices.

Authorization to work in the United States or Canada without visa sponsorship.

Compensation & Benefits

Base salary: $170,000 to $200,000 USD annually. Visa sponsorship is not available.
Location

Remote within the United States or Canada. Primary location: New York, New York, United States.