Monitoring And Evaluation Specialist
KTek Resourcing · Charlotte, United States
FULL TIMEfull time
Job Description
About the Role
We are seeking a skilled GenAI / LLM Observability & Evaluation Engineer to help ensure the reliability, performance, and quality of production AI agents and LLM-powered applications.
The ideal candidate will have hands-on experience implementing LLM evaluation frameworks, distributed tracing, telemetry, monitoring dashboards, automated testing, and production reliability practices .
Required Skills
- Hands-on experience with LLMs, Generative AI, or Agentic AI applications.
- Experience with LLM evaluation, agent testing, and performance analysis.
- Strong Python programming and test automation skills.
- Experience implementing observability, tracing, telemetry, and monitoring solutions.
- Experience building metrics dashboards and analyzing application performance.
- Understanding of SLOs, production operations, incident response, and operational readiness.
Preferred Skills
- LLM evaluation tools such as DeepEval, Ragas, Promptfoo, or equivalent.
- AI observability platforms such as LangSmith, Langfuse, Arize Phoenix, or equivalent.
- OpenTelemetry, OpenInference, Prometheus, Grafana, Datadog, or Splunk.
- Experience with LangChain, LangGraph, Google ADK, or similar agent frameworks.
- Familiarity with CI/CD pipelines, automated regression testing, and release quality gates.
Details
| Company | KTek Resourcing |
| Location | Charlotte, United States |
| Type | FULL TIME |
| Niche | tech |
| Experience | full time |