Job Description

Functional Tester AI Agents

Tampa, FL/Remote

TCS

Role Overview
As a Functional Tester specializing in AI Agents, you will be responsible for designing, executing, and maintaining comprehensive test cases that validate the functionality, reliability, and performance of AI-driven agent systems. You will collaborate closely with development teams, AI specialists, and product owners to ensure AI agents deliver accurate, safe, and efficient performance in real-world scenarios.

Key Responsibilities

• AI/LLM-Specific QA
  • Design and execute test frameworks for validating LLM outputs, including hallucination detection, factual accuracy checks, and bias/toxicity identification.
  • Develop automated evaluators and integrate human-in-the-loop validation processes.
• Automation Frameworks
  • Implement and maintain test automation using PyTest, Robot Framework, Playwright, Cypress, or custom LLM test harnesses.
• Evaluation Metrics
  • Apply industry-standard NLP evaluation metrics such as BLEU, ROUGE, BERTScore, and GPT-based evaluators.
• Data QA
  • Ensure quality of training and evaluation datasets.
  • Apply validation frameworks (e.g., Great Expectations, Deequ) to detect anomalies and maintain data integrity.
• Python Automation
  • Script test cases, perform API test automation, and develop scalable QA tools for AI systems.
• CI/CD Integration
  • Build automated test pipelines within Jenkins, GitHub Actions, or GitLab CI.
  • Enforce quality gates to ensure reliability across deployments.
• Performance Testing
  • Conduct latency, throughput, and stress testing of LLM-powered services to assess scalability.
• Leadership (Manager Level)
  • Define QA strategies for AI and LLM-based products.
  • Lead and mentor test automation teams.
  • Establish testing standards and enforce quality governance across AI initiatives.

Qualifications

• Strong experience in software testing, particularly with AI/ML or NLP-based systems.
• Proficiency in Python and test automation frameworks.
• Familiarity with AI evaluation metrics and dataset validation techniques.
• Hands-on experience with CI/CD pipelines and cloud-based testing environments.
• Excellent problem-solving, analytical, and communication skills.

Job Tags

Similar Jobs