QA LLM Engineer | Talentica.com
- Talentica
- 3 - 5 years
- Bengaluru
- 1 month ago
- Email to a friend
- Report this job
Job Description
A QA Automation Engineer with strong experience in LLMs and GenAI who can ensure the accuracy, stability, and performance of AI-driven applications.
If you have a strong understanding of how LLMs interact with data pipelines covering indexing, chunking, embeddings, cosine similarity and keyword search along with hands-on experience in LLM observability, prompt evaluation, and QA automation you ll fit right in.
What you ll do
- Design and execute QA strategies for LLM-based and search-driven products.
- Validate data pipelines involving indexing, chunking, embeddings, cosine similarity and keyword search.
- Evaluate retrieval-augmented generation (RAG) and recommendation system quality using precision, recall, and relevance metrics.
- Develop prompt test suites to measure accuracy, consistency, and bias.
- Monitor LLM observability metrics such as latency, token usage, hallucination rate and cost performance.
- Automate end-to-end test scenarios using Playwright and integrate with CI/CD pipelines.
- Collaborate with ML engineers and developers to improve model responses and user experience.
- Contribute to test frameworks and datasets for LLM regression and benchmark testing.
To be successful in this role, you should have
Qualification:
Qualification:
- BE/BTech in Computer Science, Data Engineering, or a related field from a top institute (like IIT, NIT, BITS, etc.).
Experience:
- 3.5 to 5.5 years of experience in QA engineering, with at least 1+ years of experience in GenAI or LLM-based systems.
Skills
- Strong understanding of indexing, chunking, embeddings, similarity search, and retrieval workflows.
- Experience with prompt engineering, LLM evaluation, and output validation techniques.
- Proficiency with Playwright, API automation, and modern QA frameworks.
- Knowledge of observability tools for LLMs
- Solid scripting experience in Python.
- Knowledge of different LLM providers (OpenAI, Gemini, Anthropic, Mistral, etc.)
- Exposure to RAG pipelines, recommendation systems, or model performance benchmarking.
- Strong analytical and debugging skills, with a detail-oriented mindset
What you ll find here
- A culture of innovation: We only take up projects that challenge us to innovate. Our customers come to us for our technology expertise.
- Endless learning opportunities: Continuous learning is baked into our DNA. You ll always have the chance to learn new things and stay on top of the latest trends.
- Talented peers: Lead and Work alongside engineers from IITs, NITs, BITS, and other premier institutions.
- Work-life balance: We value work-life balance and offer flexible schedules, including remote work options, so you can thrive both professionally and personally.
- A great culture: Our employees love working here! 82% recommend Talentica to their friends, according to Glassdoor. Join us, and you ll see why!
- Recognition & rewards: We don t just work hard, we celebrate success. Your contributions won t go unnoticed. We ll make sure youre recognized for the amazing work you do.
Job Classification
Industry: Software ProductFunctional Area / Department: Engineering - Software & QARole Category: Quality Assurance and TestingRole: Blockchain Quality Assurance EngineerEmployement Type: Full timeContact Details:
Company: TalenticaLocation(s): Bengaluru+ View Contact
Keyskills: Computer science Automation Product engineering Aerospace Analytical Debugging Test scenarios Telecommunication Information technology Python
Fraud Alert to job seekers!
₹ Not Disclosed
Similar positions
Talentica
Talentica, a product development companyThanks and Regards,Chetana Patil9359512528 (cp@rwindia.co)Talent Acquisition I Talentica Software (I) Pvt Ltd I http://www.talentica.com I
Job Listings
We are in open beta version of website, please let us know if any issues, at: info(at)indigojobs.in