LLM Ops Engineer
- Pattern Technologies
- 3 - 5 years
- Pune
- 3 months ago
- Email to a friend
- Report this job
Job Description
Monitor, evaluate, and optimize AI/LLM workflows in production environments. Ensure reliable, efficient, and high-quality AI system performance by building out an LLM Ops platform that is self-serve for the engineering and data science departments. Key Responsibilities:-
- Collaborate with data scientists and software engineers to integrate an LLM Ops platform (Opik by CometML) for existing AI workflows
- Identify valuable performance metrics (accuracy, quality, etc) for AI workflows and create on-going sampling evaluation processes using the LLM Ops platform that alert when metrics drop below thresholds
- Cross-team collaboration to create datasets and benchmarks for new AI workflows
- Run experiments on datasets and optimize performance via model changes and prompt adjustments
- Debug and troubleshoot AI workflow issues
- Optimize inference costs and latency while maintaining accuracy and quality Develop automations for LLM Ops platform integration to empower data scientists and software engineers to self-serve integration with the AI workflows they build
Requirements:-
- Strong Python programming skills
- Experience with generative AI models and tools (OpenAI, Anthropic, Bedrock, etc)
- Knowledge of fundamental statistical concepts and tools in data science such as: heuristic and non-heuristic measurements in NLP (BLEU, WER, sentiment analysis, LLM-as-judge, etc), standard deviation, sampling rate, and a high level understanding of how modern AI models work (knowledge cutoffs, context windows, temperature, etc)
- Familiarity with AWS
- Understanding of prompt engineering concepts
- People skills: you will be expected to frequently collaborate with other teams to help to perfect their AI workflows
- Experience Level 3-5 years of experience in LLM/AI Ops, MLOps, Data Science, or MLE
Job Classification
Industry: RetailFunctional Area / Department: Engineering - Software & QARole Category: Software DevelopmentRole: Head - EngineeringEmployement Type: Full timeContact Details:
Company: Pattern TechnologiesLocation(s): Pune+ View Contact
Keyskills: data science Debugging Programming Workflow Windows Troubleshooting AWS Monitoring Python
Fraud Alert to job seekers!
₹ Not Disclosed
Similar positions
Pattern Technologies
Pattern is the premier partner for global e-commerce acceleration and is headquartered in Utah's Silicon Slopes tech hubwith offices in Asia, Australia, Europe, the Middle East, and North America. Valued at $2 billion, Pattern has been named one of the fastest-growing tech companies in North Amer...
Job Listings
We are in open beta version of website, please let us know if any issues, at: info(at)indigojobs.in