Agentic AI Data Annotator (Freelancer)

About Us

Deccan AI Experts is a pioneering AI company founded by IIT Bombay and IIM Ahmedabad alumni, with a strong founding team from IITs, NITs, and BITS. We specialize in high-quality human-curated data, AI-first operations, and advanced AI evaluation systems.

About the Role

We are seeking an Agentic AI Data Annotator (Freelancer) to support advanced AI evaluation initiatives focused on AI agents, tool use, function calling, reasoning workflows, and human-in-the-loop data annotation.

In this role, you will annotate and evaluate AI agent interactions, tool-use traces, function-calling sequences, reasoning workflows, and multi-turn conversations. Your expertise will help improve AI systems designed for autonomous task execution, agent planning, instruction following, and decision-making.

This position is ideal for professionals with experience in AI data annotation, LLM evaluation, prompt engineering, AI operations, data labeling, software testing, or technical quality assurance.

Responsibilities

• Create deliverables addressing real-world AI agent and tool-use scenarios.
• Annotate and evaluate AI-generated agent conversations, reasoning traces, function calls, and workflow executions.
• Assess AI outputs for correctness, instruction adherence, tool selection, reasoning quality, and task completion.
• Label tool-use and function-calling traces, ensuring correct tool selection, parameter usage, execution order, and expected outcomes.
• Perform single-step action labeling by identifying whether each agent action is valid, necessary, and aligned with the user's intent.
• Annotate agent turns across multi-step workflows, evaluating reasoning flow, context retention, decision quality, and task progression.
• Tag failure modes, including hallucinations, incorrect tool selection, missing tool calls, execution errors, reasoning failures, unsafe behavior, and incomplete task execution.
• Perform instruction-following checks to verify that AI agents accurately follow user requests, system instructions, policies, and workflow constraints.
• Identify annotation inconsistencies, workflow errors, policy violations, and quality issues in AI-generated outputs.
• Provide structured feedback to improve AI agent performance, tool orchestration, and evaluation quality.
• Review peer-developed deliverables to maintain quality and consistency standards.

Requirements

• Bachelor's degree in Computer Science, Artificial Intelligence, Data Science, Information Technology, Engineering, Mathematics, Linguistics, or a related field.
• 2+ years of hands-on experience in AI data annotation, LLM evaluation, prompt engineering, AI operations, software QA, or technical annotation.
• Evaluating agent turn annotations for reasoning quality, context continuity, and decision-making.
• Identifying and categorizing failure modes, including hallucinations, planning errors, execution failures, policy violations, and tool misuse.
• Conducting instruction-following evaluations to verify compliance with user requests and system constraints.
• Reviewing complex multi-turn conversations and AI workflows for annotation quality and consistency.
• Familiarity with annotation platforms such as Label Studio, Scale AI, Surge AI, Labelbox, or similar tools is preferred.
• Basic understanding of APIs, JSON, structured outputs, and software workflows is highly desirable.
• Strong analytical thinking, attention to detail, and problem-solving skills.
• Excellent written English communication skills.
• Ability to critically evaluate AI-generated agent behaviors and annotation quality.
• Ability to work independently in a remote, fast-paced environment.

Preferred Qualifications

• Experience working on RLHF, AI model evaluation, benchmark creation, or agentic AI projects.
• Familiarity with OpenAI function calling, MCP (Model Context Protocol), tool orchestration, retrieval-augmented generation (RAG), or autonomous AI agents.
• Experience evaluating reasoning traces, chain-of-thought quality (without exposing private reasoning), workflow execution, or planning agents.
• Knowledge of Python, SQL, JavaScript, or API testing is an advantage.
• Experience with AI safety, policy evaluation, trust & safety, or responsible AI initiatives.
• Contributions to AI evaluation, annotation guidelines, or benchmark datasets are highly desirable.

Why Join Us

• Competitive hourly pay: ₹1,500/hour
• Fully remote with flexible working hours.
• Opportunity to contribute to cutting-edge AI research and agentic AI initiatives.
• Exposure to advanced AI systems focused on autonomous agents, reasoning, and tool orchestration.
• Flexible project-based opportunities with global teams.
• Work on next-generation AI solutions supporting reliable, safe, and intelligent AI agents.