🤖 15+ AI Agents working for you. Find jobs, score and update resumes, cover letter, interview questions, missing keywords, and lots more.
Cookies & analytics consent
We serve candidates globally, so we only activate Google Tag Manager and other analytics after you opt in. This keeps us aligned with GDPR/UK DPA, ePrivacy, LGPD, and similar rules. Essential features still run without analytics cookies.
Read how we use data in our Privacy Policy and Terms of Service.
🤖 15+ AI Agents working for you. Find jobs, score and update resumes, cover letter, interview questions, missing keywords, and lots more.
YO IT Consulting • Montreal, Quebec, Canada
Role & seniority
Independent contractor; senior-level candidates preferred (Senior/Staff Engineers, QA Engineers, Technical Writers)
Also a good fit for Backend/Full-Stack Developers with relevant experience; DevOps/SRE considered
Stack/tools
Python (helpful)
JavaScript, TypeScript, Java, C++, Go, Ruby, Rust, Bash (plus)
Git workflows; testing frameworks; debugging tools
Experience reviewing code, tests, and documentation; analyzing conversations and workflows
Top 3 responsibilities
Review long-form transcripts between users and AI coding assistants
Analyze AI logic, execution, and stated actions; detect mismatches between claims and actions
Score transcripts using a 10-point rubric across criteria; optionally provide brief justifications with dialogue examples
Must-have skills
Deep code review experience and execution insight
Strong verification, consistency-checking, and attention to detail
Ability to articulate how instructions map to implementation (documentation/communication skills)
Nice-to-haves
Experience with developer tooling and developer workflows
Familiarity with multiple programming languages and API/test workflow concepts
Technical writing or documentation specialization focused on instructions vs. implementation
Location & work type
Fully remote, independent contractor
Flexible, task-based with potential recurring batches
Each transcript batch must be completed within 5 ho
Partnering with a top AI research organization to evaluate and improve how coding assistants reason, act, and communicate during development workflows. We’re seeking technically sharp experts (especially those with experience in code review, testing, or documentation) to assess full transcripts of user-AI coding conversations. This short-term engagement helps shape the future of developer-assisting AI systems.
Key Responsibilities
Review long-form transcripts between users and AI coding assistants Analyze the AI’s logic, execution, and stated actions in detail Score each transcript using a 10-point rubric across multiple criteria Optionally write brief justifications citing examples from the dialogue Detect mismatches between claims and actions (e.g., saying “I’ll run tests” but not doing so)
Ideal Qualifications
Senior or Staff Engineers with deep code review experience and execution insight QA Engineers with strong verification and consistency-checking habits Technical Writers or Documentation Specialists skilled at comparing instructions vs. implementation
Also a Strong Fit
Backend or Full-Stack Developers comfortable with function calls, APIs, and test workflows DevOps or SRE professionals familiar with tool orchestration and system behavior analysis
Languages And Tools
Proficiency in Python is helpful (most transcripts are Python-based) Familiarity with other languages like JavaScript, TypeScript, Java, C++, Go, Ruby, Rust, or Bash is a plus Comfort with Git workflows, testing frameworks, and debugging tools is valuable
More About the Opportunity
Must complete each transcript batch within 5 hours of starting (unlimited tasks to be done) Flexible, task-based engagement with potential for recurring batches
Application Process
Submit your resume to begin If selected, you’ll receive rubric documentation and access to the evaluation platform Most applicants hear back within a few business days
We consider all qualified applicants without regard to legally protected characteristics and provide reasonable accommodations upon request.
Contract and Payment Terms
You will be engaged as an independent contractor. This is a fully remote role that can be completed on your own schedule. Projects can be extended, shortened, or concluded early depending on needs and performance. Your work at will not involve access to confidential or proprietary information from any employer, client, or institution. Payments are weekly on Stripe or Wise based on services rendered.