Senior Python Engineer - AI Coding Agent Evaluation (Freelance)

About the role

We are looking for a senior Python engineer to help build a high‑quality dataset for evaluating AI coding agents. You will design realistic developer environments and craft challenging tasks that test the limits of current models.

Key responsibilities

• Create virtual company environments with codebases, infrastructure, tickets, documentation, and conversation histories.
• Design tasks from intermediate states, write clear prompts, define success criteria, and ensure they are solvable by an AI agent.
• Develop functional and integration tests that accept all correct solutions while rejecting incorrect ones.
• Iterate on tasks and tests based on QA feedback, reviewing agent outputs and refining evaluation criteria.

Required profile

• 8+ years of professional software development experience.
• Strong command of English (minimum B2 level).
• Hands‑on experience with Python (FastAPI) and JavaScript/TypeScript (React).
• Familiarity with Docker, PostgreSQL, Kafka, and Redis.
• Proven ability to write comprehensive tests (functional, integration).

Required skills

• Python
• FastAPI
• JavaScript
• TypeScript
• React
• Docker
• PostgreSQL
• Kafka
• Redis
• Testing (functional & integration)

What we offer

• Project‑based freelance engagement with flexible scheduling.
• Compensation up to $150 per hour, based on experience and pace.
• Opportunity to work on cutting‑edge AI evaluation challenges.