Skip to content

Assessment: Add support for Anthropic #932

Description

@vprashrex

Is your feature request related to a problem?
The assessment module's batch pipeline only supports OpenAI and Gemini providers, preventing assessments configured with Anthropic models from running as batch jobs. This limitation hinders cost-efficient bulk evaluations of Claude models.

Describe the solution you'd like

  • Implement an AnthropicBatchProvider that meets the BatchProvider interface, using the Anthropic Message Batches API for batch creation, status polling, and result retrieval.
  • Add build_anthropic_jsonl to format assessment items for the Anthropic batch request and route assessments to the new provider.
  • Include map_kaapi_to_anthropic_params for translating Kaapi model parameters to Anthropic API format.
  • Create build_anthropic_attachment_parts to convert assessment attachments into Anthropic content blocks.
  • Extend output parsing to manage Anthropic batch responses, including error handling and text extraction.
  • Provide unit tests for the new provider, mappers, and parsing in the relevant test directories.
Original issue

Describe the current behavior
The assessment module's batch pipeline (backend/app/core/batch/) only supports OpenAI and Gemini providers, so assessments configured with Anthropic models cannot run as batch jobs. This blocks cost-efficient bulk evaluation runs on Claude models and forces Anthropic-backed assessments through the synchronous path.

Describe the enhancement you'd like

  • Add an AnthropicBatchProvider implementing the BatchProvider interface (backend/app/core/batch/anthropic.py), covering batch creation via the Anthropic Message Batches API, status polling, and result retrieval.
  • Add build_anthropic_jsonl in backend/app/crud/assessment/batch.py to format assessment items into Anthropic batch request format, and extend submit_assessment_batch to route Anthropic-provider assessments to the new provider.
  • Add map_kaapi_to_anthropic_params in backend/app/services/assessment/mappers.py to translate Kaapi model parameters (including max_tokens defaulting via DEFAULT_ANTHROPIC_MAX_TOKENS) to Anthropic API format.
  • Add build_anthropic_attachment_parts in backend/app/services/assessment/utils/attachments.py to convert assessment attachments into Anthropic content blocks.
  • Extend output parsing in backend/app/crud/assessment/processing.py to handle Anthropic batch responses, including error results and text extraction from content blocks.
  • Cover the new provider, mappers, and parsing with unit tests (app/tests/core/batch/test_anthropic.py, app/tests/assessment/).

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No fields configured for Task.

Projects

Status
Closed

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions