Is your feature request related to a problem?
The assessment module's batch pipeline only supports OpenAI and Gemini providers, preventing assessments configured with Anthropic models from running as batch jobs. This limitation hinders cost-efficient bulk evaluations of Claude models.
Describe the solution you'd like
- Implement an
AnthropicBatchProvider that meets the BatchProvider interface, using the Anthropic Message Batches API for batch creation, status polling, and result retrieval.
- Add
build_anthropic_jsonl to format assessment items for the Anthropic batch request and route assessments to the new provider.
- Include
map_kaapi_to_anthropic_params for translating Kaapi model parameters to Anthropic API format.
- Create
build_anthropic_attachment_parts to convert assessment attachments into Anthropic content blocks.
- Extend output parsing to manage Anthropic batch responses, including error handling and text extraction.
- Provide unit tests for the new provider, mappers, and parsing in the relevant test directories.
Original issue
Describe the current behavior
The assessment module's batch pipeline (backend/app/core/batch/) only supports OpenAI and Gemini providers, so assessments configured with Anthropic models cannot run as batch jobs. This blocks cost-efficient bulk evaluation runs on Claude models and forces Anthropic-backed assessments through the synchronous path.
Describe the enhancement you'd like
- Add an
AnthropicBatchProvider implementing the BatchProvider interface (backend/app/core/batch/anthropic.py), covering batch creation via the Anthropic Message Batches API, status polling, and result retrieval.
- Add
build_anthropic_jsonl in backend/app/crud/assessment/batch.py to format assessment items into Anthropic batch request format, and extend submit_assessment_batch to route Anthropic-provider assessments to the new provider.
- Add
map_kaapi_to_anthropic_params in backend/app/services/assessment/mappers.py to translate Kaapi model parameters (including max_tokens defaulting via DEFAULT_ANTHROPIC_MAX_TOKENS) to Anthropic API format.
- Add
build_anthropic_attachment_parts in backend/app/services/assessment/utils/attachments.py to convert assessment attachments into Anthropic content blocks.
- Extend output parsing in
backend/app/crud/assessment/processing.py to handle Anthropic batch responses, including error results and text extraction from content blocks.
- Cover the new provider, mappers, and parsing with unit tests (
app/tests/core/batch/test_anthropic.py, app/tests/assessment/).
Is your feature request related to a problem?
The assessment module's batch pipeline only supports OpenAI and Gemini providers, preventing assessments configured with Anthropic models from running as batch jobs. This limitation hinders cost-efficient bulk evaluations of Claude models.
Describe the solution you'd like
AnthropicBatchProviderthat meets theBatchProviderinterface, using the Anthropic Message Batches API for batch creation, status polling, and result retrieval.build_anthropic_jsonlto format assessment items for the Anthropic batch request and route assessments to the new provider.map_kaapi_to_anthropic_paramsfor translating Kaapi model parameters to Anthropic API format.build_anthropic_attachment_partsto convert assessment attachments into Anthropic content blocks.Original issue
Describe the current behavior
The assessment module's batch pipeline (
backend/app/core/batch/) only supports OpenAI and Gemini providers, so assessments configured with Anthropic models cannot run as batch jobs. This blocks cost-efficient bulk evaluation runs on Claude models and forces Anthropic-backed assessments through the synchronous path.Describe the enhancement you'd like
AnthropicBatchProviderimplementing theBatchProviderinterface (backend/app/core/batch/anthropic.py), covering batch creation via the Anthropic Message Batches API, status polling, and result retrieval.build_anthropic_jsonlinbackend/app/crud/assessment/batch.pyto format assessment items into Anthropic batch request format, and extendsubmit_assessment_batchto route Anthropic-provider assessments to the new provider.map_kaapi_to_anthropic_paramsinbackend/app/services/assessment/mappers.pyto translate Kaapi model parameters (includingmax_tokensdefaulting viaDEFAULT_ANTHROPIC_MAX_TOKENS) to Anthropic API format.build_anthropic_attachment_partsinbackend/app/services/assessment/utils/attachments.pyto convert assessment attachments into Anthropic content blocks.backend/app/crud/assessment/processing.pyto handle Anthropic batch responses, including error results and text extraction from content blocks.app/tests/core/batch/test_anthropic.py,app/tests/assessment/).