Issues

• Idea: Adding a CI/CD gate to block PRs based on these evaluations?

  OpenAgentHQ/openagent-eval#362 · Filter by author mukundzha opened on Aug 29, 2026
  1

• Add generic exponential-backoff retry to the Groq provider's existing rate-limit handling

  enhancement
  OpenAgentHQ/openagent-eval#360 · Filter by author himanshu231204 opened on Aug 28, 2026

• Add plugin sandboxing/validation for third-party metric and provider plugins

  enhancement
  OpenAgentHQ/openagent-eval#359 · Filter by author himanshu231204 opened on Aug 28, 2026

• RFC: opt-in community benchmark submission flow

  enhancement
  OpenAgentHQ/openagent-eval#358 · Filter by author himanshu231204 opened on Aug 28, 2026

• Add embedding/response cache with pluggable backends (disk, Redis)

  enhancement
  OpenAgentHQ/openagent-eval#357 · Filter by author himanshu231204 opened on Aug 28, 2026

• Add multi-agent evaluation support (parallel agent/config comparison on the same task)

  enhancement
  OpenAgentHQ/openagent-eval#356 · Filter by author himanshu231204 opened on Aug 28, 2026

• Package a reusable CI/CD regression-detection workflow template for downstream repos

  documentationenhancement
  OpenAgentHQ/openagent-eval#355 · Filter by author himanshu231204 opened on Aug 28, 2026

• Add per-provider rate limiting configurable in Config

  enhancement
  OpenAgentHQ/openagent-eval#354 · Filter by author himanshu231204 opened on Aug 28, 2026

• Build a multi-turn/agent pipeline alongside the existing single-turn RAG pipeline

  enhancement
  OpenAgentHQ/openagent-eval#353 · Filter by author himanshu231204 opened on Aug 28, 2026

• Add streaming support to the LLMProvider interface and all six adapters

  enhancement
  OpenAgentHQ/openagent-eval#352 · Filter by author himanshu231204 opened on Aug 28, 2026

• Extend blame attribution to multi-step agent trajectories

  enhancement
  OpenAgentHQ/openagent-eval#351 · Filter by author himanshu231204 opened on Aug 28, 2026

• Design and implement a tool-call trace dataset schema and correctness metric

  enhancement
  OpenAgentHQ/openagent-eval#350 · Filter by author himanshu231204 opened on Aug 28, 2026

• Add a safety/toxicity metric reusing the NLI-judge infrastructure

  enhancement
  OpenAgentHQ/openagent-eval#349 · Filter by author himanshu231204 opened on Aug 28, 2026

• Add real dollar-cost estimation using a maintained per-model pricing table

  enhancement
  OpenAgentHQ/openagent-eval#348 · Filter by author himanshu231204 opened on Aug 28, 2026

• Build oaeval history command for multi-run metric trends

  enhancement
  OpenAgentHQ/openagent-eval#346 · Filter by author himanshu231204 opened on Aug 28, 2026

• Add a ProviderContractTest base class applied to all LLM/retriever providers

  enhancement
  OpenAgentHQ/openagent-eval#345 · Filter by author himanshu231204 opened on Aug 28, 2026

• Enforce Ruff and MyPy as blocking CI checks

  bugenhancement
  OpenAgentHQ/openagent-eval#344 · Filter by author himanshu231204 opened on Aug 28, 2026

• Add a local disk response cache keyed on (provider, model, prompt hash)

  enhancement
  OpenAgentHQ/openagent-eval#343 · Filter by author himanshu231204 opened on Aug 28, 2026

• Add a generic retry/backoff wrapper shared by all LLM providers

  enhancement
  OpenAgentHQ/openagent-eval#342 · Filter by author himanshu231204 opened on Aug 28, 2026

• Register retrievers via entry points instead of the hardcoded factory dict

  enhancement
  OpenAgentHQ/openagent-eval#341 · Filter by author himanshu231204 opened on Aug 28, 2026

• Register LLM providers via entry points instead of the hardcoded factory dict

  enhancementin-progress
  OpenAgentHQ/openagent-eval#340 · Filter by author himanshu231204 opened on Aug 28, 2026
  2

  

• Seed one good-first-issue per top-level module for newcomer discoverability

  good first issue
  OpenAgentHQ/openagent-eval#336 · Filter by author himanshu231204 opened on Aug 28, 2026

• Add a CODEOWNERS file mapping directories to reviewers

  documentationgood first issue
  OpenAgentHQ/openagent-eval#335 · Filter by author himanshu231204 opened on Aug 28, 2026
  1

• Reconcile coverage target: CONTRIBUTING.md says 80%+, CI enforces 75%

  documentationgood first issue
  OpenAgentHQ/openagent-eval#333 · Filter by author himanshu231204 opened on Aug 28, 2026

• Add provider_request.md and metric_request.md issue templates

  documentationgood first issue
  OpenAgentHQ/openagent-eval#332 · Filter by author himanshu231204 opened on Aug 28, 2026