# Accuracy Benchmarks

This directory hosts accuracy benchmark integrations that run entirely through a
local `vllm-omni serve` deployment.

Current integrations:

- `text_to_image/`: GEBench generation + local judge scoring flow.
- `image_to_image/`: GEdit-Bench generation + local VIEScore-style scoring flow.

Design notes:

- Generation is executed through the OpenAI-compatible endpoints exposed by
  `vllm-omni serve`.
- Evaluation is also executed through a local OpenAI-compatible judge model
  served by `vllm-omni`.
- Both generation and judge requests accept either `http://host:port` or
  `http://host:port/v1`.
- Output directory layout intentionally stays close to the upstream repos.

Test guidance:

- Local static/self-checks live in `tests/benchmarks/test_accuracy_bench_utils.py`.
- End-to-end generation/evaluation should be validated in a remote GPU
  environment. In the current repo marker system there is `L4` but no `L5`
  marker, so benchmark smoke tests should be wired as `full_model +
  benchmark + L4` for nightly when GPU capacity is available.