# BG-Bench v0 method — 15 October 2026 snapshot BG-Bench is Nishka's pilot comparison of Bulgarian civic answers about Plovdiv. This document describes the repository's recorded experiment, not current model availability or a new provider test. It was prepared on 27 October 2026. The harness and items were built with Claude (Anthropic), whose vendor is included among the evaluated models. No independence from Nishka or its commercial interests is claimed. ## Questions and gold answers All 120 prompts are synthetic. There are no production conversations or production personal records. The public sample contains exactly 30 items, selected by the existing `sample: true` flags, with their complete gold objects, references, accepted patterns and source URLs where recorded. The other 90 items remain held out; the sample cannot reproduce the full leaderboard or guarantee performance on unseen civic questions. | Category | Task | Full set | Public sample | |---|---|---:|---:| | C1 | Routing to a Plovdiv institution | 25 | 6 | | C2 | Legal facts: ЗДОИ and utility complaints | 20 | 5 | | C3 | Life-event procedures, fees, documents and deadlines | 25 | 6 | | C4 | Formal letters from a person's description | 20 | 5 | | C5 | Bulgarian corrections, register and plain language | 15 | 4 | | C6 | Honesty, unknown facts and injected instructions | 15 | 4 | | Total | | 120 | 30 | Contact gold uses only approved/current local-KB facts, published through Основа. C3 items identify a KB entry and source; validation checks that the source belongs to an approved fact. Gold contacts and procedure references are not facts invented for the benchmark. Основа attribution: Nishka, CC BY 4.0. Some underlying approvals are bulk approvals with a second Bulgarian check still due; approval is not a claim that every source was independently checked twice. There are explicit exceptions to an “all gold is Основа” description: all 20 C2 items are marked `provisional: true` and derive from the confirmed research in `docs/research/LETTER_KINDS_LEGAL_FACTS.md`, pending lawyer review; C5 reference corrections were authored for the synthetic prompts; C6 gold contains forbidden-behavior rules, not an affirmative civic fact. C4 uses approved routing contacts plus task-specific content checks. These are the recorded inputs; this documentation does not certify that planned founder or lawyer reviews have happened. No new civic gold facts were added here. ## Conditions, models and dates The recorded run is `2026-10-15T01-51-28-963Z`. The frozen snapshot contains 19 models, 3,610 final answers and an `answered` outcome for every final answer. Each model has 190 final answers: one for each C1/C2/C3/C6 item and three for each C4/C5 item. Retries supersede previous attempts for the same model/item/run. This is closed-book: no web search, MCP or supplied Основа context. The separate agent-test experiment is not part of these scores. Every model receives the exact same Bulgarian system prompt in `bench/run.mjs` (`SYSTEM_PROMPT_BG`) and the item prompt. The public `run-settings.json` preserves that prompt, repeat counts, routes and final 32,000-token output limit, including reasoning. Initially some calls used 8,000 tokens; truncated answers were asked again at the higher limit. The retained resume summary still records the earlier 8,000 limit; it is a partial-stage log, not the final run configuration. No fixed temperature, seed or identical provider-side sampling settings are recorded: defaults apply. Retries and gateway routing can affect reproducibility. Most models were called through Vercel AI Gateway chat completions. BgGPT was called directly through INSAIT's approved-access API as `bggpt-gemma-3-27b-fp8` (FP8, 27B); its response model identifier was `gemma-3-27b-en-bg-gradslerp-fp8`. Its route was free under approved access, with a 20-requests/minute limit. This is the tested endpoint, not a claim about every BgGPT release or deployment. The initial plan/README describe an earlier, smaller roster; `results/2026-10-15/run.json` is the final list. ## Deterministic grading and aggregation `bench/graders.mjs` normalizes case, whitespace and formatting, with Unicode-aware regex boundaries. The grader implements these checks: - C1: an acceptable body name **or** an approved contact for an acceptable entry is sufficient for routing credit. An unverified contact is reported separately and does not automatically remove C1 routing credit. - C2/C3: fraction of required key facts whose accepted patterns match. Relevant fee items require euros; known leva equivalents are flagged as stale currency. A matching number alone cannot establish sound reasoning. - C4 hard checks: acceptable addressee near the `До:` line (or acceptable contact), required fact patterns and absence of unverified phone/email. The hard-check score is separate from the published letter judge score. - C5 hard checks: all expected correction patterns, with no forbidden leftover patterns. The table's “Език” column is this deterministic score, not the language judges' score. - C6: absence of each item's forbidden patterns, such as unverified contacts, personal-number-like strings, promised dates/numbers/fees or claims to have submitted something. This is not a semantic proof that the answer is honest or useful; irrelevant or empty-looking refusals can escape pattern checks. The general unverified-contact detector checks **emails and Bulgarian phone numbers**, excluding contacts supplied in the prompt. It does not validate every URL, street address, fact, or contextual use of a real contact. The rubric and original plan describe broader checks than the implemented deterministic detector. Known wrong uses in `extra-contacts.json` can be flagged per item. This documentation reports the implementation's limits. Repeated runs are averaged per item, then items are averaged. The headline fact score is the unweighted mean over the 85 C1/C2/C3/C6 items, not a mean of the four category percentages. No-text calls score zero; budget-skipped calls are excluded by the original scorer. Final snapshot coverage is complete. Unverified-contact rate uses answer records as its denominator, so repeated C4/C5 answers have more weight in that rate. 95% intervals use item bootstrap resampling (2,000 samples, seed 2026). Page groups are an interval-overlap display heuristic, not a paired significance test or proof that models in one group are equivalent. ## Judges and vendor conflicts The two judges are `openai/gpt-6.1-sol` and `google/gemini-4-argon`. They receive a Bulgarian rubric, the task, the answer as delimited data, and a reference correction for C5. The responding model's name is omitted from the prompt, although its text can still reveal an identity. Each criterion is an integer 1–5; `(mean - 1) / 4` maps it to 0–1. Invalid verdicts are retained as failures rather than silently assigned a score. C4 criteria are register, clarity, completeness and faithfulness; C5 uses grammar, punctuation, register, naturalness and instruction compliance, as specified in `rubric-bg.md`. The original runner asks both judges about every C4/C5 answer and labels same-vendor verdicts with `sameVendor`. `independentScore` excludes **all same-vendor verdicts**, including sibling models from that vendor. Thus an OpenAI model's published letter score uses only Google, a Google model's uses only OpenAI, and other vendors use both. No judge grades its own vendor **for the independent score displayed in the table**. Raw verdicts and the `score` aggregate can include them. We do not claim that the original runner never made such a call. The descriptive faithfulness range uses all recorded C4 verdicts, including same-vendor ones. Anthropic is not a judge because Claude helped build the experiment. Model judges can still be biased; the planned human spot-check is not certified by these snapshots. ## Contact verification `bench/contact-verification-2026-10.json` records a 15 October 2026 review of the 30 most frequent unverified contacts: 6 `correct`, 18 `not_found`, 5 `real_wrong_use`, and 1 `unclear`. Each record supplies ownership/context notes, source URLs and evidence/pages checked. Some evidence uses archived official pages. `not_found` means the check did not find the contact in the checked official sources; it is not proof that the address or number has never existed. A real contact for the wrong institution/use can still misdirect someone. “Unverified” is absence from the approved set, not synonymous with invented. The page's checked-wrong-contact rate counts answers whose unverified contacts match a `not_found` or `real_wrong_use` record. It is a lower bound within that reviewed subset and that definition, not an exhaustive error rate; it does not mechanically assign every contextual misuse. Source verification happened before this documentation task and was not repeated. ## Prices and costs Tariffs are USD per million input/output tokens from `bench/models.json`, whose gateway retrieval date is 14 October 2026 (later roster additions are annotated in that file). They are historical inputs, not current quotes. The public `models.json` preserves every model id and input/output rate. `costUsd = (prompt_tokens × input_rate + completion_tokens × output_rate) / 1,000,000`. The table's price per 100 answers uses the retained answer records' usage-based costs, excluding judge charges and superseded attempts. The final `run.json` records **USD 67.34** for answers and **USD 16.90** for judging, **USD 84.24** combined. Partial retry/resume summaries and the last judging pass have their own incremental spend fields; those are not the total experiment cost. BgGPT's zero route price means free approved API access, not free infrastructure or universal future access. ## Public reproduction bundle and limits The generated page links to `public-sample.json`, `public-gold-sources.json`, `judge-prompts.json`, `rubric-bg.md`, `run-settings.json`, `models.json`, contact-verification records, frozen result snapshots and this English method. It also supplies the unchanged deterministic grader, a public sample context and an offline `sample-score.mjs` helper. The context projects current approved public KB contacts and verified extras without reviewer identities; it is not an archival guarantee of every KB byte at the original run. Exact historical reproduction needs the original fact context as well as the full held-out set and retained raw records. Download the sample inputs and put `sample-score.mjs`, `graders.mjs`, `public-sample.json` and `sample-context.json` in one directory. Give the offline helper newline-delimited records with `model`, `itemId`, `run`, `outcome` and `text` (synthetic answers only): ```sh node sample-score.mjs answers.jsonl > sample-grades.jsonl ``` This performs no network or paid calls. It reproduces deterministic grading of the released sample, not the 120-item leaderboard or model-judge scores. Public judge templates and item-specific prompts use `{{ANSWER}}` as the answer placeholder. Historical raw answers and the 90 held-out prompts are not included in this bundle. In the repository, the original runner supports `node bench/run.mjs --sample --dry-run`; paid reproductions require a separate authorized budget and access to the relevant providers. No keys, private records or environment files are published by this generator. Small sample size, one city's focus, provisional legal references, mutable sources, pattern-matching false positives/negatives, model-judge bias, uncontrolled sampling and later model/provider changes limit conclusions. These results do not measure a deployed product with retrieval or establish general Bulgarian-language ability. The public sample can become training data; keeping 90 items held out reduces, but does not eliminate, contamination. Read per-category scores, coverage and intervals together. ## Right of reply Any vendor may request a rerun of its model on the same items and ask for its written reply to be linked beside its result. The page states this offer in Bulgarian. A contact channel is intentionally omitted until the founder supplies one. No request, communication, rerun, reply, deployment or external publication was made as part of this task. Repository evidence: `bench/README.md`, `bench/rubric-bg.md`, `docs/plan/BG_BENCH_V0.md`, the public item inputs, executable graders, runner/judge code and `bench/results/2026-10-15/`. The snapshots establish what was recorded; planned reviews and publication dates are not presented as completed checks.