Scoring an AI response sounds simple until you try to make the score mean something. The obvious approach is to ask another model, “How good is this answer from 0 to 100?” But that gives the judge too much authority. An overall score like 87 looks precise while hiding most of the reasoning behind it. Why 87 instead of 82? Did the response actually get the core answer right? Did it follow the instructions? Did it invent a fact? Was it rewarded because it sounded polished even though it failed the task?

Elektric takes a different approach. A dedicated judge model evaluates each response under a versioned Elektric judging policy, but the judge does not decide the authoritative final score. Instead, every task defines what success means, the judge evaluates those requirements individually, and Elektric’s backend applies the weights, enforces explicit error rules, rounds the result, and determines whether the response passed. The judge supplies structured assessments. Elektric owns the mathematics.

Every task defines what success means

There is no universal definition of a good AI response. A strong coding answer is not judged the same way as a strong rewrite, and a quantitative problem has different failure modes from an extraction task. Even two coding prompts can demand very different things. For that reason, every prompt evaluated by Elektric has its own rubric.

That rubric defines the criteria that matter for the exact task. Depending on the prompt, those criteria can include correctness, completeness, instruction following, fidelity to source material, reasoning, formatting, clarity, risk handling, and requirements unique to the task. A coding task might emphasize correctness, invariants, complexity, and edge cases. A quantitative task might care about the result, units, intermediate logic, and rounding. A writing task might emphasize preservation of facts, structure, audience, tone, and avoiding prohibited additions. An extraction task might care about exact fields, labels, ordering, normalization, and whether the model invented values that were never present.

The rubric can also describe acceptable variations and common failure modes. The point is that Elektric is not asking whether a response seems impressive in the abstract. It is asking whether the response satisfied the requirements of the task it was actually given.

What the judge receives

For each response, the judge receives the original prompt, the task-specific rubric, a controlled verification projection, the candidate response, and truncation information when relevant. The verification projection contains only the information needed for scoring. Authoring metadata is not simply passed through because it happens to exist.

The judge must return strict structured data. That includes one score for every rubric criterion, a short reason for each criterion score, any applicable error-rule IDs, a response disposition, whether truncation was detected, and an overall explanation. One important thing is deliberately excluded from the judge’s authority: it does not provide the final score that Elektric accepts as truth.

Instead of deciding that a response is “an 87,” the judge is asked to make narrower assessments about how well the response satisfied specific requirements. Those assessments become inputs to the scoring system rather than the final output of it.

Each criterion is scored independently

Every criterion uses the same 0–100 scale. A score of 100 means the response fully satisfies the criterion. A 75 means it substantially satisfies it but has a bounded weakness. A 50 means it partially satisfies the requirement and material work remains. A 25 represents minimal relevant progress, while 0 means the criterion was not satisfied.

Those anchors make an abstract concept like quality more concrete. Instead of asking whether an answer “feels good,” the judge considers narrower questions: Did it satisfy the correctness requirement? Did it follow the requested format? Did it preserve the source facts? Did it handle the required edge cases?

This also makes the result easier to inspect later. If a response performs poorly, we can see which requirement failed rather than being left with a single number and a paragraph of justification attached to it.

The rubric determines what matters most

Not every criterion should influence the result equally. A response can be beautifully formatted and still be wrong, and in many tasks correctness should matter much more than presentation. Each Elektric rubric therefore assigns explicit weights to its criteria, with the weights totaling 10,000 basis points.

Consider a simple illustrative rubric:

Correctness:            90 × 50% = 45
Completeness:           75 × 30% = 22.5
Instruction following:  80 × 20% = 16

Weighted raw score = 83.5
Final score = 84 after half-up rounding

This is only an example, not a production Elektric rubric. A real task can use different criteria and very different weights. What matters is that the judge does not decide how much correctness should matter relative to completeness. The rubric already defines that relationship, and the backend performs the arithmetic.

That creates a traceable path from the individual assessments to the final result. Instead of judge impression → mysterious number, the process becomes criterion assessments → explicit weights → weighted score.

Some errors are too serious to average away

Weighted averages have an obvious weakness. Imagine a response that is detailed, fluent, well structured, and mostly follows the instructions, but gives the wrong core answer. If every dimension is simply averaged together, strong secondary qualities can partially compensate for a fundamental failure.

Elektric handles this with explicit error rules. Rubrics can define major or critical failures that impose score caps such as 69, 49, 24, or 0. A cap might apply because the response gives the wrong core answer, violates a non-negotiable instruction, invents a required fact, produces an unusable exact-format result, or misses a critical safety condition.

Suppose the weighted score is 88, but the response triggers an error rule capped at 49. The response does not receive 88. Its score cannot exceed 49. If several error rules apply, the strictest cap wins.

This encodes an important principle: a polished answer cannot compensate for a fundamentally wrong answer.Fluency and correctness are not the same thing, even though language models are remarkably talented at making them look similar.

A concrete example

Suppose a prompt asks for a function that allocates integer cents using the largest-remainder method. The function must conserve the exact total, handle zero weights, reject invalid inputs, break ties deterministically, and include tests.

The judge does not reward the implementation because the code looks sophisticated. It works through the rubric. Does the algorithm produce the correct allocation? Does the output sum to the exact number of cents provided? What happens when all weights are zero? Does the implementation reject invalid input? Are ties resolved using the required deterministic rule? Were the requested tests included? Did the response introduce a prohibited shortcut?

A solution can be beautifully explained and still score poorly if it violates the conservation invariant. Conversely, an answer with slightly awkward prose can still score well if the implementation is correct, complete, and compliant. Elektric is evaluating task success, not confidence theater.

Empty responses are handled deterministically

An empty response is not treated like an ordinary low-quality answer. Elektric assigns zero scores to the criteria and a final score of 0. There is no need to ask a judge to debate how bad nothingness is.

A failed generation is also kept separate from a response that was generated successfully but happened to be poor. One is a generation failure; the other is a quality outcome. Treating those as the same thing would distort the evidence.

Judge output is validated before it can affect the score

Judge models are still models, which means they can produce malformed output. Elektric therefore treats the judge result as structured input that must satisfy a contract before scoring begins.

The backend checks that the output is valid structured data, that every expected criterion appears exactly once, that no unknown criteria have appeared, and that every score is an integer between 0 and 100. It also verifies that required reasons are present and within their allowed bounds, every referenced error-rule ID exists in the rubric, empty responses have not somehow received nonzero criterion scores, the rubric weights are valid, and the rubric version and checksum match what the system expects.

Malformed judge output is rejected rather than coerced into something scoreable. The goal is not to produce a number at all costs. The goal is to produce a number only when the inputs to that number are valid.

Pass or fail is also deterministic

Once Elektric has the criterion scores, weighted score, any applicable caps, and the rounded final score, the pass/fail decision is mechanical. The default pass threshold is 70 unless a particular rubric defines another threshold.

final score >= pass threshold → passed
final score < pass threshold  → failed

The judge does not get to write that a response “feels like a pass” despite the score. It can explain the assessment, but the scoring code decides the outcome.

What a score means

Elektric uses broad score bands to make results easier to interpret. Scores from 95–100 are exceptional, 85–94 are strong, and 70–84 are good and useful. Scores from 50–69 are partial or inadequate, 25–49 are weak, 1–24 are very poor, and 0 means no meaningful task success.

These numbers are task-relative. An 85 on a difficult quantitative problem and an 85 on a simple formatting task do not imply that the two tasks required the same capability. They mean that each response performed strongly against the rubric for its own task.

That is also why a single response score should not be confused with a model-level quality score. Model-level results aggregate evidence across many responses. The mechanism described here concerns the smaller unit underneath that aggregate: how one response becomes one score.

There is more than one score in the process

Several numbers exist during scoring, and keeping them separate matters. A criterion score is the judge’s assessment of one requirement. The weighted score combines those criterion scores according to the rubric. A capped score reflects any major error rule that limits the otherwise weighted result. The final score is the value after the scoring rules and deterministic rounding are applied.

The same kind of distinction applies elsewhere. A judge explanation is not the backend decision. An empty response is not the same thing as a failed generation. Fluency is not the same as correctness. A task-specific rubric is not the same as generic model preference. And one response score is not the same thing as a model-average quality score.

These distinctions can sound excessively tidy until you try to debug an evaluation system without them. Then they stop looking academic very quickly.

Every score carries provenance

A score is only useful if you know how it was produced. Elektric therefore records provenance around each judgment, including information such as the benchmark version, prompt ID, rubric version and checksum, judge-policy version, system-prompt version, verification version, candidate-response identity, raw criterion assessments, weighted score, applied cap, final score, pass/fail result, retry information, and judge usage and latency where available.

That means an Elektric score is not intended to be an isolated number floating in a database. It is the result of a particular response being evaluated against a particular rubric under a particular judging policy and scoring implementation. If those ingredients change, that change can be tracked.

Why not just ask for an overall score?

Because an overall score hides too much judgment inside one number. If a judge says “87,” the obvious questions immediately appear: Why 87 rather than 84? How much did correctness matter? Did the model fail a critical requirement? Was a polished but incorrect response rewarded for sounding convincing? Would another judge interpret 87 in the same way?

Breaking the evaluation into explicit criteria does not remove judgment entirely. Open-ended tasks still require qualitative assessment. What it does is constrain where that judgment operates.

The judge decides how well the response satisfied specific requirements. The rubric decides what those requirements are and how much they matter. Error rules identify failures that cannot be averaged away. Backend code performs the arithmetic and makes the final decision.

That is a much narrower and more auditable role for the judge than simply asking it to pronounce a number.

Turning quality into a procedure

Evaluating AI responses will never be perfectly objective. Some tasks have multiple valid answers, some criteria are qualitative, and some tradeoffs require judgment. The goal is not to pretend that subjectivity disappears.

The goal is to make the evaluation process explicit enough that the final result is understandable, reproducible, and constrained by rules rather than by an overall impression.

Elektric turns response quality into a transparent scoring procedure: define what matters, assess each requirement separately, apply explicit weights and error caps, and let deterministic backend code, not a vague overall opinion, produce the final score.