Small judgments, real work: how we use Jev in Hermes

An agent can write a useful implementation and still make a poor decision about what to do next. It can load the wrong playbook, treat a passing test as proof of an entire feature, save a temporary detail as permanent memory, or interrupt a person with something it could have handled itself.

Those are not all writing problems. Many are small judgment problems.

At Cauld & Clark, we use TypeSafe's Jev alongside the language model running Hermes. The language model investigates, plans, writes, and operates tools. Jev answers narrower questions about the state at a decision point. Our code or operating procedure decides what to do with the answer.

TypeSafe calls Jev a System One model: it consumes natural-language or structured text and returns typed judgments and probabilities rather than generated prose. It is trained for calibration, but that does not make an individual answer correct by definition.[1]

This is our integration around Hermes, not a feature every Hermes installation has enabled. Hermes is the open-source agent framework from Nous Research. Its lifecycle hooks provide places to add behavior around an agent turn.[2] Our surrounding system adds a skill-routing hook, a content-screening plugin, and reusable question sets that an agent can call when needed.

Scope and privacy. This is a snapshot of the implementation inspected on October 14, 2026. The examples and diagrams are invented teaching material, not client records or production requests. The example response below is a real Jev response to an explicitly synthetic task. Private system names, addresses, inventories, and business details are omitted.

Why give an agent a separate judgment layer?

We do not want a second general-purpose agent debating every action with the first one. We want a small interface that is easier to inspect than a paragraph of self-assurance.

Consider the difference between these two outputs:

• “The fix looks good and should be ready.”
• “Does the attached evidence directly demonstrate every acceptance criterion?” followed by a probability.

The second is not automatically more truthful. It is more usable. We can keep the question stable, compare it against labelled examples, record the answer, and make the consequence explicit. A low evidence score can mean “collect the missing proof,” not “rewrite everything.”

It also separates two jobs. The model doing the work has a large conversational context and a moving objective. A focused judge sees a deliberately prepared state and a bounded question. That gives us a separate checkpoint, not an independent ground truth. If we supply a misleading summary, the checkpoint can agree with it.

Our aim is consistent decision framing, inspectable uncertainty, and inexpensive checks where they help. We are not reporting a controlled productivity or cost-saving benchmark here.

What a Jev call actually looks like

The HTTP interface takes model, state, and a map of questions. Answers return under the question IDs. The IDs are for application code; the meaning of the question belongs in its instructions and criteria.[3]

We use three answer types:[4]

Scroll horizontally to see all table columns →

A Noul near 0.5 means uncertainty about yes versus no. It does not mean “moderately true.” Choice and Score confidence summarize the distribution; Noul has no separate confidence field.[5]

Here is the complete request body from a small synthetic check we ran while preparing this article. The fictional task has two acceptance criteria, but only one has evidence.

json · Scroll horizontally for long lines →

{
  "model": "jev-latest",
  "state": {
    "example": "Invented editor task; no production data",
    "acceptance": [
      "Saving a non-empty title persists the new title.",
      "An empty title shows an error without saving."
    ],
    "evidence": [
      "A test saves 'Sample heading', reloads, and reads 'Sample heading'.",
      "No empty-title test or UI recording is attached."
    ]
  },
  "questions": {
    "fully_evidenced": {
      "type": "noul",
      "instructions": "Does the supplied evidence directly demonstrate every acceptance criterion?",
      "criteria": {
        "true": "Every criterion has direct evidence.",
        "false": "At least one criterion lacks direct evidence."
      }
    },
    "evidence_gap": {
      "type": "choice",
      "instructions": "Which acceptance criterion lacks direct evidence?",
      "criteria": {
        "saving": "Persisting a non-empty title is not evidenced.",
        "empty_title": "Rejecting an empty title is not evidenced.",
        "none": "Both criteria are directly evidenced.",
        "both": "Neither criterion is directly evidenced."
      }
    }
  }
}

The actual answer portion was:

json · Scroll horizontally for long lines →

{
  "model": "jev-1.13.0",
  "answers": {
    "fully_evidenced": {
      "type": "noul",
      "noul": 0.05
    },
    "evidence_gap": {
      "type": "choice",
      "choice": "empty_title",
      "confidence": 1.0,
      "probabilities": {
        "saving": 0.0,
        "empty_title": 1.0,
        "none": 0.0,
        "both": 0.0
      }
    }
  }
}

This says the supplied evidence is incomplete and identifies the empty-title criterion as the gap. It does not say the product rejects empty titles incorrectly. We did not supply evidence of that behavior either way.

The practical action is to run the missing check, attach its result, and reassess. A confidence of 1.0 is not permission to ship. It describes this answer distribution, not the truth of every claim in the task.

This single request completed in 429 milliseconds, measured around the local helper process including startup and HTTP time. The API reported 495 input tokens and 69 output tokens. That is one synthetic observation, not a latency guarantee or a production benchmark. The response also shows why we retain the resolved model version even when requesting jev-latest.

Where it runs, and where it does not

An installed question set is not the same as an automated gate. A script labelled “router” is not proof that it still calls a model. We checked executable code, enabled configuration, recent logs, and the operating instructions rather than treating their descriptions as interchangeable.

Scroll horizontally to see all table columns →

There is no verified live shadow-mode model router to report in this snapshot. Earlier notes describe shadow and active experiments, but the current implementation has removed the classifier call. Proposed tuning and broader wiring remain proposals, not completed integrations.

Skill routing: suggest a playbook, not a mandate

Before an eligible turn reaches the main model, our hook gives Jev the user's message and a roster of skill names with short descriptions. It asks two independent questions: does this turn need a skill, and which skill fits best?

The first pass permits a hint when the need probability is at least 0.55 and the selected skill's probability is at least 0.30. Those are our current constants for a low-consequence suggestion. They are not recommended defaults for every classifier.

A second pass reads fuller descriptions of the leading candidates and can keep, change, or reject the suggestion. The configured default considers up to three candidates. If that verification call fails, the first-pass decision stands. If the first call fails, the hook supplies no hint and the turn continues.

The agent still receives its full skill index. The injected text explicitly allows it to ignore a poor suggestion. The router skips slash commands, turns marked as child-agent work, and specific automated continuation or completion messages. “Per-turn routing” therefore does not mean an API call on every event, nor does it mean every scheduled task is excluded.

The review process compares suggestions with later skill loads. That is a useful signal, not an accuracy label: a worker can receive a skill through preloading without making a visible load call. We do not tune a threshold simply because a log join says “unused.”

Memory: decide whether a fact belongs beyond this task

Our memory battery asks whether a candidate is durable, whether it applies across tasks, whether it appears sensitive, where it belongs, and whether it duplicates or conflicts with existing entries. Its state should contain the candidate and the relevant existing entries, not the entire conversation.

That separates questions an agent can otherwise collapse into “this seems worth remembering.” A temporary test result is not a standing preference. A reusable procedure belongs in a skill. A changed fact may need to replace an older entry rather than append a contradiction.

Our operating policy calls for this check before memory or profile writes. It is agent-gated, not mechanically enforced by a write interceptor. The battery returns judgments, not a universal storage threshold. We cannot infer complete compliance from its presence on disk.

Sensitive material also needs protection before a remote judgment request. A question asking whether something looks secret is not a reason to send known credentials to a classifier. Minimize and redact the state first; authorization to store or transmit data remains a separate requirement.

Content screening: a warning, not a security boundary

Our plugin evaluates selected fetched-text and browser-result tools before their results enter the agent's working context. It asks about instructions aimed at the agent, attempted manipulation, and potential harm if embedded instructions were followed. A further question asks whether the content looks like ordinary data, though that answer is not used in the current warning rule.

The active rule flags content when manipulation probability reaches 0.6, harm reaches 2.0 on a 0–3 scale, or instruction probability reaches 0.6 while harm is at least 1.0. On a flag, it preserves the content and appends a warning to treat embedded directives as untrusted data.

Coverage is deliberately bounded: selected tool names, text results, size checks, and a truncated input. It is not a scan of every byte the agent can encounter. A timeout leaves the original result unchanged.

The review journal records benign documentation and normal browser controls being flagged. That matters. Instructions in a tutorial can look like instructions to an agent without being an attack. A warning system that cries too often trains its reader to disregard it.

Tighter warning criteria and a return to shadow mode have been proposed. The active code we inspected still uses the rule above, so we do not describe those proposals as deployed. Regardless of the warning, fetched content stays untrusted. A missing banner is not a clean bill of health.

Engineering triage: read what the evidence supports

We keep reusable question sets, or batteries, for several engineering decisions:

• Fix triage: Is this a local patch, a structural fix, or a structural attempt with uncovered sites? Does it touch protected behavior? What could be affected if it is wrong? The state includes the issue, proposed change, and relevant code context.
• Evidence checking: Does each acceptance criterion have direct, reproducible evidence? Is any part supported only by an assertion? The state includes the criteria and actual commands, test results, logs, or other artifacts.
• Flake triage: Does the failure look intermittent, deterministic, environmental, or unclear? Should we retry or investigate? The state includes the failure excerpt and recent history, not merely “CI is red.”

These are available to the agent at decision points. They are not automatically running after every edit, and they do not share one magic pass threshold. A recommendation to retry a test is a different consequence from dismissing a failure or declaring a change ready.

A high “structural” probability cannot prove that an omitted caller was fixed. “Flake” cannot excuse repeatedly failing evidence. The judgment helps choose the next investigation; source inspection and real execution still settle the acceptance criteria.

Escalation: protect attention without hiding a real blocker

The escalation battery asks whether a situation needs the owner's action, how urgent it is, and whether it belongs in a push notification, a digest, or the log. A separate failure-classification battery distinguishes routine watcher noise, a notable event, and an actionable signal.

The notification-gate implementation reads a proposed message's source, subject, and body. Its code allows a send if any of these are true: owner-action probability is at least 0.5, urgency is at least 1.5 on a 0–2 scale, or the selected channel is push. On transport failure it allows the notification rather than silently losing a possible blocker.

That is a deliberately different policy from the content warning. The cost of suppressing a genuine request for human authorization can outweigh the cost of an unnecessary notification.

There are important limits to the deployment claim. The gate defaults to active and historical scored sends exist, but the inspected recent logs contain no scored traffic. Some workflows send directly, outside the gate. We cannot claim that every Telegram notification passes through Jev, that every watcher uses failure classification, or that the suppression path has been thoroughly validated.

Issue routing: the experiment we should not describe as current

We have a plan-then-execute workflow for issue delivery: produce a plan, verify it, then start implementation. Earlier versions used Jev judgments about work type, difficulty, visual evidence, and potential impact to choose a model and effort for a phase.

The current runner still resolves a route for each phase, but its resolver returns a fixed operation slot. It makes no Jev request. Jev also does not decide whether to skip plan verification and proceed to execution; that sequencing belongs to the workflow.

The old model-routing battery remains available. Some descriptions still mention the earlier experiment. Neither makes model selection active or shadow today. This was one of the most useful findings from preparing the article: integration inventories need to follow the executable path, not the most ambitious description of it.

The design rules we keep coming back to

Probabilities are the interface; thresholds are ours

Keep the raw answer separate from the action. A probability describes a judgment. A threshold encodes what we are willing to do with it.

Suggesting a skill, warning about a page, discarding a memory candidate, suppressing an alert, and approving a destructive action have different failure costs. They should not inherit one number merely because every output fits between zero and one. TypeSafe's guidance likewise treats thresholds as application-specific decisions to evaluate against the domain and its consequences.[5]

For authorization, exact invariants, or destructive operations, a model score is not sufficient. Deterministic permissions, required approvals, and verification remain in force. Confidence is not permission.

One atomic judgment per question, not one vague verdict per workflow

“Is this safe, correct, well-tested, and ready to ship?” bundles several properties whose failures require different responses. Ask narrower questions and combine their answers explicitly.

We sometimes describe this as one atomic judgment at a time. It does not mean one HTTP request for every question. Independent questions over the same state can share a request; they are evaluated independently and cannot consume each other's answers.[6] The synthetic example above does exactly that.

Use another request when an earlier answer changes the evidence or candidates, as our skill verifier does. Do not pay for a serial call merely to reproduce logic code already knows.

The state is part of the judgment

Before changing a threshold, inspect what the judge received. Was the relevant evidence omitted? Did truncation remove the important part? Are the choice options complete? Does the rubric ask about the actual task, or an older interpretation of it?

An evidence check over an agent-written summary is not equivalent to a check over the underlying test results. A sensitive-content classifier is not a substitute for data minimization. A choice model cannot select a missing candidate.

Our current wrappers impose size bounds; those bounds are operationally useful but create blind spots. The better long-term discipline is to select the minimum relevant evidence deliberately rather than assume the first slice of a large document contains it.

Fail open to the existing workflow, not past its safeguards

Our advisory integrations are designed not to make Jev availability a prerequisite for doing useful work:

• Skill router unavailable: continue without the hint.
• Content screen unavailable: return the fetched content without an extra warning.
• Notification gate unavailable: allow the existing notification path.
• Agent-invoked battery fails: note the failure and use the normal investigation procedure.

These fallbacks differ because the consequences differ. None authorizes a forbidden action, makes an incomplete acceptance criterion pass, or turns untrusted text into instructions. “Fail open” here means preserve the baseline workflow and its safeguards, not turn every uncertain decision into yes.

Keep policy constants together, and measure the whole path

Question wording, rubric definitions, thresholds, and fallback behavior should be easy to locate and review. Our components group their questions and thresholds near the top; the reusable batteries separate question definitions from the transport helper.

Our operating notes carry a historical estimate of roughly 1.1 seconds and US$0.0003 per call. Treat that as a documented estimate for the earlier workload, not a current rate or verified billing result. The inspected decision logs do not contain per-request charges. Input size, question count, verification passes, and service conditions all matter.

Recent successful skill-hint injections in the log window we inspected had a median elapsed time of 1,025 milliseconds, including the hook's work and optional verification. That is not the same measurement as a single API request. The synthetic check's 429 milliseconds is not evidence that every integration is faster now. We would need comparable workloads and actual usage pricing to make that claim.

Calibration is a debugging loop, not a threshold hunt

The most useful calibration lesson from our application work is to inspect the exact request before blaming the judge. A model can correctly reject a claim when the supporting fact was left out of its state. It can also hover around uncertainty because the question merges unrelated decisions.

Our method is practical:

1. Capture the serialized request body at the transport boundary. Exclude credentials and headers. Keep only authorized evidence, with retention appropriate to the data.
2. Write labelled synthetic cases. Include ordinary successes, missing evidence, ambiguous cases, and clear escalations. Have a person review the expected decisions and reasons.
3. Replay old and new question wording with the same policy thresholds. Change one factor at a time so a better result has an identifiable cause.
4. Inspect mistakes separately. A false proceed and an unnecessary handoff are not interchangeable errors. Track both, alongside latency and uncertain cases.
5. Keep a held-out set and rerun the real workflow. A cleaner classification table is not proof that the application now behaves correctly.

Do not lower a safety floor just to turn an evaluation green. First check the fixture, evidence selection, question meaning, and policy composition. If a threshold genuinely needs to change, make that a reviewed policy decision.

Our router's operating review also requires repeated same-class observations and consideration of counterexamples before proposing a tuning change. A single awkward result is a case to inspect, not a mandate to move a constant.

This is operational calibration work, not a published proof that all of our probabilities are perfectly calibrated. Broad calibration claims need representative labelled outcomes, not a few convincing examples.

A mundane outage lesson: the HTTP client matters

Our operating journal records an outage in which Python callers received Cloudflare 403 / error 1010 responses while command-line curl checks worked. The relevant difference was the client signature: the default Python urllib User-Agent was being rejected.

The recorded fix was an explicit application User-Agent. The current Jev request builders we inspected include one. The lesson is specific, not a claim that Cloudflare always blocks Python: reproduce a failing request through the same client, headers, and transport path as the integration.

A model can be healthy while the wrapper is broken. An HTTP rejection is not low confidence. Logging those as the same “unsure” verdict would contaminate both operational diagnosis and calibration data.

When we would not use it

We would not call Jev to count files, compare exact identifiers, enforce access permissions, or evaluate a rule ordinary code already expresses precisely. We would not ask it to write a plan or implement a fix. Those are different jobs.

We would also avoid a remote judgment when the necessary data cannot be sent to that service, when the evidence cannot fit without losing essential context, or when an extra dependency adds more delay than the decision is worth.

The limitations are part of the design:

• Typed output makes the response consumable, not necessarily true.
• A narrow judge can still inherit bias or missing evidence from its input.
• A changing model alias or changing skill roster can change outcomes under the same threshold.
• An advisory warning is not an injection-proof security boundary.
• An agent-invoked procedure is not proof that every operation passed through it.
• A successful synthetic example is not an accuracy study.

Start with one decision you can inspect

For a reader building a similar integration, we would start with a reversible suggestion or an evidence check, not an autonomous approval gate.

Define one decision and what a wrong answer would cost. Build a compact synthetic state. Write narrow questions with clear alternatives, including “none” or “unclear” where appropriate. Use the API to inspect the full distributions before collapsing anything into a boolean.[3][4]

Run it in shadow first: record the judgment and proposed action without changing the existing workflow. Compare those records with reviewed outcomes. Decide the threshold and fallback in code, then enable only the bounded consequence you have actually tested. TypeSafe's building guide provides the underlying pattern: deterministic control flow, focused judgments, and explicit composition.[6]

Inside Hermes, a hook is a useful integration point when the decision belongs around a turn.[2] A callable battery is simpler when the agent should consult it only during a review. Those are different levels of enforcement; choose deliberately.

The result we want is not an agent that asks another model for permission to think. It is an agent system with a few important decisions made explicit: what evidence was available, what question was asked, how uncertain the answer was, and which rule turned that answer into the next step.

Let the language model do the work. Give small judgments a clear interface. Keep responsibility for the action in the system around both.

Implementation and source notes

Implementation claims come from a read-only review of our active hook, plugin, notification gate, lane resolver, battery definitions, relevant configuration, decision logs, and operating journals. Current executable behavior takes precedence where older documentation disagrees. Recent log observations refer to the inspected October 13–14, 2026 window, not an exhaustive audit of every agent session.

We made one live Jev request for this article, using only the invented editor state printed above. No private task text was submitted for that demonstration. Public references describe the API and framework, not an independently audited result for our deployment.

Sources

[1] https://docs.typesafe.ai/concepts/system-one [2] https://hermes-agent.nousresearch.com/docs/user-guide/features/hooks [3] https://docs.typesafe.ai/api [4] https://docs.typesafe.ai/primitives [5] https://docs.typesafe.ai/confidence [6] https://docs.typesafe.ai/concepts/how-to-build-with-system-one