The wrong tool is not a acceptable approach

"The Machine of Trust: Case Study in Banking Genetic Development" — general view before technical deepening: the problem, the Silent Crash, the barrier of Four guards and what the numbers prove -- and what they don't prove.

Build routing brain from a digital bank service agent. Before any expensive LLM call, the system decides the cheapest and safest way to answer each message.

The three pillars of architecture

The non-negotiable restriction of the domain

The Border: production does not see the template

What separates adjustment engineering from template is one line: expected_tool lives in evaluation layer and never in production logic.

grep -n "expected_tool" candidate_starter/retrieval.py candidate_starter/router.py candidate_starter/taxonomy.py
# Saída: zero resultados.

In an intermediate iteration (commit c8f30d2), the project showed: 54 tests on green, 100% router, 100% Hit Rate@2, zero incorrect executions and status approved with praise. An external review, however, requested a collision test between reading and writing vocabulary.

Customer Query: "What's the email registered to my account?" — a READING.

None of the three old guards took it: the mistake It's not a tie. — is a decision confident and wrong. This illustrates the central thesis of the pillar Trust Engineering: the most dangerous error in AI does not generate code exception. He passes the green and makes a ruinous decision with loose margins.

When the problem appeared, the first attempt was to adjust the combination of weights between lexical search and intent search (score = (1−α)·lexical + α·intent). The mathematical demonstration below proves why this is impossible:

Query: "Quero mudar o e-mail vinculado a minha conta" (ESCRITA)

consultar_email_vinculado_conta  -> lexical = 0.8457 | intent = 0.4400
atualizar_email                  -> lexical = 0.1983 | intent = 0.3200

Para a escrita vencer por ajuste de α:
(1 - α)·0.1983 + α·0.3200 > (1 - α)·0.8457 + α·0.4400
α·(0.3200 - 0.4400 - 0.1983 + 0.8457) > 0.8457 - 0.1983
0.5274·α > 0.6474  =>  α > 1.227 (IMPOSSÍVEL, pois α ∈ [0, 1])

The structural lesson: Reading and writing on the same object share all nouns (e-mail, conta, cadastro). The difference between consultation and change is one word: the verb. In a bag of words with TF-IDF, this verb is diluted between ten tokens. The direction of the operation is not recoverable by textual similarity — it needs to be a semantic restriction declared.

Whoever decides on this pipeline is not a language model. There are five classic statistical and statistical techniques information retrieval, each responding to a concrete problem of banking service. Each technique below has two layers: the concept, in simple language, and the Python excerpt that implements it.

The snippets are from the branch feature/solucao-enxuta (commit) e2dcd7f), the simplified version: four modules and an execution script, about 560 lines, without taxonomy.py And no ADRs. Where the main does different, a note warns. The numbers come from the output of python -m candidate_starter.run_case in this branch and offline recounts on the same report. It's not the numbers of main quoted in the rest from the page.

1 · Supervised classification: TF-IDF and logistic regression

Operational problem. Greetings and FAQ questions ("What time is it?") do not require LLM or tool. Answer this locally has zero cost of LLM. The router needs to separate FAST_PATH of AGENT in milliseconds, before any man calls.

Concept. Supervised rating is to teach with examples already labeled: here, 53 messages marked FAST_PATH (25) or AGENT (28) The model learns what separates the two classes and applies this to new messages, in two steps: turn text into numbers and learn a boundary between classes.

O TF-IDF gives each term a weight that grows with its frequency in the message (TF) and decreases with the number of messages from the set containing it (IDF). "De" appears in almost everything and weighs little; "block" appears in few and weighs much. With n-grams from 1 to 2 terms, pairs like "second way" also saw terms. Each message becomes a sparse vector (almost everything zero) length 1.

A logistic regression transforms a weighted sum of the terms into a probability, between 0 and 1, of a message be AGENT. A regularisation L2 penalty large weights, for model not decorate 53 examples; C control force (1.0 is the scikit-learn pattern and is the value used). It is a linear and light classifier, and each term has a coefficient that you can read, which matters in an audit.

normalize() is the preprocessor vectorizer, then the same treatment (low, without accent, domain variants as "second way" → segundavia, no score) in practice and consultation. If the two were normalized differently, the vocabulary learned would not marry the Real question.

Measured in this branch:

• Vocabulary of 378 terms (unigrams and bigrams) on the 53 training examples.
• Router 30/30 in benchmark; confusion matrix FAST_PATH 10 · 0 / AGENT 0 · 20.
• predict(), already with the calibration of technique 2: median of 2,2 ms and p95 of 4,4 ms in 200 local calls (Windows, scikit-learn 1.9).

2 · Probability calibration: Platt scaling and cross-validation stratified

Operational problem. Guard G1 threshold of 0.75 (section The Barrier) is only a policy risk if "0.75" means "the model fits in about 75% of cases". Regular logistic regression, with 53 examples and 378 terms, shrink the weights and return shy odds, close to 0.5, even when the class is right. No calibration, in this branch, 20 of the 20 messages AGENT Benchmark is below 0.75 (the highest confidence is 0.713), Although the router hits them all. The threshold would no longer separate a good decision from a bad decision and only rule out coverage.

Concept. Calibrate is to correct the number scale without changing what the model decides. O Platt schedule trains a second logistic regression of a single variable on the gross score f The model. Parameters A and B are estimated by maximum likelihood in forecasts made on examples that the template no saw in practice (out-of-fold: predictions about the part of the data left out in each round). To estimate them in the very examples of training would measure the model memory, not the ability to generalize.

A cross-validation stratified divides the 53 examples into 3 folds that preserve the proportion of classes; each fold serves once as an unseen set. No shuffle (shuffle=False), the folds are the same in Any machine, and the tests are reproducible. Code does not call StratifiedKFold directly: pass cv=3 to a classifier makes the scikit-learn use this divider (checked in version 1.9). The number of folds come from the smallest class: 3 if it has at least 3 examples, 2 if it has 2, and without calibration below that, for the fit() do not break into tiny test sets.

Why sigmoid and not isotonic. Isotonic calibration is nonparametric: with few data it overfits and saturates confidence around 1, the opposite problem. The sigmoid has two parameters and is stable. A main measured this comparison in the ADR-007.

Measured in this branch:

• No calibration, 20 of 20 messages AGENT are below 0.75. With calibration, 1 of 20 (0,732).
• In the entire benchmark, G1 barred 2 messages: "I changed number, I need to update the registered phone" (AGENT, 0.732) and "What documents do I need to open an account?" (FAST_PATH, 0.718). In this branch, G1 is for both routes.
• The least confidence accepted is 0,753: 0.003 slack on the threshold. Calibration corrected the scale, but the margin remains narrow.
• The repository does not measure the calibration itself (Brier score, expected calibration error, reliability curve), only its effect on the threshold. With 20 queries AGENT, a reliability curve would have few points per range.

3 · Vector recovery: Cosine similarity, multi-field score and capacity collapse

Operational problem. The catalog has 285 tools. Putting all in the prompt of an LLM bursts the context, confuses the model and cost expensive, so the retriever chooses the 2 most relevant before any call. And the catalogue has duplicates: 8 different tools make "get the invoice".

Concept. A Cosine similarity measures the angle between two vectors: 1 when they have the same weighted vocabulary, 0 when they do not share any term. As TF-IDF already delivers vectors of length 1, cosine is only the scalar product, that is, a multiplication of sparse matrix between the query and the 285 tools.

The score combines two fields with equal weight. The tool document is nome + descrição + categoria; Intention field is nome + descrição. Keep fields separate, instead of concatenating everything, exists for that L2 normalization does not dilute the original terms when adding synonyms to the same text (the ADR-006 ed main document this problem). A list of functional words (a, de, para, quero...) comes out of vocabulary first, for a Query short not only score by prepositions.

A limit on this branch. The two fields come out of the same text: they differ only by category. Measured, the correlation between two similarities is at least 0.97, and the top-1 of the combined score is identical to that of the single field in the 30 queries. Here, α = 0,5 It hardly changes the result. A second field only helps when it brings new information, like the handwritten glossary of main.

O capacity collapse groups duplicates with an embedded dictionary (CANONICAL_TOOL_MAP: 49 tools in 12 capacities) and holds the highest score per group (max-pooling, a maximum aggregation). Like this. k=2 returns two distinct capacities, not two variants of it. Then the code orders by score decreasing and cuts into k. Candidates with score 0 are discarded: a query without any common word with the catalog returns empty list. Empates follow the order of the catalogue, because the sorted Python is stable. This is deterministic for the same file, but it is not an explicit tiebreaker (a main Untie by the name of capacity).

Measured in this branch: recounting by capacity (get_canonical_name on the result), expected capacity is in the top-1 in 19 of 20 transactional and top-2 in 20 of 20. The report Print shows another number. The explanation is in technique 5.

4 · Decision under uncertainty: thresholds, relative margin and steering guard

Operational problem. Choosing the best tool is not enough: the system needs to know when no execute. A mistake that passes in silence (see The Silent Crash) costs more than a confirmation question.

Concept. Instead of always acting on the best candidate, the pipeline tests three statistical signals, in order, and only Execute if everyone passes. None of them read the expected_tool the gauge.

• G1 · confidence: the calibrated probability of the router is ≥ 0.75.
• G2 · minimum relevance: the top-1 score is ≥ 0.10, a floor against weddings by chance.
• G3 · relative margin: (s₁ − s₂) / s₁ ≥ 0,25. Measure how much the first candidate wins the second in proportion of the score itself. It is relative not to depend on the scale: a difference of 0.03 weighs different when the Top-1 is 0.18 and when it's 0.80.

The margin in three real issues, all diverted by G3 in the benchmark:

The third failure by 0.019: 0.25 is a policy, not a constant of nature. In this branch the three thresholds are literal in code; main are named constants, and the margin is harness parameter.

G4 · steering guard. It is not a measure of similarity, and the reason is statistical: reading and writing about the even given divide almost all the terms (the section Why no weight solves shows the arithmetic). Then the direction enters as a declared sign: Query verbs and keywords say whether it is reading or writing, and the tool name prefix (consultar_, atualizar_...) says what kind she is. Reading Query with writing tool: zeroed score.

Measured and limited in this branch:

• Protected reading. "What's the email registered to my account?" confirmar_email_cadastrado (reading) first. atualizar_email doesn't appear among the first three.
• Writing→reading is not treated. The lean specification expected to lower the reading tool, but the code does nothing in this Meaning. "I want to change the email linked to my account" executes consultar_email_vinculado_conta (score 0.664): It doesn't change anything, but it doesn't comply.
• You only see what the name reveals. By prefix, 82 of the 285 tools are read, 85 are write and 118 tools (41%) they do not reveal direction by name and pass without restriction.
• Reading grouped under writing. The collapse dictionary puts consultar_email_vinculado_conta and confirmar_email_cadastrado, reading tools, under atualizar_emailThat's writing. A main prohibits this by invariant: reading and writing do not group.

5 · Evaluation and economics metrics

Operational problem. Without measurement, "works" is opinion. And a misread metric is worse than none: cost savings It goes up when the system abstains further, even if it answers less.

Concept. A accuracy is the proportion of correct answers: correct answers divided by the total. A confusion matrix 2×2 separates the two errors possible, which cost different things: send an FAQ to the agent spends LLM for nothing; send a transactional request to FAST_PATH gives the customer a generic response in place of the operation. The code initializes all cells with zero, then the structure is complete even without cross errors.

O hit rate@k is the fraction of the queries in which the expected tool appears among the first k. Not Precision@K classic (relevant between k, divided by k): with one relevant tool and k = 2, maximum precision@2 would be 0.5. The function if flame compute_precision_at_kBut what she calculates is hit rate.

A economy is the percentage reduction against baseline (send all message to the LLM, with all the tools in the prompt), with guard to baseline zero. In MVP, costs and latencies come from mock constants (US$ 0.03 per baseline call, US$ 0.01 per called the agent), so the percentage measures the pipeline structure, not a real price.

Output of python -m candidate_starter.run_case in lean branch, 30 queries:

Because 26.3% is not the quality of recovery. O ToolMatch.name which the retriever returns is the name of variant that married (e.g. verificar_saldo_conta_corrente), and expected_tool the dataset is the name canonical (consultar_saldo). The string comparison fails even with the right ability. Recounting for capacity, the result is 19/20 and 20/20. The denominator is also 19, not 20: the Query barred by G1 never reached the retriever and It's out of line.

Because 82.2% is larger than 77.8% of the rest of the page. Five messages were diverted by abstention and did not call LLM. It's the the same effect as described in Economy without success rate is disguised risk: the economy rises when the system does Less. The execution that comes out of the expected capacity is "Send the pdf of my current invoice" → reenviar_fatura_email, that the dictionary does not group into consultar_fatura. And the right 15 include the email query, whose reading tool dictionary groups under atualizar_email. The lean branch does not calculate incorrect executions or the balance point of the human service cost: these two accounts only exist in the main.

lean branch × main, technique by technique

The initial diagnosis of low recall (35% Hit Rate@2) seemed a failure of the model. The tool register examination revealed the real cause: semantic duplications accumulated by different teams over years.

consultar_fatura                  -> "Gera o PDF ou linha digitável da fatura do mês atual."
gerar_linha_digitavel_fatura      -> "Gera a linha digitável da fatura ... para pagamentos."
enviar_pdf_fatura_atual           -> "Envia o PDF da fatura atual ... para o cliente."
consultar_valor_fatura_mes_atual  -> "Consulta o valor da fatura ... do mês atual."

For a simple lexical retriever, the hyperspecific variant always wins the canonical because it repeats word for word of the query. The solution was to introduce the concept of Canon Capacity with Collapse: the retriever groups the variants under the same capacity and returns in k=2 two capacities different. The specific variant that married is preserved in matched_variant as an audit trail for runtime.

Development didn't follow a straight line. Two rounds were driven by external adverse review, where each note generated a diagnostic cycle, ADR measurement (Architecture Decision Record) and extended testing suite. The third had no reviewer: appeared because the steady state has been tested instead of green.

The origin that contaminated the measurement itself

The metric report is versioned — and a versioned JSON ages silently: the code changes, the file stays, and those who read believe in numbers that the current commit no longer produces. The project has a test that rewards decisions against the current code. This test had three defects of the same family, all with the report soiling the measurement that should audit it:

1. Unsatisfactory assertion. Requirement snapshot.git_commit == HEAD. How to commit the report creates a new commit, snapshot always points to the previous one — the assertion could never pass. The real auditable property is another: between the commit which generated the report and the HEAD, nothing outside of reports/ changed — and the failure becomes name files Late.
2. git_dirty self-sucking. build_snapshot() counted reports/ on the dirty tree check. As the report is written by the function itself, from the second execution onward it was declared irreproductive because of the file that it had just generated.
3. _git().strip() ate the status prefix space. The filter sliced line[3:] Do --porcelain, but .strip() at the exit removes the initial space from the first line: the path exited displaced (eports/...), the filter never married and the test He would jump again in silence. Solved by delegating the filter to git itself (pathspec ':(exclude)reports'), without parsing.

A green suite with 1 skipped He looked healthy. It wasn't — the skip It was a test giving up silently, and it only appeared because the stationary state was tested (regenerate the report, rotate the suite again) instead of accepting the green. It is the third instance of the thesis of this page: Green metrics are not coverage.

Principles governing decisions

• Measure before editing: No changes were made without registering the metric table before/after ADR.
• Never move the beam: Easy solutions (such as climbing probability parameters to mask errors) were summarily rejected.
• No Effect Tests: Tests that ensure that mock_tool_execution is never called If any guard fails.
• Test steady state: It is not enough for the suite to pass once — it must continue passing after the system writes its own artifacts.

Eight test files divided by security and integrity invariants. The part more valuable does not check a returned value; check that Nothing happened.:

def test_a_read_request_can_never_resolve_to_a_write_capability(retriever):
    """Antes da guarda de direcao, "Qual e o email cadastrado na minha conta?"
    devolvia atualizar_email (ESCRITA) em 1o lugar com margem relativa de 0.47 —
    acima do limiar de 0.25: a guarda de margem NAO pegava."""
    for query in read_requests:            # 5 consultas puras
        for m in retriever.search(query, k=3).matches:
            assert CAPABILITY_MODE.get(m.name) != "write"

Test Suite Categories

• No effect: mock_tool_execution is never called when any guard fails (executed.assert_not_called()).
• Taxonomy Integrity: 7 authorship invariants (existence in the catalogue, absence of collision, reading, writing, declared and inherited direction).
• Generalization without Adjustment to Gabarite: 12 paraphrases written with totally absent vocabulary from the official dataset (9/12 on top-1, 11/12 on top-2).
• Stability and Origin: Shuffle the order in the catalogue does not change any decision; the JSON report corresponds exactly to the commit and current code — and this check Can't jump (it was the fault of round 3).

Publishing limitations is what separates advertising engineering. The 7 restrictions below are registered in the README of the official repository:

1. Approved in this benchmark, not for live banking production: 30 queries, 53 training examples and molded tools. Real traffic with opponents requires layers of MFA, authorization and idempotence outside this scope.
2. The steering guard only protects what is stated: 59 of the 285 tools have mode (14 canonical capacities + 45 variants). The others 226 They pass without driving restrictions. The right architectural place for mode is the register itself — is metadatad from the endpoint, not taxonomy.
3. Paraphrased lexical recovery ceiling: The recall of pure paraphrases without keywords reaches 9/12 in the top-1. The next step requires dense/reranker embeddings.
4. Manually maintained taxonomy: 14 capabilities and 45 variants configured in Python code over 285 tools.
5. Guards 2 and 3 are no longer exercised by the official benchmark: With 20/20 and zero abstentions, they are proven only by tests of no effect. A benchmark that never triggers the security network doesn't show it works.
6. Router calibration break is narrow: A Query AGENT less confident registers 0.7674 against the threshold of 0.75 — 0.017 Off duty, registered as a liability. (On the other hand the margins of the right decisions lie between 0.592 and 0.741 against the threshold of 0.25: this, yes, loose.)
7. Human fallback cost is premise, not measurement: The reliable number is the equilibrium point, which is derived from measurements.