reasonsmith audit engine

Conformance Report

← landing · audit dossier
System under test CreditScoringPipeline
Declared Scope high-risk
Declared Domains consumer-credit
Regulation Pack table7
Executive Headline Summary
6 requirements · 4 binding: 3 observed, 1 violated · 2 interpretive: 1 unattainable, 1 not applicable · all positives observed-only
Report Provenance: Generated without an identified source commit. Command: python docs/build_example.py. Re-running that command in any checkout of this repository rewrites this page identically; test_docs_index_html_matches_the_renderer fails if it does not. No commit hash is named because a page committed into the repository it describes cannot name the commit that carries it: that commit does not exist while the page is being rendered.

Demonstration only

This is a demonstration on frozen synthetic data — not evidence about any real decision.

KEY FINDING · SEPARATE RUN · TruncatingCreditSystem

Form Completeness Does Not Imply Reason Fidelity

A second conformance run, on a different system from the one this dossier reports on above: TruncatingCreditSystem against the ecoa pack. 12 CFR 1002.9(b)(2) asks two things of an adverse-action notice, and this repository ships them as two duties. On decision APP-1042 the notice’s form is satisfied and its content is violated: the reasons stated are not all the reasons the decision’s own inference used.
ecoa_reg_b_1002_9_b_2_specific_reasons SATISFIED
Clause: ECOA / Regulation B (12 CFR 1002.9) 12 CFR 1002.9(b)(2)
Evidence strength: observed
Observed over 2 decision(s): state monitor for 'present(artifact_logs_reason_explanation) -> ( present(provenance_model_version) and present(scope_statements_local_vs_global) and not contains(artifact_logs_reason_explanation, "internal standards") and not contains(artifact_logs_reason_explanation, "internal policies") and not contains(artifact_logs_reason_explanation, "failed to achieve a qualifying score"))' satisfied at every decision step.
ecoa_reg_b_1002_9_b_2_principal_reasons_complete VIOLATED
Clause: ECOA / Regulation B (12 CFR 1002.9) 12 CFR 1002.9(b)(2)
Evidence strength: probed
Violated on 1 of 2 certified decision(s): the stated reasons are not all the reasons. On decision #1 exact inference found 5 reason(s) and the deletion probe showed the system's answer does not depend on 4 of them — C05 — Insufficient number of credit references provided; C03 — Delinquent past or present credit obligations; C04 — Too many recent inquiries on credit bureau report; C02 — Length of time credit has been established is too short. Attribution: The deleted reasons are exactly the 4 lowest-scoring of the 5, and the engine kept the top 1. This is the signature of top-k proof truncation at k=1: top-k works by discarding proofs, so the dropped reasons are lost by configuration, not by error. The missing probability mass is 0.225799. Measured against the inference artefact the system exposed, not read from its decision log.
Reason audit on decision APP-1042 (5 found · 4 deleted):
  • [stated] C01 — Income insufficient for amount of credit requested
  • [DELETED] C05 — Insufficient number of credit references provided
  • [DELETED] C03 — Delinquent past or present credit obligations
  • [DELETED] C04 — Too many recent inquiries on credit bureau report
  • [DELETED] C02 — Length of time credit has been established is too short
probe budget: 13 input(s) replayed, seed none — the proof enumeration and the deletion probes are deterministic; input space: decisions certified (2 values), facts switched off (10 values), joint deletion patterns tried (1 values), decisions whose joint search did not finish (0 values)

What this dossier does not show

This is one fixed run — the table7 pack against the committed sample log — and not the limit of what the engine does. Nothing here reads proved, because this system exposes only a decision log; the same duty reaches Z3 when a system exposes its logic, and the three rungs stand side by side in docs/three-systems.md. The engine also renders this same report for one reader at a time (--audience regulator, deployer, developer, affected-individual, auditor), ships packs beyond this one for the GDPR, ECOA and the EU AI Act — including Articles 53 and 55, the duties of providers of general-purpose AI models — and accepts engines and packs installed as plug-ins rather than vendored (docs/authoring-engines.md).

Binding Duties 4 total
3 observed 1 violated
Interpretive Items 2 total
1 unattainable 1 not applicable

Requirement Findings

eu_ai_act_art13_transparency (EU AI Act Art. 13)
Binding Scope: high-risk SATISFIED
Strength Lattice:
unattainable→ observed→ probed→ proved
Requires Signals: model_and_data_version_idsextraction_timestampdataset_snapshot_hashfidelity_coverage_metricsexplanation_scopelinkage_from_decision_to_artifact
Observed over 3 decision(s): every required signal (model_and_data_version_ids, extraction_timestamp, dataset_snapshot_hash, fidelity_coverage_metrics, explanation_scope, linkage_from_decision_to_artifact) carries a value in every record. Holds on the trace supplied; nothing here extends the claim to decisions not in it.
Scope of this positive result: this formal property was satisfied only on the supplied 3 decision records at the observed evidence rung; this run did not establish that the trace is complete, representative, or unfiltered, and it did not determine legal adequacy or compliance outside those records.
eu_ai_act_art12_record_keeping (EU AI Act Art. 12)
Binding Scope: high-risk VIOLATED
Strength Lattice:
unattainable→ observed→ probed→ proved
Requires Signals: automatic_event_logsretention_schedulesigner
Violated over 3 observed decision(s): the system declares it can emit these signals, but records carry no value for signer.
VIOLATED IN TRACE — Required Signals Absent from Decision Log:
signer
VIOLATED IN TRACE — Execution Counterexample Witness (all 1 offending record):
Trace StepDecision Record Witness
Step 1artifact_logs_decision_record: {'id': 'APP-1043', 'result': 'adverse_action'}, artifact_logs_reason_explanation: C03 - Delinquent past or present credit obligations, artifact_logs_notification_latency_days: 21, artifact_logs_counteroffer_not_accepted: 0, provenance_model_version: credit-scoring-2026.03.1, provenance_constraint_set: ['cs-rules-2026.03'], provenance_active_exceptions: [], scope_statements_local_vs_global: local, scope_statements_explanation_scope: local, scope_statements_approximation_vs_guarantee: approximation, stability_signals_artifact_drift: 0.01, model_and_data_version_ids: credit-scoring-2026.03.1 / dataset cs-data-2026.03, extraction_timestamp: 2026-07-31T22:05:00Z, dataset_snapshot_hash: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855, fidelity_coverage_metrics: fidelity: 0.9914, coverage: 1.0000, explanation_scope: local (per-applicant decision explanation), linkage_from_decision_to_artifact: AAN-2026-0731-1043 -> cs-rules-2026.03, automatic_event_logs: event_id: EVT-1043-01, status: logged, level: info, retention_schedule: retention_period: 25 months, policy: ECOA 1002.9, per_decision_reason_string: C03 — Delinquent past or present credit obligations, feature_to_named_concept_mapping: delinquency_on_file -> C03: Delinquent obligations, dpia_cross_reference: DPIA-2026-014 s.4.2 (automated credit scoring), stored_reasons_per_decision: C03 — Delinquent past or present credit obligations, model_version: credit-scoring-2026.03.1 / rules cs-rules-2026.03, score_factors: C03 0.8120; C01 0.7210, audit_ids: AAN-2026-0731-1043 / trace-9f3c1c, retention_for_regulatory_lookback: 25 months from notice date, per lender policy
gdpr_art22_meaningful_information (GDPR Art. 22 (and Rec. 71))
Binding SATISFIED
Strength Lattice:
unattainable→ observed→ probed→ proved
Requires Signals: per_decision_reason_stringfeature_to_named_concept_mappingdpia_cross_reference
Observed over 3 decision(s): every required signal (per_decision_reason_string, feature_to_named_concept_mapping, dpia_cross_reference) carries a value in every record. Holds on the trace supplied; nothing here extends the claim to decisions not in it.
Scope of this positive result: this formal property was satisfied only on the supplied 3 decision records at the observed evidence rung; this run did not establish that the trace is complete, representative, or unfiltered, and it did not determine legal adequacy or compliance outside those records.
ecoa_reg_b_adverse_action (ECOA / Reg B 12 CFR 1002.9)
Binding Domain: consumer-credit SATISFIED
Strength Lattice:
unattainable→ observed→ probed→ proved
Requires Signals: stored_reasons_per_decisionmodel_versionscore_factorsaudit_idsretention_for_regulatory_lookback
Observed over 3 decision(s): every required signal (stored_reasons_per_decision, model_version, score_factors, audit_ids, retention_for_regulatory_lookback) carries a value in every record. Holds on the trace supplied; nothing here extends the claim to decisions not in it.
Scope of this positive result: this formal property was satisfied only on the supplied 3 decision records at the observed evidence rung; this run did not establish that the trace is complete, representative, or unfiltered, and it did not determine legal adequacy or compliance outside those records.
fda_gmlp_samd (FDA GMLP agency transparency guidance)
Interpretive Domain: healthcare NOT APPLICABLE
Strength Lattice:
unattainable→ observed→ probed→ proved
Requires Signals: design_history_linksverification_logschange_control
Not applicable: this duty is about healthcare decisions, but the system's decision domain is declared as consumer-credit. reasonsmith never infers a system's decision domain, and the domain vocabulary is the pack author's rather than the regulation's — see docs/authoring-packs.md.
nist_ai_rmf_risk_evidence (NIST AI RMF 1.0)
Interpretive UNATTAINABLE
Strength Lattice:
unattainable→ observed→ probed→ proved
Requires Signals: continuous_monitoring_logsmetric_thresholds_and_alertsreviews_and_sign_offsincident_tickets
Unattainable on the evidence supplied: no record in the supplied decision trace carries a value for continuous_monitoring_logs, incident_tickets, metric_thresholds_and_alerts, reviews_and_sign_offs, and the system declared no capabilities, so nothing here can discharge this requirement. Read from that trace alone; a longer trace could show the system emitting these signals.
UNATTAINABLE AS BUILT — Missing Capability Signals:
continuous_monitoring_logsincident_ticketsmetric_thresholds_and_alertsreviews_and_sign_offs
The system declares no capability to emit these signals. No testing trace can satisfy this requirement.

Limits of this report

This report is not a compliance guarantee and is not legal advice. It assesses system capability information and trace evidence against formal specifications. Whether these findings discharge legal duties remains a determination this tool does not make and cannot make. A requirement reported without a strength was not evaluated or is not applicable, and no verdict on it should be read from this report. Recital and guidance items inform how statutory duties are interpreted but create no obligation of their own; interpretive requirements are evaluated and reported separately, and are never folded into the binding headline counts. A requirement reported not applicable was excluded on one of the independent gates. Either no regulatory class was declared for the system at all, or the class that was declared is not the one the requirement is limited to; or no decision domain was declared for the system at all, or none of the domains that were declared is one the requirement is about; or the Seoul pack self-asserted frontier_ai_status is undeclared or not-frontier. This tool infers neither the class nor the domain, and it does not infer frontier status, so an undeclared system is neither placed in scope nor cleared of the duty: read the declared scope, domain, and frontier-status lines before reading a not-applicable result. The decision-domain vocabulary is written by the pack author and by no regulation, and a duty declaring no domain reaches every system it is run against. A wrong frontier declaration remains an audited-system overclaim.