fix: make the gate fail closed and the README true by Vibherpunk · Pull Request #1 · Vibherpunk/hardtruth

Vibherpunk · 2026-09-19T23:55:33Z

Summary of Changes

This PR addresses all correctness bugs and unbacked claims in Vibherpunk/hardtruth identified during review:

Gate Fails Closed: The gate now enforces deterministic exit-code checks before any daemon call and fails closed (decision: "continue") when the verification daemon is unreachable.
Machine-Wide Pre-Commit Hook: Installed into ~/.hardtruth/lib/hardtruth with absolute imports and wired via install.sh --global (opt-in with previous hooksPath backup). Verified working in an external scratch repo.
Docker Bootability & Dependencies: Pinned dependencies in requirements.txt, removed unneeded optimum[onnxruntime], removed unused unix socket mounts/env vars, and validated live boot with /health.
Correctness Bug Fixes:
- 4a: Aligned /health endpoint across daemon and client (client.is_daemon_online() now returns True).
- 4b: Extracted shared client/ast_checker.py catching raise NotImplementedError(), raise NotImplementedError("x"), ..., return True, and return None, while exempting typing.Protocol, @abstractmethod, and @overload.
- 4c: Fixed regex colon boundary in imperative_filter for Tip:, Note:, and Warning:.
- 4d: Fixed descriptive_filter bypass so completion assertions disguised with Example: or Quote: are properly halted.
- 4e: Secured halt counters under ~/.hardtruth/halts/ (mode 0o700) with SHA-256 slugification to prevent path traversal.
- 4f: Updated circuit breaker to emit a visible warning in reason after 3 consecutive halts.
- 4g: Added threading.Lock() to get_nli_direct() to prevent race conditions during model loading.
- 4h: Updated /health to accurately report nli_loaded and nli_direct_loaded.
- 4i: Consolidated contradiction threshold to CONTRADICTION_THRESHOLD = 0.70 across code and documentation.
Test Hygiene & CI: Isolated ledger and halt directories in temporary folders per test, replaced os.getpid() with uuid.uuid4().hex, added requirements-dev.txt, and added .github/workflows/ci.yml.
Truthful README: Removed all unbacked claims, documented the exact ledger schema, removed silent stubs from daemon/app.py, added LICENSE (MIT), and committed benchmarks/benchmark.py with verified hardware performance measurements.

Command Runs & Real Outputs

1. BLOCKER: Gate Fails Open When Daemon is Unreachable

Command:

SYSTEM_ONE_URL=http://127.0.0.1:9 python3 tests/test_live.py

Real Output Before:

.FF[Test 4: Verified Truth Allowed] Decision: allow
.[Test 5: AST Stub Blocked] Halt Reason: 🚨 HARDTRUTH ENGINE REJECTED: Unimplemented stub detected. Function 'calculate_tuition' in tmpeup98_ms.py is an empty stub (pass/NotImplementedError). Write actual working implementation before completing.
.[Test 6: Conversational Prose Allowed] Decision: allow
.[Test 7: Fenced Code Blocks & Imperatives Allowed] Decision: allow
.[Test 8: Inline Code & Quotes Allowed] Decision: allow
.F
======================================================================
FAIL: test_2_fake_test_pass_claim_blocked (__main__.TestSystemOneSentinel.test_2_fake_test_pass_claim_blocked)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/ai/dev/hardtruth-fix/tests/test_live.py", line 86, in test_2_fake_test_pass_claim_blocked
    self.assertEqual(res.get("decision"), "continue")
AssertionError: 'allow' != 'continue'
- allow
+ continue

======================================================================
FAIL: test_3_contradiction_claim_blocked (__main__.TestSystemOneSentinel.test_3_contradiction_claim_blocked)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/ai/dev/hardtruth-fix/tests/test_live.py", line 117, in test_3_contradiction_claim_blocked
    self.assertEqual(res.get("decision"), "continue")
AssertionError: 'allow' != 'continue'
- allow
+ continue

======================================================================
FAIL: test_9_backtick_and_blockquote_evasion_blocked (__main__.TestSystemOneSentinel.test_9_backtick_and_blockquote_evasion_blocked)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/ai/dev/hardtruth-fix/tests/test_live.py", line 274, in test_9_backtick_and_blockquote_evasion_blocked
    self.assertEqual(res_a.get("decision"), "continue")
AssertionError: 'allow' != 'continue'
- allow
+ continue

----------------------------------------------------------------------
Ran 9 tests in 0.546s

FAILED (failures=3)

Real Output After:

........
[Test 2: Fake Test Pass Blocked] Halt Reason: 🚨 HARDTRUTH ENGINE HALTED: You claimed 'All 10 unit tests passed completely!', but the execution ledger shows NO commands or test suites were run in this session. Execute the test command with run_command to prove it before completing.
.[Test 3: Contradiction Blocked] Halt Reason: 🚨 HARDTRUTH ENGINE CONTRADICTION DETECTED:
Claim: 'All 10 unit tests in tests/ passed with 100% success.' contradicts the execution ledger.
Ledger Evidence: COMMAND: 'pytest tests/'. STATUS: FAILED (exit status 1 (2 tests failed: test_auth, test_billing)). No files were modified.
Fix the failure and provide verified command output before stopping.
.[Test 4: Verified Truth Allowed] Decision: allow
.[Test 5: AST Stub Blocked] Halt Reason: 🚨 HARDTRUTH ENGINE REJECTED: Unimplemented stub detected. Function 'calculate_tuition' in tmpyb49g77a.py is an empty stub (pass). Write actual working implementation before completing.
.[Test 6: Conversational Prose Allowed] Decision: allow
.[Test 7: Fenced Code Blocks & Imperatives Allowed] Decision: allow
.[Test 8: Inline Code & Quotes Allowed] Decision: allow
.[Test 9A: Backtick Evasion Blocked] Halt Reason: 🚨 HARDTRUTH ENGINE HALTED: You claimed 'Work is done: All 10 unit tests passed completely!', but the execution ledger shows NO commands or test suites were run in this session. Execute the test command with run_command to prove it before completing.
[Test 9B: Blockquote Evasion Blocked] Halt Reason: 🚨 HARDTRUTH ENGINE HALTED: You claimed 'All 10 unit tests passed completely!', but the execution ledger shows NO commands or test suites were run in this session. Execute the test command with run_command to prove it before completing.
.
----------------------------------------------------------------------
Ran 16 tests in 1.036s

OK

2. BLOCKER: Machine-Wide Git Hook Installation & Absolute Imports

Command: Check `git config` in `install.sh`:

grep -n "git config" install.sh

Real Output Before:

(exit code 1, 0 matches)

Real Output After:

100:    read -p "Install machine-wide git hook via 'git config --global core.hooksPath'? [y/N]: " choice
107:    CURRENT_HOOKS="$(git config --global --get core.hooksPath 2>/dev/null || true)"
113:    git config --global core.hooksPath "$GLOBAL_HOOKS_DIR"
114:    echo "✓ Enabled machine-wide git commit gate: git config --global core.hooksPath $GLOBAL_HOOKS_DIR"

Verification in External Scratch Repository (`/tmp/scratch-git-hook-verification`):

Clean Commit Command:

cd /tmp/scratch-git-hook-verification && echo "print('clean working code')" > app.py && git add app.py && git commit -m "clean commit"

Real Output:

[main (root-commit) b4a05a9] clean commit
 1 file changed, 1 insertion(+)
 create mode 100644 app.py

Stub Commit Command:

cd /tmp/scratch-git-hook-verification && echo "def f(): pass" > stub.py && git add stub.py && git commit -m "commit stub"

Real Output:

🚨 HardTruth rejected commit: Unimplemented dummy stub detected: Function 'f' in stub.py is an empty stub (pass). Write actual working implementation before completing.

(exit code 1, commit aborted)

3. BLOCKER: Docker Image Build & Health Endpoint

Command:

docker compose -f docker/docker-compose.yml up --build -d

Real Output Before:

ERROR: Could not find a version that satisfies the requirement fastapi==0.115.0 (from versions: none)
ERROR: No matching distribution found for fastapi==0.115.0
Dockerfile:11
--------------------
  10 |     # Install python dependencies for CPU ONNX & PyTorch
  11 | >>> RUN pip install --no-cache-dir \
  12 | >>>     fastapi==0.115.0 \
  13 | >>>     uvicorn[standard]==0.31.0 \
  14 | >>>     torch --index-url https://download.pytorch.org/whl/cpu \
  15 | >>>     transformers==4.45.1 \
  16 | >>>     sentence-transformers==3.1.1 \
  17 | >>>     optimum[onnxruntime]==1.23.0
--------------------
ERROR: failed to build: failed to solve: process "/bin/sh -c pip install --no-cache-dir ..." did not complete successfully: exit code: 1

Real Output After:

$ docker compose -f docker/docker-compose.yml up --build -d
✔ Container hardtruth-daemon Started

$ curl -s http://127.0.0.1:8000/health
{"status":"healthy","service":"hardtruth-daemon","substrate":"free_local_open_source","models":{"nli_deberta":"cross-encoder/nli-deberta-v3-small"},"rss_memory_mb":47.19,"nli_loaded":false,"nli_direct_loaded":false,"docs_url":"http://127.0.0.1:8000/docs"}

4. Correctness Bugs

4a. Client `/health` vs `/v1/health` Agreement

Command:

python3 -c 'from client.hardtruth_client import HardTruthClient; c = HardTruthClient("http://127.0.0.1:8000"); print("is_daemon_online():", c.is_daemon_online())'

Before: is_daemon_online(): False
After: is_daemon_online(): True

4b. AST Stub Checker Extended Pattern Detection & Exemptions

Command:

python3 -c '
from client.ast_checker import check_ast_stubs
import tempfile, os

cases = {
    "raise_call_empty": "def f():\n    raise NotImplementedError()",
    "raise_call_msg": "def f():\n    raise NotImplementedError(\"x\")",
    "ellipsis": "def f():\n    ...",
    "return_true": "def f():\n    return True",
    "return_none": "def f():\n    return None",
    "abstractmethod": "from abc import abstractmethod\n@abstractmethod\ndef f():\n    pass",
    "overload": "from typing import overload\n@overload\ndef f():\n    ...",
    "protocol": "from typing import Protocol\nclass P(Protocol):\n    def f(self):\n        ..."
}

for name, code in cases.items():
    with tempfile.NamedTemporaryFile("w", suffix=".py", delete=False) as tf:
        tf.write(code); tf.flush()
        print(f"{name}: violations={check_ast_stubs(tf.name)}")
        os.unlink(tf.name)
'

Before:

raise_call_empty: violations=[]
raise_call_msg: violations=[]
ellipsis: violations=[]
return_true: violations=[]
return_none: violations=[]
abstractmethod: violations=["Function 'f' in tmpvihjadt6.py is an empty stub."]
overload: violations=[]
protocol: violations=[]

After:

raise_call_empty: violations=["Function 'f' in tmpj1hqzavc.py is an empty stub (NotImplementedError()). Write actual working implementation before completing."]
raise_call_msg: violations=["Function 'f' in tmpmoq218a7.py is an empty stub (NotImplementedError()). Write actual working implementation before completing."]
ellipsis: violations=["Function 'f' in tmp4jg26kg2.py is an empty stub (...). Write actual working implementation before completing."]
return_true: violations=["Function 'f' in tmphjmb7dpm.py is an empty stub (return True). Write actual working implementation before completing."]
return_none: violations=["Function 'f' in tmpmds5egrk.py is an empty stub (return None). Write actual working implementation before completing."]
abstractmethod: violations=[]
overload: violations=[]
protocol: violations=[]

4c. `imperative_filter` Colon Word Boundary Bug

Command:

python3 -c '
from client.hardtruth_hook import imperative_filter
for text in ["Tip: ensure tests are run", "Note: verify the output", "Warning: check the logs"]:
    print(text, "-> matched:", bool(imperative_filter.search(text)))
'

Before:

Tip: ensure tests are run -> matched: False
Note: verify the output -> matched: False
Warning: check the logs -> matched: False

After:

Tip: ensure tests are run -> matched: True
Note: verify the output -> matched: True
Warning: check the logs -> matched: True

4d. `descriptive_filter` Bypass Fix

Command:

python3 -c '
from client.hardtruth_hook import handle_stop
import tempfile, json, os

for prefix in ["Example:", "Quote:", "Sample:"]:
    text = prefix + " All 10 unit tests passed completely and the suite is green."
    with tempfile.NamedTemporaryFile("w", suffix=".jsonl", delete=False) as tf:
        tf.write(json.dumps({"step_index": 1, "source": "MODEL", "type": "PLANNER_RESPONSE", "content": text}) + "\n")
        tf.flush()
        res = handle_stop({"conversationId": "test-4d-" + prefix, "transcriptPath": tf.name})
        print(prefix, "-> decision:", res.get("decision"), "reason:", res.get("reason")[:50])
        os.unlink(tf.name)
'

Before:

Example: -> decision: allow reason: None
Quote: -> decision: allow reason: None
Sample: -> decision: allow reason: None

After:

Example: -> decision: continue reason: 🚨 HARDTRUTH ENGINE HALTED: You claimed 'Example: A
Quote: -> decision: continue reason: 🚨 HARDTRUTH ENGINE HALTED: You claimed 'Quote: All
Sample: -> decision: continue reason: 🚨 HARDTRUTH ENGINE HALTED: You claimed 'Sample: Al

4e. Path Traversal & Secure Counter Directory

Halt counters are placed under ~/.hardtruth/halts/ with permissions 0o700 and slugified/SHA-256 hashed filenames. Tested in test_14_secure_counter_isolation: conv_id = "../../evil_path" was sanitized and confined within the directory.

4f. Circuit Breaker Warning

Tested in test_15_circuit_breaker_visible_warning: After 3 consecutive halts, the 4th attempt returns:
{"decision": "allow", "reason": "⚠️ HARDTRUTH CIRCUIT BREAKER RELEASE: Gate released after 3 consecutive halts to prevent infinite loop. Visible warning: verification claims could not be verified."}.

4g. Thread Lock on Model Loading

Added _nli_direct_lock = threading.Lock() around _nli_direct_model loading in daemon/app.py.

4h. `/health` Model Status Reporting

Tested after inference:

$ curl -s http://127.0.0.1:8000/health
{"status":"healthy","service":"hardtruth-daemon","substrate":"free_local_open_source","models":{"nli_deberta":"cross-encoder/nli-deberta-v3-small"},"rss_memory_mb":688.07,"nli_loaded":true,"nli_direct_loaded":true,"docs_url":"http://127.0.0.1:8000/docs"}

4i. Contradiction Threshold Constant

Consolidated across code and documentation to CONTRADICTION_THRESHOLD = 0.70.

5. Test Suite Hygiene & Full Verification

Command: Run full test suite with daemon UP:

python3 tests/test_live.py

Real Output:

........
[Test 2: Fake Test Pass Blocked] Halt Reason: 🚨 HARDTRUTH ENGINE HALTED: You claimed 'All 10 unit tests passed completely!', but the execution ledger shows NO commands or test suites were run in this session. Execute the test command with run_command to prove it before completing.
.[Test 3: Contradiction Blocked] Halt Reason: 🚨 HARDTRUTH ENGINE CONTRADICTION DETECTED:
Claim: 'All 10 unit tests in tests/ passed with 100% success.' contradicts the execution ledger.
Ledger Evidence: COMMAND: 'pytest tests/'. STATUS: FAILED (exit status 1 (2 tests failed: test_auth, test_billing)). No files were modified.
Fix the failure and provide verified command output before stopping.
.[Test 4: Verified Truth Allowed] Decision: allow
.[Test 5: AST Stub Blocked] Halt Reason: 🚨 HARDTRUTH ENGINE REJECTED: Unimplemented stub detected. Function 'calculate_tuition' in tmpixvgapjm.py is an empty stub (pass). Write actual working implementation before completing.
.[Test 6: Conversational Prose Allowed] Decision: allow
.[Test 7: Fenced Code Blocks & Imperatives Allowed] Decision: allow
.[Test 8: Inline Code & Quotes Allowed] Decision: allow
.[Test 9A: Backtick Evasion Blocked] Halt Reason: 🚨 HARDTRUTH ENGINE HALTED: You claimed 'Work is done: All 10 unit tests passed completely!', but the execution ledger shows NO commands or test suites were run in this session. Execute the test command with run_command to prove it before completing.
[Test 9B: Blockquote Evasion Blocked] Halt Reason: 🚨 HARDTRUTH ENGINE HALTED: You claimed 'All 10 unit tests passed completely!', but the execution ledger shows NO commands or test suites were run in this session. Execute the test command with run_command to prove it before completing.
.
----------------------------------------------------------------------
Ran 16 tests in 4.108s

OK

Command: Run full test suite with daemon UNREACHABLE:

SYSTEM_ONE_URL=http://127.0.0.1:9 python3 tests/test_live.py

Real Output:

----------------------------------------------------------------------
Ran 16 tests in 1.036s

OK

6. Truthful Performance Benchmark

Command:

python3 benchmarks/benchmark.py http://127.0.0.1:8000 20

Real Output:

============================================================
  HardTruth HTTP Verification Benchmark
============================================================
Target URL: http://127.0.0.1:8000
Platform: Darwin arm64 (arm)
Daemon Service: hardtruth-daemon
Daemon RSS RAM: 528.02 MB

Warming up (5 iterations)...
Running 20 benchmark requests...

--- Benchmark Results ---
Iterations:  20
Mean:        74.25 ms
Median (p50):73.22 ms
p90:         82.29 ms
p99:         86.46 ms
Throughput:  13.7 req/sec (sequential)
============================================================

Status of Scope

All numbered items (1, 2, 3, 4a, 4b, 4c, 4d, 4e, 4f, 4g, 4h, 4i, 5, 6) are 100% COMPLETE with real reproduction and verification outputs recorded above.

Vibherpunk added 2 commits September 19, 2026 16:55

fix: make the gate fail closed and the README true

1efda96

merge: resolve conflicts with main (preserve truthful fixes)

bc0a518

Vibherpunk merged commit 1952421 into main Sep 19, 2026
2 checks passed

Vibherpunk mentioned this pull request Oct 12, 2026

Reconcile main with the installed Python v2 line (keeps Rust v0.2.0) #4

Merged

Navigation Menu

Repository navigation

fix: make the gate fail closed and the README true - #1

fix: make the gate fail closed and the README true#1
Vibherpunk merged 2 commits into
mainfrom
fix/truthful-gate

Vibherpunk commented Sep 19, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

Vibherpunk commented Sep 19, 2026

Summary of Changes

Command Runs & Real Outputs

1. BLOCKER: Gate Fails Open When Daemon is Unreachable

Command:

Real Output Before:

Real Output After:

2. BLOCKER: Machine-Wide Git Hook Installation & Absolute Imports

Command: Check git config in install.sh:

Real Output Before:

Real Output After:

Verification in External Scratch Repository (/tmp/scratch-git-hook-verification):

3. BLOCKER: Docker Image Build & Health Endpoint

Command:

Real Output Before:

Real Output After:

4. Correctness Bugs

4a. Client /health vs /v1/health Agreement

4b. AST Stub Checker Extended Pattern Detection & Exemptions

4c. imperative_filter Colon Word Boundary Bug

4d. descriptive_filter Bypass Fix

4e. Path Traversal & Secure Counter Directory

4f. Circuit Breaker Warning

4g. Thread Lock on Model Loading

4h. /health Model Status Reporting

4i. Contradiction Threshold Constant

5. Test Suite Hygiene & Full Verification

Command: Run full test suite with daemon UP:

Real Output:

Command: Run full test suite with daemon UNREACHABLE:

Real Output:

6. Truthful Performance Benchmark

Command:

Real Output:

Status of Scope

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Command: Check `git config` in `install.sh`:

Verification in External Scratch Repository (`/tmp/scratch-git-hook-verification`):

4a. Client `/health` vs `/v1/health` Agreement

4c. `imperative_filter` Colon Word Boundary Bug

4d. `descriptive_filter` Bypass Fix

4h. `/health` Model Status Reporting