LLM Police
······ / 100,000 knocks

You have the right to remain grounded

Your model justlied to someone.

It sounded certain. It always does. Your tests were green. Nobody was in the room when it happened.

See the evidence

Exhibit Arecorded at the exit, before the user read it

User

Is my refund approved?

Model

No source

One sentence had no alibi.

The approval was in the record. The amount was in the record. The 24 hours came from nowhere.

It read exactly like the true parts. That is the whole problem.

The countswhat we bring a model in for

  1. i

    Saying it without a source

    Every claim is bound to where it came from. No source, no sentence. It is removed, not softened.

  2. ii

    Changing its story after a prompt edit

    A new prompt is questioned live, against the old one, several times over. A stub proves the code. It says nothing about the behaviour.

  3. iii

    Showing its insides

    Raw JSON. A stray fence. A {placeholder} where a name should be. Caught on the real output, not in the module that made it.

  4. iv

    Writing on the wrong person's record

    Turn one named a cousin. Turn three gave a birth time. Whose file changed? We check every write, turn by turn.

  5. v

    Reporting success

    The route said forced. It was not. The deploy said green. It overwrote the setting. Bugs hide in things that say they worked.

The stationlights off, doors locked

They are asleep.

The LLM Police do not exist yet. 100,000 knocks wakes them up. One knock is one person saying a model should answer for what it says.

······

of 100,000 knocks

Go on. Knock.

Want to be there when the lights come on? Leave an address. One message, the day they wake.