I recently reviewed six improvements that my AI assistant had marked complete.

Only two were actually complete.

The other four were not imaginary or abandoned. Code existed. Tests passed. Prompts had been deployed. Scheduled jobs were enabled. Several parts had already worked in production.

But each task still had at least one promised live outcome that had never been observed. One was waiting for its first scheduled weekly run. Another had delivered its news path but not the reading-feedback loop included in the same specification. A third had a longer observation period written into its own acceptance criteria.

The work was implemented. The outcomes were not yet accepted. We had collapsed those two states into one satisfying green tick.

“Done” drifted during implementation

A large automation task usually begins with a clear outcome and accumulates intermediate successes:

  • a script compiles;
  • unit tests pass;
  • configuration validates;
  • a prompt matches its source file;
  • a cron job reports ok;
  • one branch of the workflow works live.

Every item is useful evidence. None necessarily proves the original promise.

As implementation proceeds, those milestones become vivid while the acceptance criteria remain in an older planning document. The completion note naturally lists what was built. If nobody maps that list back to every promised outcome, “substantial progress” quietly becomes “done”.

This is not peculiar to AI. The assistant merely performs the mistake quickly and writes a very confident summary afterwards.

The bug that made an unrun job look healthy

The audit found a more interesting failure in the outcome ledger.

My daily and weekly planning rituals both wrote a shared field named:

lastRitualCompletedMin

A successful daily ritual updated the timestamp. The verification code later saw a recent ritual completion and credited it to the new weekly workflow—even though that weekly workflow had not yet run.

The field stored when, but not which producer. A correct fact was attached to the wrong claim.

The repair was to give daily and weekly rituals separate state fields and require correlation between three pieces of evidence:

  1. the ritual-specific timestamp;
  2. the identity of the producing job;
  3. a run summary confirming the intended delivery.

This is a small observability lesson with broad consequences. Shared “last successful” timestamps are dangerous when several pipelines can update them. Freshness without provenance can make unrelated work green.

An acceptance matrix changes the conversation

The corrective audit did not ask “what files changed?” It started from the promises and built one row per observable outcome.

For each row, the state must be one of:

  • implemented — the mechanism exists and static verification passes;
  • accepted — the promised effect has been observed with direct evidence;
  • operational observation pending — the implementation is ready, but a natural run or time window is still required;
  • blocked — an external dependency or decision prevents the test.

Those states stop a deployed configuration from impersonating a delivered outcome.

Examples of acceptable evidence differ by promise:

  • A prompt deployment is supported by an exact source-to-live read-back.
  • A task migration is supported by fetching the entire resulting board and checking IDs, metadata, and duplicates.
  • A notification workflow is supported by a delivery-confirming run and the actual rendered content.
  • A de-duplication promise is supported by replaying the same candidate and observing silence, then introducing a material change and observing one re-opened delivery.
  • A weekly planning ritual is supported by the scheduled weekly ritual, not by a manual test of its renderer or a successful daily job.

The evidence must match the noun in the promise. Configuration proves configuration. Delivery proves delivery. Time proves an observation period. One should not be promoted to another because the dashboard has a green icon available.

Partial completion needs an explicit scope decision

One tempting escape is to close the task for the implemented portion and describe the remaining outcome as “follow-up”. Sometimes that is sensible. It is not something the system should do silently.

If an acceptance row remains unverified, the original task stays open unless I explicitly approve a narrower scope and a separately named task owns the omitted result. Otherwise difficult last-mile work steadily migrates out of tracked commitments and into prose.

In this audit, four tasks were reopened. Their completed components were not discarded. The notes were corrected to distinguish what was built from what still needed live evidence.

The research workflow is a useful example. Its collection, scoring, de-duplication, and delivery path had passed. But the original task also promised an organic reading tracker in which feedback affected later recommendations. The database schema existed; the live loop did not. Closing the whole task hid the missing product behind a successful subsystem.

Reopening it led to the actual repair: imported reading history, append-only feedback events, source-strength rules, conversational updates, and recommendation constraints that preserve negative as well as positive signals.

Tests and live evidence have different jobs

This is not an argument against tests. Deterministic tests are the fastest way to prove edge cases repeatedly. They are particularly good for idempotency, state transitions, de-duplication, formatting limits, and failure recovery.

But a test cannot prove that the production scheduler used the new prompt, that Telegram accepted the message, that the live API preserved every task field, or that a weekly ritual was genuinely useful when it arrived.

Conversely, one successful live run is poor evidence for all the edge cases the test suite covers.

The sensible stack is:

unit and replay tests
        ↓
source-to-live configuration read-back
        ↓
production run correlated to its producer
        ↓
observable outcome checked against the acceptance row

Each layer proves something the layer below cannot.

Completion should survive an unfriendly question

My earlier cron health checks asked whether scheduled work was running. The newer acceptance layer asks a less comfortable question: what observable fact would make this promise true?

For a message, show the delivery and content. For a migration, show the resulting records. For suppression, run the replay. For a weekly behaviour change, wait for the weekly event. If the answer is “the command said OK”, the task is probably not finished.

The audit reduced six green tasks to two accepted and four honestly in progress. That looked worse for approximately five minutes. In reality, nothing had regressed. The tracker had simply stopped flattering the implementation.

That may be the most useful recent improvement of all: teaching the assistant that a convincing completion note is evidence of writing ability, not evidence that the outcome happened.

This concludes a short series on recent improvements to my personal AI system. It began with a task queue shared by human and assistant, continued through an intelligent capture inbox, and then a research desk with durable memory.