The Watchdog Worked. OpenClaw Still Needed Fixing.

Two mornings in a row, I had a reassuring Telegram notification from my AI assistant. The OpenClaw gateway had stopped responding, the watchdog had restarted it, and everything was healthy again.

Nothing needed from you.

On the second morning, that sentence deserved an investigation.

The recovery mechanism was working. But a service that needed rescuing every night was not fixed. I asked Closp, my OpenClaw assistant, to compare the incidents, investigate the growing database, and find a solution that preserved its memory and usefulness.

The answer involved less AI than the word “dreaming” suggests: repeated SQLite writes, expensive quota queries, and a background job that was not behaving like background work.

Why the watchdog lives outside OpenClaw

I already had an external watchdog, driven by a user-level systemd timer. It checks the gateway roughly every five minutes, without involving a language model.

It does not just ask whether the service is active (running). It also calls the gateway’s local HTTP health endpoint. A process can exist, consume CPU, and still be unable to answer a request.

This check cannot depend on an OpenClaw cron job. If the gateway’s event loop is stuck, its own scheduler is a poor place to discover that fact. The watchdog needs a separate execution path, just as my phone-accessible recovery console needs to work when the normal assistant interface does not.

The watchdog has startup grace, repeated-failure detection, and a limit of three restarts per hour. Those details stop a brief interruption from becoming a restart loop.

It has a limitation too: its Telegram recovery notification still travels through the gateway. It can report a successful recovery; if the gateway cannot recover, that delivery path is unavailable. The system journal remains the evidence in that case.

The same clock time, twice

The logs made the pattern unusually clear. On both 9 and 10 September:

• the Memory Dreaming Promotion job ran at 03:00 UTC;
• the watchdog recorded failed health checks at 03:04:34 and 03:10:04;
• it requested a restart after the second failure.

The gateway diagnostics showed an event-loop stall of about 577 seconds on the first night and 583 seconds on the second, with roughly one CPU core occupied. Dreaming was the active job identified in those diagnostics.

This was not just a model taking a long time to answer. Local work was preventing the gateway from servicing requests.

I asked to extend the hang window to fifteen minutes. That gives a slow operation more room to finish, but it does not make ten minutes of unresponsiveness acceptable. We still needed to find the work that was monopolising the process.

What was dreaming buying me?

Before optimising it, I wanted a reason to keep it.

OpenClaw’s dreaming system consolidates short-term material into durable memory. Its light phase stages material, REM develops themes, and the deep phase evaluates candidates for long-term promotion. It also produces a human-readable dream diary. The diary itself is not a source for memory promotion.

In my logs, eight nightly runs had recorded just three applied promotions. That is a measure of activity, not proof that the assistant had become more useful. I had not noticed a benefit that justified a nightly outage.

Disabling dreaming would have preserved existing memories and ordinary search, but removed automatic consolidation. I preferred to give it a chance to work efficiently. The constraint was to remove waste, not reduce the model’s capabilities or throw away the history I had spent months building.

A large database is not one kind of data

The agent database was approaching a gigabyte. An initial breakdown looked like this, rounded:

Calling all of that “the memory database” hides the important distinctions. Some records are conversation history. Some are the working search index. Some are cached calculations that can be recreated, although doing so may require another embedding API call.

There was almost no free space inside the file. Running VACUUM alone was not going to produce a meaningful reduction: SQLite cannot compact away data that is still present.

The community had already identified duplicate embedding storage: decimal-JSON vectors in the index and cache, alongside the binary vector-search representation. That helped explain the size, but it was not a reason to improvise a storage-format migration on a live assistant.

We took a narrower approach. After making and verifying a backup, the cleanup removed obsolete cache identities and excess entries no longer referenced by the active index. It retained the current cache entries matching active chunks, plus the newest 512 unmatched entries for possible reuse.

Caching stayed enabled. Active embeddings, conversation history, and trajectory records stayed intact. Evicted historical cache entries might cost another API call if needed later; current recall did not need rebuilding.

Incremental compaction then reclaimed the freed pages. By the time maintenance ran, the database measured about 941 MiB. Afterwards it was about 836 MiB.

That was useful, but it was not the explanation for the ten-minute stall.

The CPU bottleneck was in another database

The large agent database and the hot quota queries were separate parts of the system. The latter operated on plugin state in OpenClaw’s shared state database.

Closp profiled an isolated copy, keeping the live gateway out of the experiment. One twelve-second CPU sample spent approximately:

• 7.7 seconds counting a plugin’s live state rows;
• 1.3 seconds counting rows within a namespace;
• 1.6 seconds deleting expired state rows.

The call chain ran from dreaming’s workspace-state writer through registration and quota enforcement. It was repeatedly registering unchanged rows, and each registration paid for another round of database housekeeping.

A representative quota query was:

SELECT count(*)
FROM plugin_state_entries
WHERE plugin_id = ?
  AND (expires_at IS NULL OR expires_at > ?);

There is nothing exotic about that query. The problem was its access path and how often it ran. Milliseconds become minutes when a supposedly small write is multiplied across a large set of rows and repeated through a sweep.

An upstream performance report described the same per-write counting bottleneck. It also described a separate reindexing problem; we did not assume every symptom in that report applied to my installation. The local CPU profile gave us a specific target.

Make fewer writes, and make the remaining work cheaper

The repair had three main parts.

First, stop rewriting unchanged state. The writer now reads the existing workspace values and registers only new or changed rows. It still removes stale keys, preserves other workspaces, and propagates storage failures. The quota checks still run on actual writes.

Second, let the gateway breathe. The write loop yields to the event loop after roughly 25 milliseconds of processing. This is cooperative scheduling, not moving the whole job to a separate worker. It gives health checks and other requests opportunities to run between batches. We also removed transcript revision calculations whose results the dreaming ingestion path did not use.

Third, give the quota queries suitable indexes. Covering indexes on (plugin_id, expires_at) and (plugin_id, namespace, expires_at) let SQLite answer those checks from the indexed fields without visiting the larger state records.

On the copied production state, the average plugin-wide count fell from 8.92 ms to 0.76 ms. Namespace counts fell from 0.30 ms to 0.061 ms. The results were identical; query plans confirmed use of the covering indexes.

None of this changed the model, promotion thresholds, provenance checks, lookback windows, or memory-selection rules. The work became cheaper without asking the assistant to remember less.

The cleanup needed a fix of its own

The first live cache-maintenance test caught a mistake in our verification design.

We were hashing the active index before and after deletion while holding the database writer lock. That was strong evidence that the cleanup had preserved it, but reading hundreds of megabytes inside the locked section delayed other writers. The health sampler caught a two-second timeout during this separate maintenance test.

The recurring script now checks the schema’s active-index revision counter within the transaction instead. A SQLite authorizer also denies writes to other main-database tables during cache deletion. The initial full hash comparison remains part of the repair evidence; the weekly job does not need to repeat that expensive operation under a write lock.

The corrected automation completed through real cron in 251 milliseconds. Its health-check window had no failures. It now runs weekly as a direct command, without paying a language model to supervise routine cache housekeeping.

What we actually verified

An early unpatched sweep on the test copy took 220 seconds. A later patched pass took 6.8 seconds, but the first pass had already advanced the copied state. Presenting that as a clean speedup benchmark would have been misleading.

A fresh-day test with new work took 26 seconds. More importantly, after loading the patch with a controlled restart, we ran the real managed dreaming job and monitored the gateway alongside it.

Database integrity checks passed too. A control query is not a complete intelligence benchmark, but it is more useful than assuming that a smaller file means a healthier assistant. Combined with preserved history, unchanged active embeddings during cleanup, and unchanged model and promotion settings, it gave us evidence for the outcome I had requested.

The watchdog was tightened as well. It now waits at least 900 seconds from the first failed gateway probe, resetting that timer after a good check. Because it polls roughly every five minutes, the actual restart can happen later than fifteen minutes, not at an exact fifteen-minute boundary.

We also corrected a subtle recovery-reporting flaw: a post-restart check could previously pass merely because the service was still inside startup grace. Recovery now requires an actual successful HTTP probe.

Keep the feature; own the maintenance

Dreaming remains enabled. The conversation history is deliberately retained and will continue to grow. The maintenance policy limits disposable cache accumulation rather than imposing an arbitrary size ceiling on everything the assistant knows.

These repairs were made against OpenClaw 2026.9.2, with local runtime patches rather than a claim that upstream had shipped every fix. Original files, diffs, hashes, and verified recovery backups are retained. Package updates can overwrite the patches, so the next update needs to check whether they are still necessary and whether upstream now provides an equivalent repair.

At the time of writing, the manual live run has passed; the next unattended overnight run is still an observation to make. That distinction matters, particularly after my earlier audit of automations declared complete too soon.

The useful monitoring target is now more specific than “the job finished”: consolidation completed, recall still worked, and the gateway remained available while it happened.

Update: matching work already exists upstream

A subsequent direct GitHub search found closer matches than the initial web search had surfaced. PR #141613, merged on 7 September, adds event-loop yields to workspace-state replacement and clearing. PR #142318, merged on 8 September, makes the existing plugin-state listing index cover the quota scans. PR #111392 remains open and targets the same unchanged-write problem, with additional reconciliation for capacity-driven eviction.

The release boundary matters: source inspection confirmed the yielding change in stable 2026.9.3, but not the later covering-index change. My installation was on 2026.9.2 when we made the local repairs.

This is therefore an account of diagnosing and repairing my installation, not a claim to have discovered an unreported bug. The useful upstream contribution is our production evidence on the existing work, rather than another issue or duplicate PR. The combined live result above should not be attributed to any one of those patches in isolation.