Ingest a source on a schedule
Set up a daemon so Brain keeps a repository, a Drive account or a ClickUp workspace in memory, and read what each run took, skipped or refused.
A daemon is one source, ingested on a schedule. It names what Brain reads, what to take from it, and how much the resulting memories are trusted. Once a daemon exists, Brain keeps memory in step with that source without anyone asking it to.
A daemon needs a connection to the source already available to the product — with the one exception of a public GitHub repository, which needs none. See connect a source.
Set up a daemon
- Pick the source type — GitHub, Google Drive or ClickUp.
- Narrow it. For GitHub: the connected account, the repository, and the branch. One daemon reads exactly one branch of one repository, so several daemons on the same repository — one per branch — are normal. For Google Drive: the connected account, whose picked files and folders are the daemon’s scope. For ClickUp: the workspace, optionally narrowed to one space or one list, and the chat channels to follow.
- Choose what to take. The categories differ per source and each is an independent switch; they are listed in what each source gives Brain. For GitHub, Markdown can also be narrowed with include and exclude patterns — the default is
**/*.mdwith nothing excluded. - Set the source trust and the load mode. Both apply to every daemon; see the table below.
A daemon starts enabled and is picked up by the next scheduled cycle. You do not have to wait for it — a run can be started on demand.
The two settings every daemon has
| Setting | What it does |
|---|---|
| Source trust | Left on automatic, each category is stamped with the level that suits it — official, team or informal depending on the source, as listed in trust of what is ingested. Set it explicitly and everything this daemon ingests is stamped at that level instead, including policy. See what a memory is for what the levels do. |
| Load mode | Hot processes everything as it is read, and a run finishes in seconds to minutes. Batch defers processing, which costs less on large volumes but can take hours. A batch daemon batches every sync. A hot daemon batches its first sync only, and only if that first sync turns out to be large; a small first sync is processed hot after all, and every later sync of a hot daemon is processed hot. |
How runs work
There is one ingestion cycle for the whole product and it runs hourly. It is not configurable per daemon: every enabled daemon is considered on every cycle, and a daemon that already has a run in flight is left alone until that run finishes. Between the schedule and an on-demand run there is no other trigger — Brain does not react to a push, a file edit or a new task as it happens.
A run walks the source in three movements: it discovers what is in scope, fetches each item, then writes what changed into memory. Nothing is re-read for the sake of it — an item whose content has not changed since the last run is recognised and skipped, so a routine run over a large repository does very little work. At the end of a run, items that have disappeared from the source are removed from memory, so deleting a document at the source eventually removes it from Brain. Removals are applied by runs processed hot; a run processed as a batch ingests and updates, but never removes.
A run ends in one of these states:
| Status | Meaning |
|---|---|
queued | Accepted and waiting to start. |
running | In progress. |
done | Finished. Everything reachable was processed. |
failed | Finished, but at least one item failed for a reason worth retrying — a timeout, a rate limit, a network or server error. The next cycle tries again. |
An item Brain is not allowed to read does not fail a run. That is a permanent outcome, not a transient one: the run finishes
doneand records the item as denied, because retrying would only produce the same answer. If you expected that item, the access has to be fixed on the connection.
A run that stops responding — a process that died mid-run, for example — is closed automatically once it goes stale, and the daemon resumes on the next cycle. Runs blocked because the product is over its plan limit are skipped for that cycle rather than failed; see what Brain meters and which limits apply.
Inspect what a run ingested
Every daemon carries its latest run, the items of that run, and its run history. The items are listed individually, and each carries an outcome:
| Outcome | Meaning |
|---|---|
discovered | Found in scope by the run. |
fetched | Content retrieved from the source. |
ingested | Written into memory. |
skipped_unchanged | Unchanged since the last run, so not re-read. |
denied | Permanently refused by the source — not found, or not permitted. |
error | Failed for a reason worth retrying. |
Filtering that list by outcome is the fastest way to answer “what did this run refuse to take?”. The run history shows the recent runs with the same counts per run, so a run that suddenly ingested far less than the one before it is visible without reading either in detail.
Two counts on the daemon itself accumulate across runs rather than describing one run. One counts documents whose text was stored but whose meaning could not be fully extracted. The other counts documents ingested before Brain began measuring that, which are not known to be either good or damaged. They are reported separately on purpose: neither can be read as “clean”.
Re-ingest a damaged document
An item is normally re-read only when it changes. That keeps runs cheap, and it also means a document that was ingested badly would stay that way. Brain handles that itself: on every run, a daemon retries the documents whose meaning could not be extracted last time.
- A document is retried immediately, then after one hour, then after six, then after twenty-four.
- At most four consecutive attempts, after which it is left alone.
- One run repairs at most 100 documents; the rest carry over to the following runs.
- A document edited at the source re-enters memory through the normal path regardless of that budget.
Forcing a re-ingest of specific documents is not available today.
Pause or remove a daemon
Disabling a daemon takes it out of the schedule, and it refuses a run on demand until it is enabled again. Its configuration and everything it has already ingested are untouched, and enabling it resumes from where it left off rather than starting over.
Removing a daemon deletes the daemon and Brain’s record of what it had ingested from that source. The memories it created are not deleted — they stay in memory as ordinary memories, with nothing left to keep them in step with the source. Removing and recreating a daemon on the same source therefore re-ingests everything from scratch, and the two sets of memories are reconciled by content rather than by daemon.
To stop ingesting for a while, disable the daemon instead of removing it.
If it did not work
There are no memories at all. Check, in this order: that a daemon exists, that it has a completed run, and that the run ingested items rather than denying them.
The daemon could not be created, or its runs fail on access errors. The source was probably never connected at the organization level, or the connection was not attached to your product — or it was detached afterwards, which stops every daemon that used it. Brain can only read what the connection already reaches; widening it is done on the organization side. See connect a source.
The daemon exists but has never run. The cycle is hourly, so a daemon created just now may not have been picked up yet. Start a run on demand rather than waiting. A disabled daemon is skipped by the schedule and refuses a manual run — enable it first.
A run is still in progress. A first ingestion of a large source takes as long as it takes, and a daemon in the deferred load mode can take hours. A daemon with a run in flight is left alone by the hourly cycle until it finishes.
The run finished done but ingested very little. Look at the items by outcome. Items marked denied were refused by the source permanently — an access problem to fix on the connection. Items marked skipped_unchanged are normal on a repeat run and mean the source has not moved.
Everything ran, but a document you expected is missing. Brain does not read everything a source holds. Each source has a defined list of what it takes and what it skips, and each category is a switch on the daemon — check that the one you expected is on. Check what each source gives Brain before assuming a failure.
Runs are skipped and nothing is ingested. If your organization’s allowance for the month is exhausted, runs are skipped rather than failed, and cognition passes are skipped silently. See what Brain meters and which limits apply.
Next steps
- Connect a source: what each connection requires and exactly what Brain reads from it.
- Add, search and correct memories: check what an ingestion actually landed, and fix what is wrong.
- What a memory is: what the source trust level a daemon stamps does to a search result.
- What Brain meters and which limits apply: what ingestion counts against, and what happens when the month’s allowance runs out.
- Look up a Brain tool: the tools your agents call to put a source on a schedule and read its runs.