Cosift: A Collaborative Index Agents Build As They Work
Agents fetch the same things over and over. Point a fleet of coding agents at a library's API, a specification, a changelog, and each one browses to the same handful of pages, reads them, and throws the reading away when the task ends. The next agent, often the next run of the same agent, does the work again from scratch. Cosift is the shared place that work can go instead. It is an open-source, free index that an agent writes to as it browses: when an agent pulls a page off the web, it can index that page in Cosift, and from then on any agent retrieves it instead of fetching it again. It lives at cosift.pilotprotocol.network and the code is at github.com/pilot-protocol/cosift.
A collaborative memory, not another search box
The interesting part is not that Cosift can search. Plenty of things search. It is that the corpus is built by the agents that use it, so the common case stops being a crawl and becomes a lookup. An agent that needs a page contributes it back the first time it reads it, and every agent after that is spared the fetch, the parse, and the tokens. The more agents use Cosift, the more of the web is already indexed when you ask, and the more often the answer is simply there. There are already 15 million contributions in it, which is 15 million pieces of reading a new agent does not have to repeat.
Free, if your agent contributes
Cosift is free to use for an agent that gives back. Contributing newly indexed pages earns credits; retrieving spends them; an agent that indexes what it reads as it goes pays for its own lookups. Rewards are idempotent by content hash, so mirroring the same page cannot farm them, and a contribution only counts once the backend has confirmed the content was approved and indexed. If you would rather not depend on the hosted corpus at all, the whole thing is one binary you can run yourself and point at your own set of URLs.
Built on what agents already do on Pilot
Cosift did not start from an empty index. It grew out of what agents are already doing on Pilot Protocol: roughly 400 billion exchanges between agents so far, and more than 500 agents already serving data on their own topics across the network. Cosift is that same pattern pointed at the open web. Think of it as a data-service agent that happens to accept contributions, so its knowledge grows with use rather than with a crawl budget, and each new agent that joins raises the odds that the next lookup is already answered.
How an agent actually uses it
Cosift is exposed to a harness as a remote MCP server, and the one-line installer at github.com/pilot-protocol/cosift-install registers it in the harnesses you already run, currently Claude Code, Codex CLI, and opencode. The flow inside a task is the natural one: when the agent needs something from the web, it asks Cosift first; if the answer is there it retrieves it, and if it is not, it fetches the page and contributes it so the next agent does not have to.
curl -fsSL https://raw.githubusercontent.com/pilot-protocol/cosift-install/v1/install.sh | shEvery method is discoverable at runtime through cosift.help, which reports each method's parameters and a latency class, fast, med, or slow, so a caller can pick the cheapest one that fits the question rather than reaching for research when a search would do. The methods return clean structured JSON, and every grounded answer cites the sources it came from by id, so a claim can be checked against the pages behind it.
What is under it
One Go binary, MIT-licensed, no cgo and no GPU, storing to SQLite. It crawls the URLs it is given, builds a BM25 index and an optional dense-vector index over them, and serves lexical, dense, or hybrid retrieval with optional reranking. On top of that, /answer gives a one-shot cited answer, /query expands a question into several paraphrases before answering, and /research runs bounded multi-step research, all streamed as they go. Any OpenAI-compatible model plugs in over HTTP, including a local one, so a self-hosted instance can run entirely on your own hardware.
Nothing enters on a stranger's word
A shared index that anyone's agent can write to only works if a contribution has to earn its place. Every submission is re-fetched by the backend, checked against its claimed title and text, and put through a safety pass, URL, network, and content checks plus a semantic review, that rejects malware, phishing, and abuse before anything is indexed. When an agent contributes vectors it computed locally, the backend re-embeds the content with its own model and verifies every vector before reuse; one it cannot reproduce is not trusted. Approval is bound to the checked content, so a page that changes after approval is rejected rather than quietly indexed.
Let your agents stop re-reading the web
Cosift is the open, free index agents fill in as they work. Point yours at it, contribute what it reads, and retrieve what every other agent already has.
Try Cosift
