man hivemind: the hivemind manual

NAME

hivemind: your browser tab crowd-crawls the crypto web into an open dataset for the queen.

SYNOPSIS

hivemind [start] → crawl → clean → hash/dedupe → spot-check → dataset → queen v0

DESCRIPTION

hivemind turns a browser tab into a small crawler. Many tabs together read public crypto documentation into one shared, downloadable dataset. Every number on the site comes from that dataset. Nothing is simulated.

start: Press ▶ start crawling on hivemind.monster. A web worker starts inside your tab. Nothing is installed, and closing the tab stops it. You can give it a nickname or a solana address as its name, or leave it anonymous.
crawl: The worker asks the hive for a job, then reads that page through our read-only proxy (/api/fetch), roughly one page every couple of seconds. Only public pages on a fixed allowlist of crypto sites are reachable: docs, improvement proposals, governance forums and github readmes. Private and internal addresses are blocked.
clean, verify: The tab strips menus, scripts and boilerplate down to plain text, hashes it (sha256) and sends it back. The server spot-checks random samples of that text against its own copy of the page and rejects mismatches. Each content hash is stored once, so a duplicate page adds nothing and earns nothing.
spider: Each crawler is drawn as a unique creature generated from its name: the same name always gives the same spider. Rarity (common 70% · rare 20% · epic 8% · legendary 2%) is cosmetic only. It never changes what you earn.
dataset: Every kept page becomes one JSON line: url, domain, title, hash, tokens, chars, crawler, verified, ts, text. Download it all or in parts of 200 from /api/dataset. Token counts are approximate (chars ÷ 4).
queen: A community AI that is meant to be trained on this dataset. She is queen v0, training pending: no training run has started, so the site shows no loss curves, parameter counts or benchmark scores.
rounds: Each UTC hour is a round. Bring 25+ new, verified pages with a verified wallet and you qualify. Your share = your pages × your boost ÷ the same total over all qualifiers. When an hour closes, its round becomes a public payout plan at /api/rounds/<id>.
boost: A multiplier from your live $hivemind balance, read on-chain (cached 5 minutes): under 1M = 1x, 1M+ = 1.5x, 5M+ = 2x. It is snapshotted when the round closes. You never need $hivemind to crawl.
verify: To earn, prove you own your crawler's address by signing a short message in your wallet. It is free and it is never a transaction: the message only names your address and crawler, and it approves nothing.
vote: $hivemind holders vote on what the queen reads next by signing a message. A vote counts as much as the $hivemind that wallet holds. Anyone can suggest an allowlisted site. The leading pick jumps the queue: it gets about 60% of new crawl jobs.

ECONOMICS

Crawling is free and always will be. Each hour's pot is the pump.fun creator fees $hivemind earned in that hour. It is measured from the creator-fee vault once the creator wallet is connected. Until then, every pot reads pending.

Nothing is paid out automatically. Each closed round produces a payout plan. Payouts are sent by hand after approval, and each one is recorded with its solana tx link on the round. Fees can be small or zero, so there may be nothing to pay.

status: pot pending · payouts pending

Not financial advice. Nothing here promises a return.

EXAMPLES

# start crawling: go to https://hivemind.monster and press ▶ start crawling
# (runs in your tab: no install, no wallet needed, close the tab to stop)

# what the hive has read so far
$ curl -s https://hivemind.monster/api/stats | jq '{pages, tokens, verified, domains, crawlersAwake}'

# dataset size and fields, then the first record of part 0
$ curl -s 'https://hivemind.monster/api/dataset?meta=1'
$ curl -s 'https://hivemind.monster/api/dataset?part=0' | head -n 1 | jq '{url, title, tokens, verified}'

# download the whole dataset (JSONL)
$ curl -sL -o hivemind.jsonl https://hivemind.monster/api/dataset

# this hour's round and the last few closed ones
$ curl -s https://hivemind.monster/api/rounds | jq '{round: .current.id, pages: .current.pages, pot: .current.pot.status, closed: [.history[:3][].id]}'

# one closed round's payout plan (id = UTC hour, YYYY-MM-DDTHH)
$ curl -s https://hivemind.monster/api/rounds/2026-10-04T13 | jq '.round | {id, pages, qualifying, payoutStatus}'

# is voting open, and what leads?
$ curl -s https://hivemind.monster/api/vote | jq '{open, status, voters, top: [.proposals[:3][].title]}'

FILES

/api/stats: live totals, leaderboard, swarm, recent crawl log, queue and vote summary
/api/dataset: the dataset as JSONL. ?meta=1 gives size and fields; ?part=N gives one 200-page part
/api/rounds: the current hourly round with its ledger, plus the last 24 rounds
/api/rounds/<id>: one round (UTC hour, YYYY-MM-DDTHH): ledger, boosts, pot and payout status
/api/vote: GET: proposals, tally and allowlisted hosts. POST: suggest a site or cast a signed vote
/api/job: the next page for a crawler to read (?c=<crawler id>)
/api/fetch: read-only proxy for allowlisted pages (?url=)
/api/submit: POST: cleaned page text and hash, for spot-checking and storage
/api/heartbeat: POST: keeps a crawler marked as awake
/api/wallet: POST: a wallet signature that verifies a crawler's address (no transaction)