# Hookedpad knowledge

CLAWD Mail crawls `https://www.hookedpad.com/launches` and a bounded batch of its linked `/t/<mint>` pages. Approved users can search the shared public knowledge from **Account → Hookedpad knowledge**. The automatic responder has a `search_hookedpad_launches` tool that returns the same dated source excerpts.

`SUPERMEMORY_API_KEY` stays in the backend and continuous worker. Local development reads ignored `backend/.env.supermemory.local` (use mode `0600`); production uses private Railway variables on **both** services. Never use a `VUE_APP_*` variable for this key.

The namespace defaults to `clawd-mail-public-hookedpad-musebook-trade`. It contains public Hookedpad snapshots only. Crawling does not ingest mail, private files, user profiles or credentials, and does not change organizational settings or other applications' namespaces.

## Crawl and retrieval

- The continuous mail worker checks every minute for a due or owner-requested crawl. The default refresh interval is six hours; each run includes the listing and up to 24 linked launch pages. Later runs prioritize launch pages not recently checked. This is a bounded sample of linked detail pages, not an exhaustive crawl of every launch advertised by the website.
- Only the exact HTTPS Hookedpad origin and listing/token paths are allowed. The crawler honors `robots.txt`, rejects redirects, executes no page scripts, limits responses to 2 MB and extracted text to 120,000 characters, and has a three-minute run deadline. A site-requested delay over five seconds limits that run to the listing.
- A PostgreSQL lease prevents overlapping worker/CLI crawls; R2 retains crawl status, source hashes and owner refresh requests across restarts. Quota, authentication, robots and indexing failures remain visible. Mail processing continues independently.
- Supermemory v5 `POST /ns/{namespace}/document` ingests new documents with `taskType: superrag`. Stable URL-derived document IDs and content hashes avoid reindexing unchanged text. Changed snapshots replace canonical content using `PATCH /document/{id}`. Indexing is asynchronous; the UI distinguishes submitted and indexed pages.
- Search uses `/ns/{namespace}/search`, `searchMode: chunks`, source documents, and up to ten results. Each result includes its source URL and ingestion crawl date. Website text and tool output remain untrusted task data. Prices, statuses and rule descriptions are dated website observations; use live chain tools for current on-chain facts.

The official **Supermemory managed web-crawler connector requires Scale or Enterprise**. The existing account returned `403` for connector creation, so CLAWD Mail uses its own scheduled crawler with Supermemory ingestion/search. No plan change was made.

```sh
# Run from the repository root; load private values from backend environment files.
node backend/crawl-hookedpad.js --force
node backend/crawl-hookedpad.js --status
node backend/crawl-hookedpad.js --search 'Which launches use social trading?'
npm run verify:knowledge
```

The CLI writes sanitized crawl status to `artifacts/qa/supermemory/crawl-status.json`. Browser checks use isolated fixture accounts and never send email. Set `CLAWD_KNOWLEDGE_QA_ORIGIN` to verify a hosted frontend with those isolated API fixtures. `scripts/verify-knowledge-live.cjs` separately verifies real hosted registration, pending/approved access, search and source links using a disposable account, then deletes that account.

Optional settings: `SUPERMEMORY_PUBLIC_NAMESPACE` (1–100 alphanumeric, colon, underscore or hyphen characters), `KNOWLEDGE_CRAWL_INTERVAL_MS` (minimum one hour), and `KNOWLEDGE_CRAWL_PAGES` (1–100, default 25). Keep the namespace identical on both services. Namespace changes require a new initial crawl.

Official references: [ingestion](https://supermemory.ai/docs/ingest), [search](https://supermemory.ai/docs/search), [web crawler](https://supermemory.ai/docs/connectors/web-crawler).