Knowledge Graph

The knowledge graph is a live, interactive visualization of the OpenCosmos wiki — every entity, concept, and connection rendered as a glowing constellation on a dark canvas. It lives at opencosmos.ai/knowledge/graph and updates automatically when wiki articles are merged to main.

This guide explains what it is, how it works, and how to maintain it.

---

What It Is

The graph represents the wiki (knowledge/wiki/), not the source corpus (knowledge/sources/). Every markdown file in wiki/entities/, wiki/concepts/, and wiki/connections/ becomes a node. Edges are inferred from shared synthesized sources — two nodes are connected if they both cite the same source document in their frontmatter.

The result is a force-directed map of intellectual kinship: nodes that share lineage cluster together, nodes that bridge traditions sit at the crossroads, isolated nodes float at the periphery.

Visual Design

• Glowing nodes — each domain has its own color; node size reflects connection count; a breathing animation (synchronized per-node, offset by hash) gives the graph a living quality
• Constellation aesthetic — additive blending (gl.SRC_ALPHA, gl.ONE) makes overlapping nodes bloom instead of occlude
• Focus mode — clicking a node isolates its ego-network; camera animates to center it; neighbors pulse at full opacity
• Search overlay — ⌘K (or the "Search graph" button on mobile) opens a fuzzy-search panel over the graph
• Canvas fallback — Safari and iOS do not always support the WebGL extensions sigma requires; the graph silently falls back to a three-layer canvas renderer with equivalent visuals

---

The Data Pipeline

knowledge/wiki/**/*.md
        │
        ▼
scripts/knowledge/generate-wiki-graph.ts   (pnpm graph)
        │  — reads frontmatter + ## Summary
        │  — infers edges from shared sources
        │  — runs ForceAtlas2 layout (graphology)
        │  — seeds positions from existing Redis data (layout stability)
        │  — computes vibrancy score per node
        │  — writes gzip+Base64 payload to Upstash Redis
        │
        ├──▶ knowledge:graph          (full graph, gzip+Base64, ~7 KB at 25 nodes)
        └──▶ knowledge:graph:preview  (top-40 nodes, plain JSON, for SSR skeleton)
                │
                ▼
        apps/web/app/api/knowledge/graph/route.ts
                │  — server route, 1h ISR revalidation
                │  — gunzips and streams raw JSON
                │
                ▼
        apps/web/app/knowledge/graph/graphWorker.ts
                │  — Web Worker fetches /api/knowledge/graph
                │  — parses JSON off the main thread
                │  — posts parsed data back to main thread
                │
                ▼
        @opencosmos/ui/knowledge-graph   (KnowledgeGraph component)
                │  — builds graphology graph from nodes + links
                │  — detects WebGL availability
                │  — renders sigma.js or CanvasGraph fallback
                │  — crossfades from SVG skeleton on first load

---

Running the Generator Locally

# From the repo root
pnpm graph

This runs scripts/knowledge/generate-wiki-graph.ts via tsx. It reads every .md file in knowledge/wiki/entities/, knowledge/wiki/concepts/, and knowledge/wiki/connections/, computes the graph, and writes to Upstash Redis.

Prerequisites:

# .env at repo root
UPSTASH_REDIS_REST_URL=https://...
UPSTASH_REDIS_REST_TOKEN=...

Expected output:

[graph] Reading wiki articles…
[graph] 25 nodes, 62 edges
[graph] Running ForceAtlas2 layout (100 iterations)…
[graph] Vibrancy computed (max: 0.91, median: 0.78)
[graph] Writing to Redis…
[graph] knowledge:graph → 7.3 KB gzipped
[graph] knowledge:graph:preview → 40 nodes
[graph] Done in 1.8s

Once the generator finishes, start the web app:

pnpm dev --filter web
# → http://localhost:3000/knowledge/graph

The graph page will load the SVG skeleton (from Redis preview) immediately, then fetch the full graph in a Web Worker and crossfade to the live sigma renderer.

---

How New Wiki Articles Appear

1. Write the article following the wiki workflow guide. Place it in knowledge/wiki/entities/, knowledge/wiki/concepts/, or knowledge/wiki/connections/.

2. Include synthesized_from frontmatter — this is what creates edges:

   synthesized_from:
     - sources/philosophy-the-republic.md
     - sources/taoism-tao-te-ching.md
   

   Any two articles that share a source will be connected in the graph.

3. Run pnpm graph locally to regenerate and push to Redis, then verify at localhost:3000/knowledge/graph.

4. Merge to main — the GitHub Action (.github/workflows/knowledge-sync.yml) detects changes under knowledge/**, runs pnpm graph automatically, and calls POST /api/revalidate to trigger ISR. The live graph at opencosmos.ai updates within seconds.

---

Vibrancy

Each node has a vibrancy score (0–1) that controls how strongly it breathes in the constellation animation and how prominently its glow blooms. It is computed from three components:

Foundational floor: Any node in wiki/entities/ (primary thinkers, primary texts) has its raw score raised to a minimum of 0.75, so foundational nodes never fade into the background regardless of age.

High vibrancy: a recently-updated, heavily-connected concept article. Low vibrancy: a newly-created entity with no connections yet.

---

Domain Colors

Nodes are colored by domain — set in the domain frontmatter field of each wiki article. The color palette lives in @opencosmos/ui/knowledge-graph as DOMAIN_COLORS.

To add a new domain: add an entry to DOMAIN_COLORS in packages/ui/src/components/data-display/knowledge-graph/constants.ts in the opencosmos-ui repo, publish a new version, and update the version ref in apps/web/package.json.

---

The Rendering Stack

WebGL Path (sigma.js v3)

The primary renderer uses sigma.js v3 with a custom GlowNodeProgram written in GLSL:

• Vertex shader — positions each node with a sinusoidal offset that varies by nodeIndex (deterministic per-session breathing, not synchronized jank); amplitude is modulated by vibrancy
• Additive blending — overlapping nodes bloom rather than occlude, producing the constellation effect
• ForceAtlas2 layout — computed server-side by the generator, not in the browser; positions are stable across reloads (seeded from existing Redis data)
• Camera animation — focus mode uses sigma's camera API to animate to the selected node at ratio: 0.3 (zoom in)

Canvas Fallback (three-layer canvas)

When WebGL is unavailable:

• Edge layer (offscreen canvas, drawn once) — all edges, then composited
• Node layer (rAF loop) — domain-batched circles with pulsing globalAlpha
• Highlight layer (on-demand) — redrawn only when selection changes

Hit testing respects coarse pointer devices: on touch screens, the hit radius is max(node.radius, 20px) to ensure tappability.

SVG Skeleton (SSR)

On first page load, before the Web Worker has parsed the full graph, a lightweight SVG skeleton is rendered from the Redis preview data (top 40 nodes by connection count). This is generated server-side in page.tsx and appears immediately — no layout shift, no blank screen. The skeleton crossfades out as the live sigma graph fades in (600ms transition).

Web Worker

Parsing a 2–3 MB JSON payload on the main thread blocks the UI for 80–150ms. The graph JSON is fetched and parsed in a dedicated Web Worker (graphWorker.ts), which posts the parsed data to the main thread when ready. The worker is created in a useEffect and terminated in the cleanup function.

---

File Map

Why domain-colors.ts is local

DOMAIN_COLORS is a plain constant object — no WebGL, no React. But it lives in @opencosmos/ui/knowledge-graph, which re-exports from sigma. Even though KnowledgeGraph itself is wrapped in dynamic({ ssr: false }), Next.js/Turbopack still traces the static import chain of every client component during the server build. That trace reaches sigma's module-level code, which references WebGL2RenderingContext — a browser global that doesn't exist on the server — and crashes the build.

Keeping DOMAIN_COLORS in a local file breaks the import chain: the server component graph never touches sigma. The dynamic() import is the only path that reaches the package, and it only runs in the browser.

If you update DOMAIN_COLORS in @opencosmos/ui, remember to mirror the change in apps/web/app/knowledge/graph/domain-colors.ts.

---

Environment Variables

The REVALIDATE_SECRET should be a random string (e.g., openssl rand -hex 32). The GitHub Actions workflow sends it as x-revalidate-secret in the curl POST after each graph generation.

---

Automatic Updates (GitHub Actions)

The workflow at .github/workflows/knowledge-sync.yml triggers on any push to main that touches knowledge/**. It:

1. Checks out the repo
2. Installs dependencies (pnpm install --frozen-lockfile)
3. Runs pnpm graph (writes to Redis)
4. Calls curl -X POST $NEXT_PUBLIC_APP_URL/api/revalidate with x-revalidate-secret header

The concurrency.cancel-in-progress: true setting means rapid wiki edits don't queue up — only the latest push runs.

After the workflow completes, opencosmos.ai/knowledge/graph serves the updated graph within the ISR revalidation window (1 hour max, usually seconds via the explicit revalidation call).

---

Production Checklist

The graph is fully implemented. Before it works in production, complete these steps:

• Publish @opencosmos/ui@1.4.0 — the KnowledgeGraph component and knowledge-graph subpath export live in the opencosmos-ui repo. Run pnpm changeset there, merge the changeset PR, and the npm publish will fire automatically. Then update apps/web/package.json from file: to ^1.4.0.
• Install sigma.js dependencies — @react-sigma/core, sigma, graphology, graphology-layout-forceatlas2 are peer deps. After switching from file: to the npm version, install them in apps/web/: pnpm add sigma graphology @react-sigma/core graphology-layout-forceatlas2 --filter web
• Add environment variables to Vercel — UPSTASH_REDIS_REST_URL, UPSTASH_REDIS_REST_TOKEN, REVALIDATE_SECRET on the opencosmos.ai deployment
• Add GitHub Actions secrets — UPSTASH_REDIS_REST_URL, UPSTASH_REDIS_REST_TOKEN, REVALIDATE_SECRET, NEXT_PUBLIC_APP_URL=https://opencosmos.ai
• Run pnpm graph once to seed the initial Redis data before deploying
• Deploy and verify — navigate to opencosmos.ai/knowledge/graph, check the skeleton appears immediately, check the sigma graph crossfades in after ~1s

---

Troubleshooting

"Graph unavailable" error on the page

The Web Worker failed to fetch /api/knowledge/graph. Usually means:

• UPSTASH_REDIS_REST_URL or UPSTASH_REDIS_REST_TOKEN is missing from the Vercel deployment
• pnpm graph has never been run (Redis key does not exist)
• The knowledge:graph Redis key expired (it has no TTL by default — check the Upstash console)

Graph loads but is empty / shows only a few nodes

pnpm graph only indexes knowledge/wiki/**/*.md. Wiki articles must be in entities/, concepts/, or connections/ subdirectories. Source corpus files (knowledge/sources/) do not become nodes.

Node positions scramble on every regeneration

The generator seeds positions from existing Redis data. If UPSTASH_REDIS_REST_URL is missing when running locally, it falls back to random seeding. Check your .env.

Skeleton appears but live graph never loads

Check browser DevTools → Application → Service Workers; look for a stale worker. Hard-reload with Cmd+Shift+R. If the issue persists, check that the graphWorker.ts Web Worker build is included in the Next.js bundle (it should be, via new URL('./graphWorker.ts', import.meta.url)).

Canvas fallback looks different from screenshots

Expected — the canvas fallback is a faithful reproduction of the sigma aesthetic but not pixel-identical. The breathing animation uses globalAlpha pulsing rather than GLSL vertex offsets, and lacks the additive bloom. It is designed for correctness and accessibility, not visual parity.