2026-08-17-module3-data-repo-module4-dynamic-charts

10 min read

LinkedIn Tool Market — Module 3 Data Repo + Dynamic Module 4 Charts

For Hermes: Implement this plan task-by-task. The keystone is Module 3 (the central data repository); Module 4 charts become pure functions of it. Old charts stay untouched on their current URLs so Rahul can compare old vs new.

Confirmed decisions (Rahul): (1) Module 3 storage = single data/tools-data.js (one file, edit one number, redeploy — zero build step, guaranteed deployable through Quartz). (2) Plans live in a GitHub-visible folder linkedin-market/plans/ so Rahul can read them (a hook will automate this later). (3) Visualization (Module 4) costs zero AI credits — it’s deterministic browser rendering of the numbers; AI credits are only spent on Module 1/2 research+analysis for genuinely new tools.

Goal: Create a single source-of-truth data repository (Module 3) for all ~68 LinkedIn tools, and refactor the existing three D3 charts into “dynamic” Module-4 charts that read that repo — on new URLs so the old charts remain available for side-by-side comparison.

Architecture: JSON schema (tools/*.json — one file per tool) is the single source of truth. A small loader (tools.js) exposes it to the browser. Each new chart (*2.htm) is a pure render function of that data — no hardcoded tool arrays. Small edits (price change) edit one JSON row; only the charts re-render. Full research (Module 1/2) is only re-run for a genuinely new tool.

Tech Stack: D3 v7 (already used), static JSON + vanilla JS (no build step, matches existing .htm assets served by Quartz/Cloudflare Pages), Node for a validation script.

Current context / ground truth (verified)

  • 68 tools across 7 categories: data(12), smb(11), prem(12), content(12), pods(3), adv(8), comment(10).
  • Data is currently hardcoded in 3 places: 12-market-viz-d3.htm, wardley-map.htm, agency-price-seats-map.htm — each has its own const TOOLS = [...]. This is the duplication that makes updates error-prone (e.g. the Linqin fix, the commodity→product fix).
  • The source of truth is the markdown research files (13-field template per tool) + the derived analyses (jtbd, journeys, market-analysis) + the comment-segment docs. These are Module 1/2.
  • Old chart URLs (keep untouched): /linkedin-market/12-market-viz-d3.htm, /wardley-map.htm, /agency-price-seats-map.htm.
  • New dynamic chart URLs (create): /linkedin-market/v2/12-market-viz.htm, /v2/wardley-map.htm, /v2/agency-price-seats-map.htm.
  • Quartz build: npx quartz build -d /opt/data/hermes-repo in web/; deploy via wrangler. .htm files are copied byte-for-byte as static assets.

Module 3 schema (the keystone)

One JSON file per tool under linkedin-market/data/tools/<slug>.json. Fields cover everything the charts need plus source-of-truth context, so Module 4 needs nothing else. This is the schema (superset of Rahul’s fields):

{
  "id": "linqin",
  "name": "Linqin",
  "slug": "linqin",
  "category": "comment",
  "categoryColor": "#f472b6",            // derived from category map, not stored per-tool
 
  // ---- Source of truth (Module 1/2 provenance) ----
  "source": "comment-segment/01-comment-tools-research.md#2",
  "researchDate": "2026-08-16",
  "hero": "Win on LinkedIn without *living on it*.",
  "positioning": "Linqin comments in your voice...",
  "offer": "A quiet AI agent that comments and posts in your voice daily",
  "url": "https://linqin.ai",
  "compliance": { "method": "cloud-agent", "safetyScore": 2, "officialApi": false },
  "status": "live",                        // live | dead | pivoted | merged | off-market
 
  // ---- Rahul's Module-3 fields (chart-ready) ----
  "pricing": {
    "tiers": [
      { "name": "Starter", "price": 19, "per": "month" },
      { "name": "Growth",  "price": 39, "per": "month" }
    ],
    "agency":   { "price": 500, "seats": 25, "seatFlag": "S", "label": "Team/Agency $500/mo · 25 seats" },
    "free":     false,
    "entryPrice": 19
  },
  "midPlanPrice": 39,                      // dot-size + Wardley mid-plan value
  "seatPriceMap": { "price": 500, "seats": 25, "flag": "S" },   // used by agency-price-seats chart
 
  // ---- Positioning (Wardley) ----
  "wardley": { "stage": 3, "evolution": 2 },   // stage 1..5 (Find..Compound), evolution 0..3
 
  // ---- Scatter / quadrant (main dashboard) ----
  "scatter": { "safety": 2, "maturity": 0.2, "entryPrice": 19 },
 
  // ---- Targeting ----
  "personas": ["a","b","d","f"],          // 9-persona letters
  "jobs": ["J8","C1","C4"],               // canonical job ids
 
  // ---- Flags ----
  "isDead": false,
  "isPivoted": false
}

Schema completeness for Module 4 (each chart’s needs)

ChartNeeds from schema
wardley-mapwardley.stage, wardley.evolution, midPlanPrice, category, name, isDead, isPivoted
12-market-viz (scatter/quadrant/heatmap/etc.)scatter.safety, scatter.maturity, scatter.entryPrice, category, isDead, isPivoted, wardley (for the hero Wardley mini-view), persona×stage matrix (a separate static table)
agency-price-seats-mapseatPriceMap.price, seatPriceMap.seats, seatPriceMap.flag, pricing.agency.label, category, name
Future (next Module-4 steps)pricing.tiers (price-ladder per tier), personas, jobs, compliance.safetyScore

Note on persona×stage heatmap & journey river: the heatmap matrix and the STAGES (Find/Send/Engage/Publish/Compound tool-counts) are currently static data in the old 12-market-viz-d3.htm (the HEAT and STAGES arrays), separate from the per-tool TOOLS array. These are derived/aggregate, not per-tool, so they get a small separate derived JSON (data/meta/dashboard-meta.json) so the dynamic dashboard can render all 8 views from Module 3 + this one aggregate file.

Module 4 — dynamic chart design

Each new chart:

  • Loads ../data/tools.json (a single concatenated index) or ../data/tools/*.json via fetch.
  • Because these are static .htm assets served over HTTP, use fetch() at runtime. Caveat: Quartz copies .htm but does NOT publish raw .json as pages — so the JSON must live under a path Quartz will emit. Quartz’s Assets emitter copies files referenced from .md pages; untracked .json in the repo may be dropped. Mitigation (decided): keep the data as a single JS file data/tools-data.js that defines window.LINKEDIN_TOOLS = [ ... ] (a const), loaded via a <script src> tag. This is still ONE source of truth (single file, not three inline copies), is copied byte-for-byte by Quartz, and avoids the .json-emission problem entirely. It also works from file:// for local dev. This is the pragmatic call — Rahul can edit one .js data file, not three chart files.
    • Alternative considered: a .json file + fetch. Rejected because Quartz may drop untracked .json from the build output (verified behavior: only .md → pages; .htm → static asset; .json unconfirmed/dropped).
  • Rendering logic is a pure function: render(svg, data) — no tool data embedded in the chart HTML.

New URLs (old stay for comparison)

ChartOld (keep)New dynamic (create)
Main dashboard12-market-viz-d3.htmv2/12-market-viz.htm
Wardley mapwardley-map.htmv2/wardley-map.htm
Agency price×seatsagency-price-seats-map.htmv2/agency-price-seats-map.htm

Old charts remain byte-identical; we do NOT touch them. Only if Rahul confirms the new charts are correct do we later delete the old.

File layout

linkedin-market/
  data/
    tools-data.js                  # window.LINKEDIN_TOOLS = [ {..}, ... ]  (Module 3 single source)
    tools/
      apollo.json, uplead.json, ... # optional per-tool JSON (for human editability) — SEE task 1 decision
  v2/
    12-market-viz.htm              # dynamic dashboard
    wardley-map.htm                # dynamic Wardley
    agency-price-seats-map.htm     # dynamic agency×seats
  scripts/
    validate-tools-data.js         # node script: schema + cross-chart consistency checks
  12-market-viz-d3.htm             # OLD — untouched
  wardley-map.htm                  # OLD — untouched
  agency-price-seats-map.htm       # OLD — untouched

Step-by-step plan (bite-sized tasks)

Task 1: Extract the exact current data from the three charts

Objective: Get the ground-truth per-tool values so the dynamic charts match the old ones exactly. Files: Create scripts/extract-data.js (a one-off node script), output scripts/extracted.json. Steps:

  1. Write a node script that regex-extracts the TOOLS, WARD, DEAD, HEAT, STAGES, BANDS arrays from all three old .htm files.
  2. Run it, write scripts/extracted.json.
  3. Manually cross-check a sample (e.g. Linqin should show entry 19 / wardley stage 3 evo 2 / agency 500@25) against the research file. Verify: node scripts/extract-data.js prints counts (68 tools, cat distribution) and no name mismatches across the three files. Commit: chore: extract current chart data as ground truth

Task 2: Build the Module 3 storage (single tools-data.js)

Objective: Create the one canonical data file from the extracted data. Decision (Rahul): single data/tools-data.js — one window.LINKEDIN_TOOLS array of full-schema objects. Edit one number, redeploy. No build step. Guaranteed byte-copied by Quartz (.js is a static asset, not a dropped .json). Files: Create data/tools-data.js. Steps:

  1. Transform scripts/extracted.json into the full-schema objects (add personas/jobs/compliance where the research files give them; fill from cluster files for the 58 originals, comment-segment for the 10).
  2. Write data/tools-data.js exposing window.LINKEDIN_TOOLS. Verify: node script counts 68, all required schema fields present for every tool. Commit: feat: add Module 3 central data repository (tools-data.js)

Task 3: Write the schema validator

Objective: A repeatable check so edits to tools-data.js can’t silently break the charts. Files: Create scripts/validate-tools-data.js. Steps:

  1. Assert every tool has: id, name, category, pricing.tiers, pricing.agency, midPlanPrice, wardley{stage,evolution}, scatter{safety,maturity,entryPrice}, personas, jobs, isDead, isPivoted.
  2. Assert category ∈ 7 known, stage ∈ 1..5, evolution ∈ 0..3, safety ∈ 0..5, maturity ∈ 0..1.
  3. Assert cross-file consistency: same tool appears in all three expected chart datasets (the validator simulates the chart data derivation from the schema and checks counts). Verify: node scripts/validate-tools-data.js → PASS. Commit: test: add Module 3 schema validator

Task 4: Build dynamic Wardley map (v2)

Objective: v2/wardley-map.htm renders from ../data/tools-data.js with no inline tool array. Files: Create v2/wardley-map.htm, modify data/tools-data.js (already done in task 2). Steps:

  1. Copy wardley-map.htm → v2/wardley-map.htm.
  2. Delete the inline const TOOLS = [...], const WARD, const DEAD; replace with a <script src="../data/tools-data.js"></script> and read window.LINKEDIN_TOOLS.
  3. Keep the same rendering code (zoom, labels, collision, search) — only the data source changes. Verify: local serve; browser asserts 68 dots, comment tools = 10, search works. Commit: feat: dynamic Wardley map (v2) reads Module 3

Task 5: Build dynamic agency-price×seats map (v2)

Objective: v2/agency-price-seats-map.htm renders from Module 3. Files: Create v2/agency-price-seats-map.htm. Steps:

  1. Copy agency-price-seats-map.htm → v2/...; remove inline TOOLS; load ../data/tools-data.js.
  2. Map seatPriceMap + pricing.agency.label to the dots. Verify: local serve; 68 dots, Linqin shows $500@25. Commit: feat: dynamic agency-price×seats map (v2) reads Module 3

Task 6: Build dynamic main dashboard (v2)

Objective: v2/12-market-viz.htm renders all 8 views from Module 3 + the aggregate meta file. Files: Create v2/12-market-viz.htm, data/meta/dashboard-meta.js (STAGES/HEAT/BANDS). Steps:

  1. Extract STAGES, HEAT, BANDS from old dashboard into data/meta/dashboard-meta.js.
  2. Build v2/12-market-viz.htm reading ../data/tools-data.js + ../data/meta/dashboard-meta.js for the aggregate views. Verify: local serve; 68 dots across scatter/quadrant/wardley; heatmap unchanged. Commit: feat: dynamic main dashboard (v2) reads Module 3

Task 7: Verify old vs new produce identical output

Objective: Prove the dynamic charts match the old ones (so we can trust deleting old later). Files: test only. Steps:

  1. Serve both old and new. For each chart, browser-console compare: dot count, per-tool (x,y,r) after the same separation, category colors, tooltip content for a sample (Linqin, HeyReach, GaggleAMP).
  2. Diff the rendered DOM count of circles/labels for wardley and agency maps. Verify: identical counts; spot-check 3 tools’ tooltip text matches. Commit: test: verify v2 charts match v1 output

Task 8: Rebuild Quartz + deploy, keep old URLs live

Objective: Ship v2 on new URLs; old charts untouched. Files: build/deploy only. Steps:

  1. cd web && rm -rf public && npx quartz build -d /opt/data/hermes-repo.
  2. wrangler pages deploy public --project-name=hermesvps --branch=main.
  3. Verify all 6 URLs (3 old + 3 new) return 200; new ones render 68 dots. Verify: curl all URLs; browser-check one new chart. Commit: chore: deploy dynamic Module-4 charts (v2)

Task 9: Document the pipeline + edit workflow

Objective: Rahul can edit one row and re-run Module 4. Files: Create linkedin-market/14-module-3-data-repo.md. Steps:

  1. Document the schema, where data lives, how to add a new tool (Module 1→2→3→4), how to edit a price (edit tools-data.js, run validator, rebuild+deploy).
  2. Document the old-vs-new chart comparison. Commit: docs: Module 3 data repo + Module 4 workflow

Files likely to change

  • Create: data/tools-data.js, v2/wardley-map.htm, v2/agency-price-seats-map.htm, v2/12-market-viz.htm, scripts/extract-data.js, scripts/validate-tools-data.js, 14-module-3-data-repo.md, data/meta/dashboard-meta.js.
  • Modify: none of the old charts (keep untouched).
  • Deploy: rebuild Quartz, push, wrangler deploy.

Tests / validation

  • node scripts/validate-tools-data.js → schema + cross-chart consistency.
  • Local python3 -m http.server + browser-console asserts (dot counts, tooltip, search, zoom).
  • Task 7 old-vs-new equivalence check.
  • Live: curl all 6 URLs return 200.

Risks / tradeoffs / open questions

  • .json emission risk (resolved): Quartz may drop untracked .json; we use tools-data.js (single JS file) instead — still ONE source of truth, byte-copied, works from file://.
  • Single-file vs per-tool (decided): Rahul chose the single tools-data.js (simplest, edit one number, redeploy). Per-tool split is possible later without re-running analysis — charts only read the merged file.
  • Aggregate meta (STAGES/HEAT/BANDS) is not per-tool; it lives in data/meta/dashboard-meta.js — still Module 3, just a derived/aggregate section.
  • Behavior changes in v2: none — v2 is a faithful re-render of v1 from the new data source. Any intentional visual change (e.g. a new field) is a SEPARATE future Module-4 task, not part of this refactor.
  • Old charts: kept live until Rahul confirms v2 matches. No deletion in this plan.
  • Zero AI credit cost for Module 4 (confirmed): charts are deterministic browser rendering; only Module 1/2 consume AI credits for genuinely new tools.

Graph View