Module 3 Data Repo + Module 4 Dynamic Charts

Properties1
tagslinkedin-market, market-research, growreach, module3, data-repo, dynamic-charts, pipeline

5 min read

LinkedIn Tool Market — Module 3 Data Repo + Module 4 Dynamic Charts

Summary: This is the architecture that turns the LinkedIn tool-market research into a maintainable system. Module 3 is a single source-of-truth data file (linkedin-market/data/tools-data.js + data/meta/dashboard-meta.js) holding every per-tool metric the charts need. Module 4 is the set of dynamic D3 charts that render purely from that data — edit a number in Module 3, redeploy, and every chart updates without touching chart code. This document explains the schema, how to add a new tool, how to edit a price, and how the old vs new charts compare.

The 4-module pipeline

ModuleWhat it doesAI credits?Edit trigger
M1 · IngestFetch a new tool’s website: hero, positioning, personas, page copy, pricing (the 13-field template) → cluster-*.md / comment-segment/01-comment-tools-research.mdYes (research)Only for a genuinely new tool
M2 · AnalyzeDerive jobs-to-be-done, user journeys, market analysis → 08-jtbd-analysis.md, 09-persona-journeys.md, 10-market-analysis.md, comment-segment equivalentsYes (analysis)Only for a materially changed tool
M3 · Central repoThe single structured data file the charts consume (data/tools-data.js + data/meta/dashboard-meta.js)NoAny small change (price edit) — edit one number here
M4 · VisualizeThe dynamic D3 charts (v2) that read Module 3 and render the maps/dashboardsNoRe-run the chart build + redeploy

Key property: Module 3 is the single source of truth. A small change (e.g. a tool’s price 99) edits one row in Module 3 and only Module 4 re-runs (rebuild + redeploy). Modules 1–2 (full research + analysis) are re-run only when the tool itself is new or materially changed — not for a price tweak.

Module 3 — the schema

data/tools-data.js — per-tool records

One object per tool inside window.LINKEDIN_TOOLS. Example (Linqin):

{
  "id": "linqin",
  "name": "Linqin",
  "category": "comment",
  "categoryColor": "#f472b6",
  "source": "comment-segment/01-comment-tools-research.md#2", // provenance (M1/M2)
  "researchDate": "2026-08-16",
  "compliance": { "method": "cloud-agent", "officialApi": false }, // connection method
  "status": "live", // live | dead | pivoted | merged | off-market
  "pricing": {
    "entryPrice": 19,           // lowest paid tier
    "midPlanPrice": 39,         // middle tier (Wardley dot size)
    "agency": { "price": 500, "seats": 25, "flag": "S", "tier": "Team/Agency $500/mo · 25 seats ..." }
  },
  "positions": {
    "wardley": { "stage": 3, "evolution": 2 },              // stage 1..5, evolution 0..3
    "scatter": { "safety": 2, "maturity": 0.2, "entryPrice": 19 },
    "seatMap": { "price": 500, "seats": 25, "flag": "S" }   // mirrors pricing.agency
  },
  "personas": ["a","b","d","f"],   // 9-persona letters (a–i)
  "jobs": ["C1","C4","C6","J8"],   // canonical job ids
  "flags": { "dead": false, "pivoted": false }  // chart rendering flags
}

seatFlag meanings (for the agency price × seats chart):

  • S STATED — vendor explicitly states the seat count (HeyReach 25 senders @ $999, Skylead 50, GaggleAMP 100).
  • C COMPUTED — per-user price × 5 seats (no multi-seat tier published), e.g. Crystal, Dripify, Taplio.
  • E ESTIMATED — assumed employee count for data/credit/“unlimited users” plans (NOT prospect volume), e.g. UpLead, Scalelist, PhantomBuster, Reply, lemlist.

data/meta/dashboard-meta.js — aggregate market structure

window.LINKEDIN_META holds the non-per-tool aggregates the dashboard needs:

  • stages — the 5 journey stages (Find/Send/Engage/Publish/Compound) with tool counts + verdicts (journey river).
  • heat — the 9-persona × 5-stage heatmap matrix (which persona needs which stage, and the -1 MARKET GAP cells).
  • bands — price bands per category (price-ladder view).
  • deadTable — the dead/consolidated tools table (market-structure evidence).

Module 4 — the dynamic charts

All three v2 charts are pure functions of Module 3. They load ../data/tools-data.js (and dashboard-meta.js) via <script> and render; no tool data is hardcoded in the chart HTML.

ChartNew dynamic URL (v2)Old hardcoded URL (kept for comparison)
Main dashboardlinkedin-market/v2/12-market-viz.htmlinkedin-market/12-market-viz-d3.htm
Wardley maplinkedin-market/v2/wardley-map.htmlinkedin-market/wardley-map.htm
Agency price × seatslinkedin-market/v2/agency-price-seats-map.htmlinkedin-market/agency-price-seats-map.htm

All three v2 charts include the three UX fixes:

  1. Content-zoom — scroll/drag zoom transforms the SVG <g> group, so dots genuinely spread apart (zoom in) and compress (zoom out), not just a bigger canvas.
  2. Collision separation — overlapping dots are nudged apart with a force simulation, so co-located tools are each clickable (no more stacked-on-top-of-each-other).
  3. Always-visible names + search — every dot is labeled (bolded for the 10 comment tools); a search box flies to and highlights any tool (e.g. type “HeyReach”).

How to use it

Edit a tool’s price (the common case)

  1. Open linkedin-market/data/tools-data.js.
  2. Find the tool by name, edit the number(s) under pricing (e.g. entryPrice, midPlanPrice, pricing.agency.price).
  3. Run the validator to catch mistakes: node linkedin-market/scripts/validate-tools-data.js.
  4. Rebuild + deploy (Module 4): cd web && rm -rf public && npx quartz build -d /opt/data/hermes-repo, then npx wrangler pages deploy public --project-name=hermesvps --branch=main.

That’s it — all three v2 charts pick up the change. No AI credits, no re-research.

Add a new tool (full pipeline)

  1. M1 — run the 13-field research template against the new tool’s site → add to the relevant cluster .md or comment-segment/01-comment-tools-research.md.
  2. M2 — generate its JTBD + journey + market-delta analysis.
  3. M3 — add its full schema object to data/tools-data.js (copy an existing tool’s shape, fill in the fields).
  4. M4 — validate, rebuild, redeploy.

Regenerate from research (if data drifted)

scripts/extract-data.py re-extracts the old charts’ hardcoded data; scripts/build-tools-data.py merges that with the enrichment (personas/jobs/compliance) to regenerate data/tools-data.js. scripts/check-equivalence.js confirms v2 == v1. Normally you won’t need these — Module 3 is now the source of truth and you edit it directly.

Validator & equivalence scripts

ScriptPurposeRun
scripts/validate-tools-data.jsSchema + range + category-mix checks on tools-data.jsnode scripts/validate-tools-data.js
scripts/check-equivalence.jsConfirms v2 (from Module 3) matches old hardcoded charts for all 68 toolsnode scripts/check-equivalence.js
scripts/extract-data.pyOne-off: extract old chart data to scripts/extracted.jsonpython3 scripts/extract-data.py
scripts/build-tools-data.pyOne-off: regenerate tools-data.js from extracted + enrichmentpython3 scripts/build-tools-data.py

Design decisions

  • Single tools-data.js over per-tool JSON: chosen by Rahul for simplicity — edit one number, redeploy, zero build step. Also avoids Quartz’s unreliable raw-.json emission (.js files are reliably copied as static assets).
  • Old charts kept live: the v2 charts are faithful re-renders of v1 (proven by check-equivalence.js). The old URLs stay up so Rahul can compare; delete them only after confirming v2 is correct.
  • Zero AI credits for M4: visualization is deterministic browser rendering of the numbers — only M1/M2 (research/analysis for genuinely new tools) consume AI credits.

Graph View