QCLenss — Detailed Workflow

QCLenss is an 8-stage AI pipeline that turns thousands of raw, unsolicited quick-commerce reviews into structured, decision-ready category-exploration insights for Blinkit, Swiggy Instamart, and Zepto.

This document explains how the engine works stage by stage — the models, the data flow, and the methodology choices that make the numbers trustworthy. It is the "how it works" companion to the source code and the live engine.

Data flow at a glance

brand name
   │  ① Resolve        free-text → the exact app(s), confirmed
   ▼
raw reviews  ───②────► ③ Clean ───► ④ Classify ───► ⑤ Cluster
 (Play·YouTube)         normalise    per-review AI     themes
   Collect              + de-dupe    labels            (LLM)
                                                          │
   ⑧ Report ◄──── ⑦ Synthesize ◄──── ⑥ Aggregate ◄───────┘
   HTML dashboard   narrative        the 8 questions,
                                     denominator-partitioned

Each product gets its own folder on disk (data/<product>/) holding every intermediate artifact — raw_feedback.jsonl, cleaned.jsonl, dropped.jsonl, classified.jsonl, themes.json, meta.json, and the rendered output/report.html — so any stage can be re-run or audited independently.

The 8 stages

① Resolve · src/resolve.ts

Free-text input ("Blinkit", "zepto", even "Bli nkit") is matched against known brands and live app-store search. The UI shows a confirm dialog with the candidate apps and whether each is already analysed (instant) or new (full run), so the user always analyses exactly the brand they mean. No scraping or AI runs here — it is a cheap lookup.

② Collect · src/scrape/

Pulls real user feedback from three public sources:

• Google Play (google-play-scraper) — the primary, highest-volume source.
• Apple App Store (iTunes RSS) — used when available (often sparse for these India apps).
• YouTube (Data API v3) — search.list for relevant videos, then commentThreads.list for comments; filtered to on-topic content.

Swiggy Instamart is dual-sourced: the standalone Instamart app plus the Swiggy app filtered to Instamart-relevant reviews (the report shows the by-app split). Reddit is wired but disabled (anonymous access is 403; no OAuth app configured).

③ Clean · src/clean.ts

Normalises and de-noises every item before it costs a token:

• Normalise — decode HTML entities, strip URLs and @user / u/handle mentions, remove emoji, collapse repeated punctuation and character runs.
• Drop rules — empty, too short (< 15 chars), no letters, spam/bot patterns, exact duplicates, and near-duplicates (Jaccard word-set similarity ≥ 0.85).

Every dropped item and its reason are written to dropped.jsonl for transparency.

④ Classify · src/analyze/classify.ts

A map-reduce pass: each cleaned review is sent to the LLM and labelled independently with a strict structured-output schema — is it relevant to category exploration, its barrier/theme, the user segment, and sentiment. Running per-review (rather than stuffing everything into one prompt) keeps labels consistent and lets the work parallelise.

• Model (as run): Claude Sonnet 5, extended thinking disabled — benchmarked at ~92% agreement with Opus on this task at a fraction of the cost. (Model is overridable via CLASSIFY_MODEL.)

⑤ Cluster · src/analyze/cluster.ts

The free-text themes coming out of classification are grouped into a small set of named, human-readable themes using an LLM clustering approach: chunk → cluster → merge/label. This turns hundreds of loosely-worded complaints into a handful of clean, countable buckets.

• Model: Claude Opus 4.8.

⑥ Aggregate · src/analyze/aggregate.ts

Answers the eight brief questions with counts and evidence:

1. Why do users repeatedly buy from the same categories?
2. What prevents users from exploring new categories?
3. How do users discover products today?
4. What role do habits play in shopping behaviour?
5. What information do users need before trying a new category?
6. What frustrations emerge repeatedly?
7. Which user segments are more likely to experiment?
8. What unmet needs emerge consistently across discussions?

Key methodology — denominator partitioning. Every question assigns each review in its denominator to exactly one bucket, adds a residual "other / not-stated" row, and prints a coverage line — so the rows sum to the stated n and no reviews are quietly dropped or cherry-picked. (Q7 is reported as an experimentation rate per segment, not a share.)

⑦ Synthesize · src/analyze/synthesize.ts

Turns the aggregated numbers into a short, evidence-anchored narrative — the "so what" a PM would write — without inventing anything beyond what the counts support.

• Model: Claude Opus 4.8.

⑧ Report · src/report/render.ts + template.ejs

Renders a self-contained HTML dashboard: each question with its partitioned rows, a per-question evidence browser (All + per-theme tabs), a paginated all-reviews table, red "n = evidence count" notes, and the by-app split. Served at /view/<brand> on the live engine.

Why the numbers hold up

• Map-reduce classification — every review is labelled individually against a fixed schema, not summarised in bulk.
• Denominator-partitioned answers — each question accounts for 100% of its reviews; the report shows the residual and coverage, so there is no cherry-picking.
• Aggressive, logged cleaning — HTML/emoji/URL noise removed; exact and near-duplicates (Jaccard ≥ 0.85) dropped, with every removal recorded.
• Live & reproducible — all intermediates are on disk; any brand can be re-run end to end, and every headline number is backed by browsable evidence in the report.

Models & sources

Reproduce it

npm install
cp .env.example .env      # add ANTHROPIC_API_KEY (+ YOUTUBE_API_KEY for YouTube)
npm run serve             # web UI at http://localhost:3000
# or a one-shot CLI run for a single brand:
npm run start -- --company=Instamart

Reports for the three brief brands are pre-computed under data/<brand>/output/report.html and served live at https://qcom-discovery-engine.slogicone.com.