Every block on this map is clickable — click one to zoom in and see what's inside, in plain words.
reading the code…
working as designed
works, but flagged for a look
missing / risky — has a fix plan
Step 1 · Where the numbers come from
Amazon Seller API
orders · revenue · fees · stock · search terms — free
Amazon Ads API
every advertising number — free
Keepa
ratings · competitor sales — paid sub
Amazon website (read directly)
search ranks · new-product leads · BSR & "bought last month" · page extras — via proxies
▼reports get requested, downloaded and saved — every night, plus every 15 minutes for "today"
Step 2 · The collectors (the only "server code" in the app)
9 programs in Supabase
"edge functions" — talk to Amazon & Keepa, plus 2 doors for buttons & the Chrome add-on
10 scheduled jobs
run on GitHub's computers — the scrapers, the checks, the alerts and the weekly off-site backup
▼everything lands in one place
Step 3 · The database — the heart of the app (Supabase) · tables
~100 built-in timers
the alarm clock — runs the whole night shift
Product catalog
models · ASINs · costs — everything joins through it
Raw report tables
Amazon's numbers, exactly as sent
Night-made summaries
pre-computed answers so pages load in seconds
The "amazon" shelf
ranks · competitors · leads · health · market — newer tables, own shelf
The "platform" shelf
the app watching itself — who used which page
Plumbing & logs
job queue · activity logs (self-clean on a retention window)
Login & permissions
no login = no data (since Jul 22, 2026)
▼pages only read ready-made numbers — the heavy math already happened at night
Step 4 · The website (app.comtrig.com, hosted on Vercel) · URLs
pages
reading the code…
Shared design & login
one look, one sidebar, one login for every page
The team
each person sees only the pages their access tags allow
▼and watching over all of it…
Step 5 · The safety nets
System Status lights
is every data feed fresh? — daily history kept
Automated tests
broken code can't reach the live site
Outage banner
if the database is down, every page says so
Backups ✓
daily by Supabase (7 days) · weekly copy to the company Google Drive since Sep 2026 (8 weeks)
Database-vs-files check ✓
compared nightly since Jul 2026 — see System Status
The logbook ✓
every applied change signs itself in
All processeslast reviewed 4 Sep 2026 · times are Pacific
| Process | What it does & where it shows | When & how it runs | Est. cost / mo |
|---|---|---|---|
| Amazon data — pulled from Amazon's own APIs (no extra cost) | |||
| Sales / orders SP-API → sales_data |
Pulls daily orders, revenue and business/traffic reports. Powers the Sales P&L and feeds ad/sales totals into every page downstream. | Nightly early AM (yesterday + corrections back ~60 days), and every 15 min all day for today's live numbers. | Free |
| PPC — performance Ads API → ppc_* tables + amazon.ppc_placement |
Sponsored Products / Brands / Display reports (spend, clicks, sales, ACOS…). Powers the PPC dashboard and the PPC totals on Sales & Account Health. | Nightly ~12–2 AM, plus 2- / 7- / 14-day re-fetches (ad sales keep updating for ~2 weeks). | Free |
| PPC — settings Ads API → ppc_* snapshots |
Daily snapshot of your ad settings — bids, budgets, targets, negative keywords, Amazon's suggested bids. Powers bid history and the Market page's "Targeted by". | Nightly ~12 AM (one snapshot after another so they never collide). | Free |
| Inventory SP-API → inventory tables |
Nightly full stock snapshot + a live sweep every 15 min. Powers the Inventory / Orders dashboard and the stockout projections. | Nightly ~12 AM snapshot; live refresh every 15 min through the day. | Free |
| SQP — monthly Brand Analytics → search_query_performance |
Amazon Search Query Performance — the top search terms per product. Powers the SQP dashboard and the "top 10 keywords per model" on Keywords. | Drips all day: one Amazon report ordered every 5 minutes until the month is complete (migration 269). | Free |
| SQP — weekly Brand Analytics → search_query_performance |
Same report but per week — adds week-by-week search-volume detail on SQP and the Keywords trend chart. Since migration 269 both also carry impressions and add-to-carts. | Same drip. A nightly 12:30 AM pass re-queues any of the last 8 weeks / 3 months a brand is missing, so a lost week fills itself in. | Free |
| Product images fetch-sales → sales_data.image_url |
Each ASIN's main photo, picked up by the nightly sales fetch. Powers the product thumbnails everywhere in the app. | Nightly, inside the sales fetch. | Free |
| Order lookup SP-API → amazon.orders / order_items |
Every order with its lines, so a single order can be looked up by number, buyer or SKU. Powers Amazon → Orders. | Every 6 hours re-reads recently updated orders. A 2-year history backfill ran once (migration 133). | Free |
| Fees, late refunds & deals SP-API → amazon.fee_charges · sales_data.refunds_late · amazon.promo_deals |
Three small daily sweeps that keep the Sales P&L honest: the real settlement fees per SKU, refunds Amazon posts more than 15 days after the sale, and which products are on an Amazon deal today (the deal fee shows on Sales). | Fees ~3:50 AM · late refunds every 10 min · deals ~4:35 AM. | Free |
| Account health sweep health-sweep → amazon.health_alerts · listing_snapshots |
Photographs every listing daily (title, bullets, photos, price, rating) and raises an alert when something changed, a rating dropped, a bad review landed, or Amazon moved a category. Also pulls seller feedback and FBA returns. Powers Account Health. | Nightly ~4 AM sweep, alerts derived ~4:40 AM. | Free |
| Outside data — paid subscriptions | |||
| Keyword Rank Trackernew Geonode + Evomi → amazon.rank_results |
Our and competitors' organic rank for every tracked keyword, per state/province, on amazon.com and amazon.ca — the whole page 1 is stored. Powers Keywords → Our rank (today & 7-day) and the competitor lines. | Nightly ~3 AM + three catch-up passes (~7 AM, ~11 AM, ~3 PM) that only redo what failed, on GitHub Actions (not Supabase), through two independent residential proxies: Geonode first, Evomi (added Jul 2026) automatically takes any scrape Geonode fails — so one provider's bad night no longer blanks the ranks. US + Canada. | ≈ $18–20 |
| Product Page Scrapernew Geonode → amazon.competitors (video · A+ · bullets) |
Grabs each tracked product's page extras Keepa doesn't provide — videos, A+ "From the brand" images, and the "About this item" bullets — so the Market → Compare Products sheet can show them. Only ASINs not yet checked are fetched, so it's near-zero after the first pass. | Automatic — no button, no GitHub. Every 2 hours it picks up newly added products (a new one fills within ~2h), plus a daily full refresh to catch listing changes. Runs on GitHub Actions through the same Geonode proxy; only the product PAGE is read (~1 MB), the media streams from Amazon in the browser. Force a run now (rarely needed): open Actions → grey Run workflow button → Run workflow. |
≈ $0* |
| Lead Extractor scrapernew residential proxy → amazon.lead_candidates / lead_niches |
Finds new products worth selling: searches Amazon for each queued term, reads page 1 (title · price · rating · reviews · "bought in past month"), scores each product by est. monthly revenue earned despite a weak rating & few reviews. Only products that clear the editable qualifying criteria (min revenue · max reviews · max rating, set on the Leads page) are kept. Drops known brands two ways: a manual brand list at scrape time, plus the AI flagging any well-known brand nobody listed for one-click approval. Since 2026-08-11 an AI generator also proposes WHAT to search next — it reads the essence of every qualifying product we already have, how many matches each niche already holds (the same count the Leads table shows, 2–4 being the cheapest win), and every term ever searched (never repeating one), then suggests new searches for a person to queue. The AI niche labelling is a separate, manual step (owner 2026-08-11): a scrape leaves the Niche blank, and the Sort with AI button on the Leads page sends each product's facts and its main picture to whichever AI the Model ID box names (a name starting with gpt = OpenAI, otherwise Claude), cached per product so only new ones cost anything. Every answer — with the value it replaced — is kept in amazon.lead_ai_labels, so the page can show what the AI changed and put it back. Since 2026-09-03 the table asks two plain questions instead of the old one confusing “Set by” column — AI right? and Why — in your words, once beside the niche and once beside the brand — and the AI no longer learns by itself: every example it is shown is written in the prompt, where a person can read it, and a better prompt comes only from the ✎ Teach the AI button on the table, which proposes a rewrite that a person edits and approves. Feeds the one searchable list on New Products → Leads. | User-driven (since Jul 2026): during the day you queue the search terms you want on the Leads page (up to 300). That night the scraper searches only those and adds the results; queue nothing → no run (no junk, no bandwidth spent). Nightly ~3:30 AM + a ~9:30 AM pass that only retries terms whose scrape failed, on GitHub Actions. Runs on a residential proxy — Evomi since 24 Aug 2026, and the pool is expected to keep changing while we find the right one, so nothing names one as permanent. IPRoyal, the pool before it, is still wired and selectable by hand from the Leads page (marked NOT recommended). Every run records the pool it used, so a bad night can always be traced to one. Force a run now: open Actions → grey Run workflow button → Run workflow. |
≈ $10–40* |
| BSR → units curvesnew bsr-scraper (GitHub) + bsr-harvest (Keepa) → amazon.market_bsr_* |
Reads each tracked product's Best-Seller-Rank and its "bought in past month" badge straight off the Amazon page, then tops up with Keepa only what the scrape missed. The pairs are fitted into rank → monthly-units curves that the Market page uses to estimate every competitor's sales. | Scrape nightly ~9:40 PM on GitHub Actions (shares the lead scraper's proxy pool); Keepa top-up and curve fit follow in the early morning inside Supabase. A person can also press it from the Market page. | ≈ $0–3* |
| Whole-market search termsnew search-terms-fetcher (GitHub) → the brand-analytics project |
Amazon Brand Analytics "Top Search Terms" for the ENTIRE marketplace — every keyword, not only ours — so the Keywords → KW search volume tab can size any keyword. Too big for the app database, so it lives in a second, separate Supabase project. | Weekly, Wednesday morning, on GitHub Actions; an hourly pass filled in a year of history and switches itself off when done. | 2nd Supabase project |
| On-demand scrapers market-builder · returns-badge (GitHub, no schedule) |
Two jobs that never run by themselves: the AI market builder (the Market page's "Find it for me" button turns a market name into a tracked market, ~10 page loads) and the "Frequently returned item" badge check (which of our products carry Amazon's warning — read off the page, since no API has it). | Only when a person presses the button. Each build ≈ 3 minutes of proxy bandwidth. | ≈ cents per run |
| Product ratings & prices Keepa → amazon.asin_ratings |
Star rating, review count, Best-Seller-Rank, selling price, strike-through List Price and live-deal flag per ASIN. Powers ratings / BSR / the Unit Price color and Discount % on Sales. |
A kick every 5 minutes, each fetching only what is still missing within the Keepa token budget. Tiered automatically by recent sales: • sold in last 3 days → every 6 hours • sold in last 90 days → daily • no sale in 90 days → every 3 days A product moves lanes by itself the day its sales change. ≈645 of the 1,440 daily Keepa credits; ≈450/day stay free for on-demand Market pulls. |
Keepa sub* |
| Market competitors Keepa → amazon.competitor_daily |
Daily competitor snapshot (BSR, price, ~units/mo, offers, deals) for the ASINs you track on Market. Powers the whole Market page. | A kick every 5 minutes (shares the Keepa budget with ratings). Top sellers daily, the rest weekly. | Keepa sub* |
| Housekeeping — internal, no outside calls | |||
| Manual refresh queue manual_fetch_runs |
Runs the "Run now" jobs you trigger on the Refresh Data page. | Checks the queue every 5 min. | Free |
| Status snapshot system_status_history |
Saves the System Status lights each day so System Status can show Week / Month / Year history. | Daily 4 AM. | Free |
| Rank data cleanup rank-retention-purge |
Prunes old rank rows to keep the table small — but keeps our own products and tracked competitors forever. | Daily ~4:30 AM. | Free |
| Database drift checknew drift-check (GitHub) → drift_runs |
Every night it reads the live database's real shape and compares it to the SQL files saved in GitHub — the "book vs kitchen" check. Any unexplained difference turns the Database drift check row red on System Status the next morning, while the backups still reach back far enough to fix it cleanly. | Nightly ~6 AM, on GitHub Actions (not Supabase). After an intentional database change the baseline is refreshed (Actions → drift-check → Run workflow → type YES). | Free |
| Off-site database + code copynew db-backup (GitHub) → Google Drive |
Once a week the whole app database is copied to the company Shared Drive (Comtrig - Shared drive › Management › General - Idan only › Comtrig app › Data base back up), outside the Supabase account. Supabase's own daily backups live inside that account, so losing one login would lose the data and its copies together — this copy is the answer. The last 8 weekly copies are kept; older ones are binned by the job. The code goes too: the whole GitHub repository, every branch and its full history, as one file in the sibling folder Code back up (restorable with one git command). Then it proves the copy: downloads it back from Drive, restores it into a fresh empty database on GitHub's machine and counts every table against the live one — 152 of 152 identical on the first proof (5 Sep 2026, 9 minutes for the whole run). | Sundays ~8 AM, on GitHub Actions. If the copy or the proof fails, a GitHub issue opens (label health-alert) and emails the team. Restore steps and what the copy does not contain: backend/BACKUPS.md §5. | Free |
| Fetch-log cleanup fetch_logs_retention |
Trims the pipeline's own activity log (every fetch attempt ever made) so it stops growing forever. Keeps the last 30 days — plenty for debugging. | Daily ~4:45 AM. | Free |
| Amazon API change watchernew changelog-watcher (GitHub) |
Reads Amazon's SP-API and Ads-API change announcements and opens a GitHub issue (which emails us) ONLY when a change touches an API our pipeline uses, or announces a deprecation — filtering out the ~90% that don't concern us. | Weekly, Monday morning, on GitHub Actions. | Free |
| Health alerter health-alerts (GitHub) → ops.status_incidents + a GitHub issue |
Reads the Health page's lights. A red feed becomes an item on the page's "Needs sign-off" list (the red number in the sidebar) that only a person can clear, and — the first time only — a GitHub issue that emails the team. | Every 2 hours, on GitHub Actions. | Free |
| Code tests tests (GitHub) → test_runs |
Runs every automated check on every code change (broken code cannot reach the live site) and once a day against the live data, so a feed that quietly stopped is noticed even on a day nobody pushed code. Results show on Health → Code. | On every push, plus daily ~6:20 AM, on GitHub Actions. | Free |
Notes.
Free (Amazon) = pulled from Amazon's own APIs, which cost nothing beyond our seller account.
Keepa sub* = ratings and Market competitors both draw from one shared Keepa subscription (a single monthly fee covers both).
The Keyword Rank Tracker and the Product Video Scraper are pay-per-use feeds — both buy residential-proxy bandwidth, mainly from the Geonode plan (currently 25 GB, ≈ $18/mo; usage is well under that). The rank tracker's backup provider Evomi (pay-as-you-go, ≈ $1/GB, free 100 MB trial first) only carries the scrapes Geonode fails, so it normally adds $0–2/mo — a whole-night failover ≈ $1. ≈ $0* = the video scraper adds only a few product-page loads (a 5-product comparison ≈ a few cents), so it fits inside that same $18 with room to spare.
The Lead Extractor scraper runs on a residential proxy — Evomi since 24 Aug 2026, which is also the rank tracker's failover pool, so heavy load here can affect what keyword ranking falls back on (the known cost of that move). The pool is expected to keep changing. A full night reads ~550 Amazon pages ≈ ~200 MB, so running every night ≈ ~6 GB/month. ≈ $10–40* = the exact price depends on the plan (~$7/GB pay-as-you-go, much less on a monthly-GB plan). It has no fallback: if the pool it is set to runs out, or its login is missing, it simply stops until that is fixed — it never quietly switches pool behind your back.
Not listed here because they're fixed platform bills, not data feeds: Supabase (the database + the schedules that run all of the above) and Vercel (hosting this website).
For live health of each feed see System Status; to re-run one by hand see Refresh Data.