Scrapely Ads — Extension ↔ Backend Integration Guide

Backend: /root/scrapely-ads-api (Node/TS + Express + Prisma + Postgres) · Live production base URL: https://scrapelyads.abdulawal.cloud/api (local dev: http://localhost:8787, no /api prefix)

What's already built and tested in the backend

• Admin auth — email/password login, returns a JWT (POST /admin/auth/login)
• Plans — named credit packages (e.g. "Starter" = 200 credits) admin creates once, with an optional reference price in BDT — POST /admin/plans, GET /admin/plans, PATCH /admin/plans/:id
• License issuing — admin takes payment manually (bKash/Nagad), then picks a plan (or a one-off custom credit amount) and generates a license key (raw key shown once, only its hash is stored) — POST /admin/licenses
• License list + usage — for every license: credits issued, credits used, current balance, leads extracted, active devices — GET /admin/licenses, GET /admin/licenses/:id
• Credit top-up / status change (suspend, extend expiry) — PATCH /admin/licenses/:id, POST /admin/licenses/:id/credits
• Extension activation — license key → session token — POST /auth/activate
• Session validation — session token → current balance/status — GET /auth/validate
• Logout / revoke session — POST /auth/logout
• Batch lead extraction — the core billing endpoint — POST /leads/extract-batch

The batch endpoint accepts up to 50 leads per call, deduplicates per license using a DB unique constraint on (licenseId, adId) — an ad already billed on this license is never billed twice, even across reinstalls/devices — and deducts exactly 1 credit per genuinely new lead, atomically, inside a row-locked transaction. If a batch has more new leads than the license can afford, it bills as many as it can and reports the rest as rejected (not saved, not charged — safe to resend after a top-up). Verified end-to-end with real requests: dedup, partial-affordability, and balance tracking all behave correctly.

Why the flow is designed this way

The extension is client-side code — anyone can inspect it, so it can never be trusted to decide how many credits to deduct. The backend is the only source of truth for billing. But calling the backend once per scraped lead would be too slow for the user. So:

• Credits are checked/displayed from a cached balance fetched at login (instant UI).
• Scraping happens entirely locally, at full speed, with an optimistic local counter.
• Every ~20–30 leads (or when a scrape run ends), the extension sends one batch call to the backend, which does the real, authoritative billing in a single fast transaction and returns the true balance.
• A capped local buffer (don't let more than ~25–30 unsynced leads pile up) bounds how far a tampered client could drift from the real balance before being forced to sync.

Integration flow for the extension

1. Activation (first run / no valid session)

POST /auth/activate
Content-Type: application/json

{ "licenseKey": "SCRA-XXXX-XXXX-XXXX-XXXX", "deviceLabel": "Chrome on <hostname>" }

Response 201:

{
  "token": "opaque-session-token",
  "license": { "label": "Customer Name", "creditBalance": 500, "expiresAt": null }
}

Errors: 401 invalid_license, 403 license_inactive, 403 license_expired.

Store token in chrome.storage.local (never localStorage, never in a content script/page context). This token authenticates every later call: Authorization: Bearer <token>.

2. Session validation (on extension startup, and periodically)

GET /auth/validate
Authorization: Bearer <token>

Response 200:

{ "status": "ACTIVE", "creditBalance": 500, "label": "Customer Name", "expiresAt": null }

401 invalid_session → session was revoked, force back to the activation screen. 403 license_inactive / 403 license_expired → same, with the specific reason shown to the user.

3. Scraping (local, no network call per lead)

Extension scrapes ads from Meta Ads Library as normal, buffers results locally, shows an optimistic decreasing credit count in the UI. No backend call here.

4. Batch sync — call this every 20–30 buffered leads, or when a scrape run ends

POST /leads/extract-batch
Authorization: Bearer <token>
Content-Type: application/json

{
  "leads": [
    { "adId": "1234567890", "data": { "pageName": "...", "adText": "...", "...": "..." } },
    { "adId": "1234567891", "data": { "...": "..." } }
  ]
}

leads max length: 50 per call.

Response 200:

{
  "accepted": ["1234567890"],
  "duplicates": ["1234567891"],
  "rejected": [],
  "creditBalance": 498
}

• accepted — newly billed this call, count against the credit balance
• duplicates — already extracted on this license before (any device); not billed again, safe to show as "already have this one"
• rejected — genuinely new, but the license didn't have enough credit; not saved, safe to resend in a later batch once the user tops up
• creditBalance — the real, authoritative balance; overwrite whatever the UI was showing optimistically with this number

If creditBalance reaches 0, stop scraping locally and show the "out of credits" state. If any rejected items come back, do the same — the user is out of credit mid-batch.

5. Logout / switch license key

POST /auth/logout
Authorization: Bearer <token>

204 — revokes the session server-side. Clear the stored token and return to the activation screen.

What the extension side still needs to implement (Phase 1 — build this first)

1. Activation screen (license key input)
2. Secure token storage in chrome.storage.local
3. Startup call to /auth/validate, with proper handling of each error case above
4. Credit balance display in the popup, kept in sync with server responses
5. Local scrape buffer with a hard cap (~25–30) forcing a sync before continuing
6. The /leads/extract-batch call wired into the buffer-flush logic
7. UI states for: normal, low/out of credit, session invalid/expired/revoked, sync failed (retry, don't drop the buffer)
8. Logout/switch-key option
9. manifest.json → host_permissions: https://scrapelyads.abdulawal.cloud/*
10. All backend calls made from the background service worker, not content scripts
11. CSV/export only ever built from leads the backend actually confirmed (accepted or previously-synced), never straight from the pre-sync buffer

Email outreach relay — live in production (built 2026-08-23)

Design: a Chrome extension (Manifest V3) cannot open a raw SMTP socket, so only the actual send is relayed through the backend. Everything else — interval timing, the lead loop, template-variable merging (e.g. {{business_name}}) — runs client-side in the extension's side panel.

• GET /account/smtp — saved mailbox config, flat shape: {"configured":false} or {"configured":true,"host":...,"port":...,"username":...,"fromEmail":...}. Never returns the password.
• PUT /account/smtp — save/replace the mailbox. Method is PUT, not POST. Body: {fromName, fromEmail, host, port, secure, username, password, replyTo?}. Runs nodemailer transporter.verify() before persisting — a bad password returns 400 smtp_verify_failed and saves nothing.
• DELETE /account/smtp — disconnect the mailbox.
• POST /account/smtp/test — sends a real test message to the customer's own fromEmail. Always answers HTTP 200; failure is {"ok":false,"error":"..."} in the body, not the status code.
• POST /account/send-email — the relay itself, rate-limited to 20/min per license. Body: {to, subject, html?, text?} (already merged — send exactly what arrives). 200 sent; 400 not_configured no mailbox saved; 401 session dead; 502 send_failed this recipient bounced (extension skips and continues).

SMTP passwords are AES-256-GCM encrypted at rest (src/lib/crypto.ts), keyed by SMTP_ENCRYPTION_KEY in .env — never stored in the extension. Gmail/Workspace and Zoho need an App Password; cPanel/Titan/domain mail use the normal mailbox password.

WhatsApp outreach relay — live in production (backend built 2026-09-03, extension tab built 2026-09-04, per-license account isolation built 2026-09-04)

Design: each license gets its own dedicated SafeWA account (not just a session under one shared account), auto-provisioned on the backend the first time the customer clicks Connect, via SafeWA's (a separate in-house WhatsApp automation platform — see the SafeWA project) narrow-scope POST /internal/provision-user endpoint. That endpoint is gated by SAFE_WA_PROVISION_TOKEN — a credential that can only create a new pre-approved user, deliberately not SafeWA's platform admin token (which can read/ban/delete every user on that platform). This means one customer's WhatsApp connection being suspended, rate-limited, or deleted on SafeWA's side never touches any other customer — no single shared account is a point of failure. The provisioned account's own SafeWA auth token is stored AES-256-GCM-encrypted on WhatsAppConnection.safeWaTokenEnc (same encryption helper as SMTP passwords). The extension never sees SafeWA's URL, the provisioning token, or any per-account token — only a QR image and a connected/not-connected status. A sent message costs 1 credit, same pool as lead extraction.

• GET /account/whatsapp — connection status. {"connected":false,"status":"not_connected"} when nothing is set up yet; otherwise {"connected":bool,"status":"qr_required"|"connecting"|"connected"|..., "qrDataUrl":"data:image/png;base64,..." (only when status is qr_required), "lastError":"..."}.
• POST /account/whatsapp/connect — creates the SafeWA session on first call, restarts it (fresh QR) on repeat calls. Returns the same status shape as GET / above. Poll GET /account/whatsapp every few seconds after this while status is qr_required, until it flips to connected (the QR image itself expires and needs refreshing).
• DELETE /account/whatsapp — disconnects and removes the connection (204, empty body). A future Connect click re-provisions a clean session.
• POST /account/whatsapp/send — the relay itself, rate-limited to 20/min per license. Body: {to, message} (to — any reasonable phone format, non-digits are stripped server-side; message — plain text, already merged). 200 {"ok":true,"status":"sent"|"queued"} — "queued" means SafeWA's warmup pacing is holding it, not an error, don't treat it as one; 400 not_connected no WhatsApp connection yet; 402 insufficient_credits; 403 safety_blocked this one message tripped SafeWA's anti-ban rules (skip this recipient, keep going — it's a per-message verdict, not a config problem); 502 send_failed genuine send failure.

Credits are reserved before the SafeWA call and refunded automatically on safety_blocked/send_failed, so a blocked or failed send never actually costs the customer a credit.