From 39af9a31118b8abe816237f8acfc8a8d39e1fa76 Mon Sep 17 00:00:00 2001 From: IgorVolochay Date: Sun, 25 Oct 2026 12:03:18 +0300 Subject: [PATCH] docs: redesign around VidIQ data and add the card fixture Co-Authored-By: Claude Sonnet 5.5 --- .../plans/2026-10-25-outlier-saver.md | 2 + .../plans/2026-10-25-vidiq-rework.md | 1596 +++++++++++++++++ .../specs/2026-10-25-outlier-saver-design.md | 150 +- test/fixtures/home-card.html | 1 + 4 files changed, 1669 insertions(+), 80 deletions(-) create mode 100644 docs/superpowers/plans/2026-10-25-vidiq-rework.md create mode 100644 test/fixtures/home-card.html diff --git a/docs/superpowers/plans/2026-10-25-outlier-saver.md b/docs/superpowers/plans/2026-10-25-outlier-saver.md index ba15df4..5816663 100644 --- a/docs/superpowers/plans/2026-10-25-outlier-saver.md +++ b/docs/superpowers/plans/2026-10-25-outlier-saver.md @@ -1,3 +1,5 @@ +> **Заменён:** этот план описывает версию 1 (YouTube Data API). Актуальный план — `2026-10-25-vidiq-rework.md`. + # Outlier Saver Implementation Plan > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. diff --git a/docs/superpowers/plans/2026-10-25-vidiq-rework.md b/docs/superpowers/plans/2026-10-25-vidiq-rework.md new file mode 100644 index 0000000..7a6f162 --- /dev/null +++ b/docs/superpowers/plans/2026-10-25-vidiq-rework.md @@ -0,0 +1,1596 @@ +# Outlier Saver: переход на данные VidIQ — план реализации + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Переделать готовое расширение с YouTube Data API на чтение DOM: outlier score и подписчиков брать из блоков VidIQ на карточках видео, просмотры, название, канал и превью — из разметки YouTube. + +**Architecture:** Content script разбирает карточки (`scrapeCard`) и шлёт в background только те, где VidIQ уже вывел score. Background проверяет сообщение, сравнивает с порогом и пишет в IndexedDB. Сети, ключей и очередей больше нет. Вся логика, кроме работы с DOM, — чистые функции под `node:test`; разбор карточки проверяется в headless Chrome на фикстурах. + +**Tech Stack:** Chrome Manifest V3, чистый JavaScript (ES-модули), IndexedDB, `node:test` (Node 22), headless Chrome только для одного теста (пропускается, если Chrome нет). Без npm-зависимостей и сборки. + +## Global Constraints + +Из `docs/superpowers/specs/2026-10-25-outlier-saver-design.md` (версия 2). Каждая задача неявно включает эти пункты. + +- Manifest V3, чистый JS с ES-модулями, **без сборки и без зависимостей**; расширение загружается как «распакованное». +- Тесты: только встроенный `node:test` (`npm test` → `node --test`). +- **Никакой сети.** В манифесте нет `host_permissions`; нет `fetch` к YouTube, VidIQ или API. Разрешения: `storage`. +- Сохраняем видео, если `outlierScore >= threshold` и `views >= minViews`. Уже сохранённое видео обновляется всегда. Настройки (`chrome.storage.local`, ключ `settings`): `threshold` по умолчанию 5, `minViews` по умолчанию 10 000. +- Привязка к VidIQ только через `[data-hover-popover="OutlierPopover"]` (+ параметр `outlier_score` в его ссылке), `[data-vidiq-subscriber-count]` и контейнер `.vidiq-scope`. Сгенерированные классы вида `vidiq-c-…` не использовать. +- Число подписчиков и просмотров — значения с экрана (округлённые); языки интерфейса: русский и английский. +- IndexedDB: база `outlier-saver`, версия 1, хранилище `outliers` (ключ `videoId`, индекс `updatedAt`). +- `url` записи всегда `https://www.youtube.com/watch?v=`, включая Shorts. `isShort` — `true`, если видео хоть раз встретилось по ссылке `/shorts/`. +- Сканирование страниц: главная `/`, просмотр `/watch`, поиск `/results`, канал `/@…`, `/channel/…`, `/c/…`, `/user/…`. Собираются только карточки (не само открытое видео). +- Тексты интерфейса — на русском. Недоверенный текст (названия, каналы) выводится через `textContent`, не через `innerHTML`. +- Сообщения коммитов заканчиваются трейлером `Co-Authored-By: Claude Sonnet 5.5 ` (вторым `-m`). +- Временные файлы тестов создаются через `os.tmpdir()`; при запуске из этой среды выставлять `TMPDIR` на scratchpad. + +--- + +## Структура файлов + +``` +manifest.json # изменить: без host_permissions, web_accessible_resources: src/lib/*.js +src/background.js # переписать: принять карточки -> buildRecord -> IndexedDB +src/content.js # переписать: сканер карточек +src/lib/numbers.js # новый: parseCount, parseViews +src/lib/vidiq.js # переписать: селекторы VidIQ, outlierScoreFromHref, readVidiqData +src/lib/record.js # новый: shouldSave, sanitizeItem, buildRecord +src/lib/card.js # новый: scrapeCard, CARD_SELECTOR, LINK_SELECTOR +src/lib/video-id.js # без изменений +src/lib/export.js # изменить: колонки +src/lib/storage.js # изменить: только threshold и minViews, без status +src/lib/db.js # изменить: только хранилище outliers +src/lib/api.js, process.js, score.js # удалить +src/popup/*, src/options/* # изменить: убрать ключ и статус +tools/browser.js # новый: запуск headless Chrome и статического сервера для теста +test/fixtures/home-card.html # уже лежит: настоящая карточка с главной с блоками VidIQ +test/fixtures/{harness,search-card,sidebar-card}.html # новые +test/{numbers,vidiq,record,card.dom,export,storage}.test.js +test/{api-helpers,api-requests,process,score}.test.js # удалить +README.md # обновить +``` + +Форма сообщения content → background (`{ type: 'cards', items: Item[] }`), `Item`: +`{ videoId, title: string|null, thumbnailUrl: string|null, channelTitle: string|null, channelUrl: string|null, subscribers: number|null, views: number, outlierScore: number, isShort: boolean, page: 'home'|'watch'|'search'|'channel' }`. + +--- + +### Task 1: Разбор чисел («59 млн», «571 тыс.», «1.2M») + +**Files:** +- Create: `src/lib/numbers.js` +- Test: `test/numbers.test.js` + +**Interfaces:** +- Consumes: ничего. +- Produces: + - `parseCount(text: unknown): number | null` — целое ≥ 0 или `null`. Принимает число с необязательной единицей и необязательным словом «просмотров / подписчиков / views / subscribers». Весь текст должен разобраться целиком. + - `parseViews(text: unknown): number | null` — то же, плюс «Нет просмотров» / «No views» → `0`. + +- [ ] **Step 1: Write the failing test** + +`test/numbers.test.js`: + +```js +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { parseCount, parseViews } from '../src/lib/numbers.js'; + +test('parseCount understands Russian abbreviations', () => { + assert.equal(parseCount('59 млн'), 59000000); + assert.equal(parseCount('59 млн'), 59000000); // non-breaking space, as YouTube renders it + assert.equal(parseCount('571 тыс.'), 571000); + assert.equal(parseCount('1,8 млн'), 1800000); + assert.equal(parseCount('3,4 млрд'), 3400000000); + assert.equal(parseCount('123 тысячи'), 123000); + assert.equal(parseCount('1,8 миллиона'), 1800000); + assert.equal(parseCount('146 миллионов'), 146000000); +}); + +test('parseCount understands English abbreviations', () => { + assert.equal(parseCount('1.2M'), 1200000); + assert.equal(parseCount('1.2m'), 1200000); + assert.equal(parseCount('123K'), 123000); + assert.equal(parseCount('1.5B'), 1500000000); + assert.equal(parseCount('2.5 million'), 2500000); +}); + +test('parseCount reads plain and grouped integers', () => { + assert.equal(parseCount('37'), 37); + assert.equal(parseCount('0'), 0); + assert.equal(parseCount('12 345'), 12345); + assert.equal(parseCount('12 345'), 12345); + assert.equal(parseCount('1,234,567'), 1234567); + assert.equal(parseCount('1.234.567'), 1234567); +}); + +test('parseCount ignores trailing "views" and "subscribers" words', () => { + assert.equal(parseCount('59 миллионов просмотров'), 59000000); + assert.equal(parseCount('1 просмотр'), 1); + assert.equal(parseCount('571 тыс. подписчиков'), 571000); + assert.equal(parseCount('1.2M views'), 1200000); + assert.equal(parseCount('1.2M subscribers'), 1200000); +}); + +test('parseCount rejects anything that is not wholly a count', () => { + for (const text of ['', 'Wun Wun', '10 л. назад', '2 г. назад', '7 дн. назад', '3 нед. назад', '1,2 тыс. зрителей', '5 Minute Crafts', 'abc', '1.2', '1,5']) { + assert.equal(parseCount(text), null, text); + } + for (const value of [null, undefined, 123, {}]) { + assert.equal(parseCount(value), null, String(value)); + } +}); + +test('parseViews adds the "no views" cases', () => { + assert.equal(parseViews('Нет просмотров'), 0); + assert.equal(parseViews('No views'), 0); + assert.equal(parseViews('59 млн'), 59000000); + assert.equal(parseViews('1,8 млн просмотров'), 1800000); + assert.equal(parseViews('1.2M views'), 1200000); +}); + +test('parseViews rejects dates and names', () => { + for (const text of ['10 л. назад', 'Wun Wun', '2 years ago', null, undefined]) { + assert.equal(parseViews(text), null, String(text)); + } +}); +``` + +- [ ] **Step 2: Run the test to verify it fails** + +Run: `node --test test/numbers.test.js` +Expected: FAIL with `ERR_MODULE_NOT_FOUND`. + +- [ ] **Step 3: Write the implementation** + +`src/lib/numbers.js`: + +```js +const UNITS = new Map([ + ['k', 1e3], ['тыс', 1e3], ['тысяча', 1e3], ['тысячи', 1e3], ['тысяч', 1e3], ['thousand', 1e3], + ['m', 1e6], ['млн', 1e6], ['миллион', 1e6], ['миллиона', 1e6], ['миллионов', 1e6], ['million', 1e6], + ['b', 1e9], ['млрд', 1e9], ['миллиард', 1e9], ['миллиарда', 1e9], ['миллиардов', 1e9], ['billion', 1e9], +]); + +// "просмотров", "подписчиков", "views", "subscribers" and their other forms at the end of the text. +const COUNT_WORDS = /\s*(просмотров|просмотра|просмотр|подписчиков|подписчика|подписчики|подписчик|views?|subscribers?)$/; +const GROUPED_INTEGER = /^\d{1,3}(?:[ ,.]\d{3})+$/; +const DECIMAL = /^\d+(?:[.,]\d+)?$/; + +// Reads "59 млн", "571 тыс.", "1.2M", "12 345" and the like; the whole text must be a count. +export function parseCount(text) { + if (typeof text !== 'string') return null; + const normalized = text.replace(/\s+/g, ' ').trim().toLowerCase().replace(COUNT_WORDS, '').trim(); + const match = /^(\d[\d ,.]*?)(?: ?([a-zа-яё]+)\.?)?$/.exec(normalized); + if (!match) return null; + const [, number, unit] = match; + + if (unit === undefined) { + if (/^\d+$/.test(number)) return Number(number); + return GROUPED_INTEGER.test(number) ? Number(number.replace(/\D/g, '')) : null; + } + const multiplier = UNITS.get(unit); + if (multiplier === undefined || !DECIMAL.test(number)) return null; + return Math.round(Number(number.replace(',', '.')) * multiplier); +} + +export function parseViews(text) { + if (typeof text !== 'string') return null; + if (/^\s*(no|нет) (views|просмотров)\s*$/i.test(text)) return 0; + return parseCount(text); +} +``` + +- [ ] **Step 4: Run the test to verify it passes** + +Run: `node --test test/numbers.test.js` +Expected: PASS, 7 tests. + +- [ ] **Step 5: Commit** + +```bash +git add src/lib/numbers.js test/numbers.test.js +git commit -m "feat: parse abbreviated counts like \"59 млн\" and \"571 тыс.\"" -m "Co-Authored-By: Claude Sonnet 5.5 " +``` + +--- + +### Task 2: Чтение данных VidIQ с карточки + +**Files:** +- Modify (rewrite): `src/lib/vidiq.js` +- Modify (rewrite): `test/vidiq.test.js` + +**Interfaces:** +- Consumes: `parseCount` from `./numbers.js`. +- Produces: + - `OUTLIER_SELECTOR = '[data-hover-popover="OutlierPopover"]'`, `SUBSCRIBERS_SELECTOR = '[data-vidiq-subscriber-count]'` + - `parseVidiqScore(text: string|null|undefined): number|null` — первое число в тексте (`>100x` → 100) + - `outlierScoreFromHref(href: string|null|undefined): number|null` — параметр `outlier_score`, округлённый до двух знаков + - `readVidiqData(card): { outlierScore: number|null, subscribers: number|null }`; `card` — любой объект с `querySelector(selector)`; найденные узлы используют `textContent`, `querySelector('a')` и `getAttribute('href')`. + +- [ ] **Step 1: Write the failing test** + +Replace `test/vidiq.test.js`: + +```js +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { + OUTLIER_SELECTOR, + SUBSCRIBERS_SELECTOR, + outlierScoreFromHref, + parseVidiqScore, + readVidiqData, +} from '../src/lib/vidiq.js'; + +const HREF = + 'https://app.vidiq.com/outliers?utm_source=chrome&utm_medium=extension&utm_content=&filters=%7B%22mode%22%3A%22long%22%7D' + + '&outlier_score=146.19205697700949&video_id=Tts9NHL8D8I'; + +test('selectors point at VidIQ data attributes, not generated classes', () => { + assert.equal(OUTLIER_SELECTOR, '[data-hover-popover="OutlierPopover"]'); + assert.equal(SUBSCRIBERS_SELECTOR, '[data-vidiq-subscriber-count]'); +}); + +test('parseVidiqScore takes the first number in the text', () => { + assert.equal(parseVidiqScore('>100x'), 100); + assert.equal(parseVidiqScore('12.5x'), 12.5); + assert.equal(parseVidiqScore('5,2x'), 5.2); + assert.equal(parseVidiqScore('Outlier 8'), 8); + for (const text of ['no digits', '', null, undefined]) assert.equal(parseVidiqScore(text), null, String(text)); +}); + +test('outlierScoreFromHref reads and rounds the outlier_score parameter', () => { + assert.equal(outlierScoreFromHref(HREF), 146.19); + assert.equal(outlierScoreFromHref('https://app.vidiq.com/outliers?outlier_score=2'), 2); +}); + +test('outlierScoreFromHref returns null without a usable parameter', () => { + for (const href of ['https://app.vidiq.com/outliers?video_id=x', 'https://app.vidiq.com/outliers?outlier_score=', 'https://app.vidiq.com/outliers?outlier_score=abc', '', null, undefined]) { + assert.equal(outlierScoreFromHref(href), null, String(href)); + } +}); + +// A card stub: querySelector answers only for the two VidIQ selectors. +const cardWith = ({ outlier, subscribers }) => ({ + querySelector: (selector) => (selector === OUTLIER_SELECTOR ? outlier ?? null : selector === SUBSCRIBERS_SELECTOR ? subscribers ?? null : null), +}); +const outlierNode = (text, href) => ({ + textContent: text, + querySelector: (selector) => (selector === 'a' && href !== undefined ? { getAttribute: (name) => (name === 'href' ? href : null) } : null), +}); + +test('readVidiqData reads the exact score from the link and subscribers from the data attribute', () => { + const card = cardWith({ outlier: outlierNode('>100x', HREF), subscribers: { textContent: '571 тыс.' } }); + assert.deepEqual(readVidiqData(card), { outlierScore: 146.19, subscribers: 571000 }); +}); + +test('readVidiqData falls back to the badge text when the link has no score', () => { + const card = cardWith({ outlier: outlierNode('>100x', 'https://app.vidiq.com/outliers'), subscribers: null }); + assert.deepEqual(readVidiqData(card), { outlierScore: 100, subscribers: null }); + const noLink = cardWith({ outlier: outlierNode('7.5x'), subscribers: null }); + assert.deepEqual(readVidiqData(noLink), { outlierScore: 7.5, subscribers: null }); +}); + +test('readVidiqData returns nulls when VidIQ has not rendered its blocks', () => { + assert.deepEqual(readVidiqData(cardWith({})), { outlierScore: null, subscribers: null }); +}); +``` + +- [ ] **Step 2: Run the test to verify it fails** + +Run: `node --test test/vidiq.test.js` +Expected: FAIL (`does not provide an export named 'OUTLIER_SELECTOR'`). + +- [ ] **Step 3: Write the implementation** + +Replace `src/lib/vidiq.js`: + +```js +import { parseCount } from './numbers.js'; + +// VidIQ's own class names are generated (vidiq-c-xxxx) and change between builds; these hooks are +// data attributes it sets deliberately. +export const OUTLIER_SELECTOR = '[data-hover-popover="OutlierPopover"]'; +export const SUBSCRIBERS_SELECTOR = '[data-vidiq-subscriber-count]'; + +export function parseVidiqScore(text) { + const match = /(\d+(?:[.,]\d+)?)/.exec(text ?? ''); + return match ? Number(match[1].replace(',', '.')) : null; +} + +// The badge link carries the exact value (e.g. 146.19) while the badge text is capped (">100x"). +export function outlierScoreFromHref(href) { + if (!href) return null; + let raw; + try { + raw = new URL(href, 'https://app.vidiq.com').searchParams.get('outlier_score'); + } catch { + return null; + } + if (raw === null || raw === '') return null; + const value = Number(raw); + return Number.isFinite(value) ? Math.round(value * 100) / 100 : null; +} + +export function readVidiqData(card) { + const outlier = card.querySelector(OUTLIER_SELECTOR); + const href = outlier?.querySelector('a')?.getAttribute('href'); + const outlierScore = outlier ? (outlierScoreFromHref(href) ?? parseVidiqScore(outlier.textContent)) : null; + const subscribersNode = card.querySelector(SUBSCRIBERS_SELECTOR); + return { outlierScore, subscribers: subscribersNode ? parseCount(subscribersNode.textContent) : null }; +} +``` + +- [ ] **Step 4: Run the test to verify it passes** + +Run: `node --test test/vidiq.test.js` +Expected: PASS, 7 tests. + +- [ ] **Step 5: Commit** + +```bash +git add src/lib/vidiq.js test/vidiq.test.js +git commit -m "feat: read outlier score and subscribers from VidIQ blocks" -m "Co-Authored-By: Claude Sonnet 5.5 " +``` + +--- + +### Task 3: Сохранение записи (порог, проверка входа, сборка записи) + +**Files:** +- Create: `src/lib/record.js` +- Test: `test/record.test.js` + +**Interfaces:** +- Consumes: `isVideoId` from `./video-id.js`. +- Produces: + - `shouldSave({ views, outlierScore }, { threshold, minViews }): boolean` + - `sanitizeItem(raw: unknown): Item | null` — `Item` как в разделе «Структура файлов»; возвращает `null`, если нет корректных `videoId`, `outlierScore` (число ≥ 0) или `views` (число ≥ 0); невалидные необязательные поля становятся `null` + - `buildRecord(item: Item, previous: Outlier | undefined, settings: { threshold, minViews }, now?: Date): Outlier | null` — запись с 13 полями из спецификации; `null`, если видео новое и не проходит порог + +- [ ] **Step 1: Write the failing test** + +`test/record.test.js`: + +```js +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { buildRecord, sanitizeItem, shouldSave } from '../src/lib/record.js'; + +const settings = { threshold: 5, minViews: 10000 }; +const NOW = new Date('2026-10-25T10:00:00.000Z'); +const NOW_ISO = NOW.toISOString(); + +test('shouldSave accepts a video exactly at both limits', () => { + assert.equal(shouldSave({ views: 10000, outlierScore: 5 }, settings), true); +}); + +test('shouldSave rejects a low score, low views or a missing score', () => { + assert.equal(shouldSave({ views: 50000, outlierScore: 4.99 }, settings), false); + assert.equal(shouldSave({ views: 9999, outlierScore: 150 }, settings), false); + assert.equal(shouldSave({ views: 50000, outlierScore: null }, settings), false); +}); + +const raw = (over = {}) => ({ + videoId: 'Tts9NHL8D8I', + title: 'Agar.io - 88k Score', + thumbnailUrl: 'https://i.ytimg.com/vi/Tts9NHL8D8I/hq720.jpg', + channelTitle: 'Wun Wun', + channelUrl: 'https://www.youtube.com/@wunwun9522', + subscribers: 571000, + views: 59000000, + outlierScore: 146.19, + isShort: false, + page: 'home', + ...over, +}); + +test('sanitizeItem passes a valid item through unchanged', () => { + assert.deepEqual(sanitizeItem(raw()), raw()); +}); + +test('sanitizeItem rejects items without a usable id, score or views', () => { + for (const bad of [null, undefined, 'x', {}, raw({ videoId: 'short' }), raw({ outlierScore: NaN }), raw({ outlierScore: null }), raw({ outlierScore: -1 }), raw({ views: null }), raw({ views: -5 }), raw({ views: '5' })]) { + assert.equal(sanitizeItem(bad), null, JSON.stringify(bad)); + } +}); + +test('sanitizeItem nulls out unusable optional fields', () => { + const item = sanitizeItem( + raw({ title: ' ', thumbnailUrl: 'javascript:alert(1)', channelTitle: 5, channelUrl: 'https://evil.example.com/@x', subscribers: -3, isShort: 'yes', page: 'nowhere' }), + ); + assert.deepEqual(item, raw({ title: null, thumbnailUrl: null, channelTitle: null, channelUrl: null, subscribers: null, isShort: false, page: 'home' })); +}); + +test('sanitizeItem trims and caps the title and rounds the score', () => { + const item = sanitizeItem(raw({ title: ` ${'x'.repeat(400)} `, outlierScore: 12.3456 })); + assert.equal(item.title.length, 300); + assert.equal(item.outlierScore, 12.35); +}); + +test('buildRecord creates a full record for a new outlier', () => { + assert.deepEqual(buildRecord(sanitizeItem(raw()), undefined, settings, NOW), { + videoId: 'Tts9NHL8D8I', + title: 'Agar.io - 88k Score', + url: 'https://www.youtube.com/watch?v=Tts9NHL8D8I', + thumbnailUrl: 'https://i.ytimg.com/vi/Tts9NHL8D8I/hq720.jpg', + channelTitle: 'Wun Wun', + channelUrl: 'https://www.youtube.com/@wunwun9522', + subscribers: 571000, + views: 59000000, + outlierScore: 146.19, + isShort: false, + foundOn: 'home', + firstSeenAt: NOW_ISO, + updatedAt: NOW_ISO, + }); +}); + +test('buildRecord skips a new video below the threshold', () => { + assert.equal(buildRecord(sanitizeItem(raw({ outlierScore: 2 })), undefined, settings, NOW), null); + assert.equal(buildRecord(sanitizeItem(raw({ views: 500 })), undefined, settings, NOW), null); +}); + +test('buildRecord keeps updating an already saved video even below the threshold', () => { + const previous = buildRecord(sanitizeItem(raw()), undefined, settings, new Date('2026-10-20T00:00:00.000Z')); + const updated = buildRecord(sanitizeItem(raw({ outlierScore: 1.5, views: 60000000, page: 'search' })), previous, settings, NOW); + assert.equal(updated.outlierScore, 1.5); + assert.equal(updated.views, 60000000); + assert.equal(updated.updatedAt, NOW_ISO); + assert.equal(updated.firstSeenAt, '2026-10-20T00:00:00.000Z'); + assert.equal(updated.foundOn, 'home'); +}); + +test('buildRecord keeps known values when the new sighting lacks them', () => { + const previous = buildRecord(sanitizeItem(raw({ isShort: true })), undefined, settings, new Date('2026-10-20T00:00:00.000Z')); + const updated = buildRecord( + sanitizeItem(raw({ subscribers: null, channelTitle: null, channelUrl: null, thumbnailUrl: null, title: null, isShort: false })), + previous, + settings, + NOW, + ); + assert.equal(updated.subscribers, 571000); + assert.equal(updated.channelTitle, 'Wun Wun'); + assert.equal(updated.channelUrl, 'https://www.youtube.com/@wunwun9522'); + assert.equal(updated.thumbnailUrl, 'https://i.ytimg.com/vi/Tts9NHL8D8I/hq720.jpg'); + assert.equal(updated.title, 'Agar.io - 88k Score'); + assert.equal(updated.isShort, true); +}); + +test('buildRecord falls back to the id when a new video has no title', () => { + const record = buildRecord(sanitizeItem(raw({ title: null })), undefined, settings, NOW); + assert.equal(record.title, 'Tts9NHL8D8I'); +}); +``` + +- [ ] **Step 2: Run the test to verify it fails** + +Run: `node --test test/record.test.js` +Expected: FAIL with `ERR_MODULE_NOT_FOUND`. + +- [ ] **Step 3: Write the implementation** + +`src/lib/record.js`: + +```js +import { isVideoId } from './video-id.js'; + +const PAGES = new Set(['home', 'watch', 'search', 'channel']); +const MAX_TEXT = 300; + +export function shouldSave({ views, outlierScore }, { threshold, minViews }) { + return typeof outlierScore === 'number' && outlierScore >= threshold && views >= minViews; +} + +const finiteOrNull = (value) => (typeof value === 'number' && Number.isFinite(value) ? value : null); +const textOrNull = (value) => (typeof value === 'string' && value.trim() ? value.trim().slice(0, MAX_TEXT) : null); +const urlOrNull = (value, prefix) => (typeof value === 'string' && value.startsWith(prefix) ? value : null); + +// Validates a message coming from the content script before anything is stored. +export function sanitizeItem(raw) { + if (!raw || !isVideoId(raw.videoId)) return null; + const outlierScore = finiteOrNull(raw.outlierScore); + const views = finiteOrNull(raw.views); + if (outlierScore === null || outlierScore < 0 || views === null || views < 0) return null; + const subscribers = finiteOrNull(raw.subscribers); + return { + videoId: raw.videoId, + title: textOrNull(raw.title), + thumbnailUrl: urlOrNull(raw.thumbnailUrl, 'https://i.ytimg.com/'), + channelTitle: textOrNull(raw.channelTitle), + channelUrl: urlOrNull(raw.channelUrl, 'https://www.youtube.com/'), + subscribers: subscribers !== null && subscribers >= 0 ? subscribers : null, + views, + outlierScore: Math.round(outlierScore * 100) / 100, + isShort: raw.isShort === true, + page: PAGES.has(raw.page) ? raw.page : 'home', + }; +} + +export function buildRecord(item, previous, settings, now = new Date()) { + if (!previous && !shouldSave(item, settings)) return null; + const nowIso = now.toISOString(); + return { + videoId: item.videoId, + title: item.title ?? previous?.title ?? item.videoId, + url: `https://www.youtube.com/watch?v=${item.videoId}`, + thumbnailUrl: item.thumbnailUrl ?? previous?.thumbnailUrl ?? null, + channelTitle: item.channelTitle ?? previous?.channelTitle ?? null, + channelUrl: item.channelUrl ?? previous?.channelUrl ?? null, + subscribers: item.subscribers ?? previous?.subscribers ?? null, + views: item.views, + outlierScore: item.outlierScore, + isShort: item.isShort || previous?.isShort || false, + foundOn: previous?.foundOn ?? item.page, + firstSeenAt: previous?.firstSeenAt ?? nowIso, + updatedAt: nowIso, + }; +} +``` + +- [ ] **Step 4: Run the test to verify it passes** + +Run: `node --test test/record.test.js` +Expected: PASS, 10 tests. + +- [ ] **Step 5: Commit** + +```bash +git add src/lib/record.js test/record.test.js +git commit -m "feat: add record building with threshold and input validation" -m "Co-Authored-By: Claude Sonnet 5.5 " +``` + +--- + +### Task 4: Разбор карточки и тест в headless Chrome + +**Files:** +- Create: `src/lib/card.js`, `tools/browser.js` +- Create: `test/fixtures/harness.html`, `test/fixtures/search-card.html`, `test/fixtures/sidebar-card.html` +- Existing: `test/fixtures/home-card.html` (настоящий образец от пользователя) +- Test: `test/card.dom.test.js` + +**Interfaces:** +- Consumes: `parseVideoLink` (video-id.js), `readVidiqData` (vidiq.js), `parseViews` (numbers.js). +- Produces: + - `LINK_SELECTOR`, `CARD_SELECTOR` (строки CSS-селекторов) + - `scrapeCard(card: Element, link: Element, context?: { channelTitle?: string|null, channelUrl?: string|null } | null): { videoId, isShort, title, thumbnailUrl, channelTitle, channelUrl, subscribers, views, outlierScore } | null` — `null`, если у ссылки нет корректного ID; поля, которые не удалось прочитать, равны `null` + - `tools/browser.js`: `CHROME_PATH: string|null`, `startStaticServer(root): Promise<{ origin, close }>`, `launchBrowser(): Promise<{ openPage(url): Promise<{ evaluate(expr), close() }>, close() }>` + +- [ ] **Step 1: Create the helper and the fixtures** + +`tools/browser.js`: + +```js +import { spawn, spawnSync } from 'node:child_process'; +import { createServer } from 'node:http'; +import { existsSync, mkdtempSync, readFileSync, rmSync, statSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { extname, join, resolve, sep } from 'node:path'; + +const sleep = (ms) => new Promise((done) => setTimeout(done, ms)); + +export const CHROME_PATH = + ['google-chrome', 'google-chrome-stable', 'chromium', 'chromium-browser'].find( + (name) => spawnSync('which', [name], { stdio: 'ignore' }).status === 0, + ) ?? null; + +const TYPES = { '.html': 'text/html; charset=utf-8', '.js': 'text/javascript; charset=utf-8', '.css': 'text/css' }; + +// Serves a directory over http://127.0.0.1 so a page can import ES modules from it. +export function startStaticServer(root) { + const base = resolve(root); + const server = createServer((req, res) => { + const path = resolve(join(base, decodeURIComponent(new URL(req.url, 'http://localhost').pathname))); + let isFile = false; + try { + isFile = path.startsWith(base + sep) && statSync(path).isFile(); + } catch { + // not found + } + if (!isFile) { + res.writeHead(404).end('not found'); + return; + } + res.writeHead(200, { 'content-type': TYPES[extname(path)] ?? 'application/octet-stream' }); + res.end(readFileSync(path)); + }); + return new Promise((done) => { + server.listen(0, '127.0.0.1', () => { + done({ origin: `http://127.0.0.1:${server.address().port}`, close: () => new Promise((closed) => server.close(closed)) }); + }); + }); +} + +export async function launchBrowser() { + const profile = mkdtempSync(join(tmpdir(), 'outlier-saver-chrome-')); + const proc = spawn( + CHROME_PATH, + ['--headless=new', '--remote-debugging-port=0', `--user-data-dir=${profile}`, '--no-first-run', '--no-default-browser-check', 'about:blank'], + { stdio: 'ignore' }, + ); + const portFile = join(profile, 'DevToolsActivePort'); + let port = null; + for (let attempt = 0; attempt < 80 && port === null; attempt += 1) { + if (existsSync(portFile)) port = Number(readFileSync(portFile, 'utf8').split('\n')[0]); + else await sleep(250); + } + if (port === null) { + proc.kill(); + throw new Error('Chrome did not start'); + } + + return { + async openPage(url) { + const target = await (await fetch(`http://127.0.0.1:${port}/json/new?${url}`, { method: 'PUT' })).json(); + const ws = new WebSocket(target.webSocketDebuggerUrl); + await new Promise((opened, failed) => { + ws.onopen = opened; + ws.onerror = failed; + }); + let nextId = 0; + const pending = new Map(); + ws.onmessage = ({ data }) => { + const message = JSON.parse(data); + const waiter = pending.get(message.id); + if (!waiter) return; + pending.delete(message.id); + if (message.error) waiter.fail(new Error(message.error.message)); + else waiter.done(message.result); + }; + const send = (method, params = {}) => + new Promise((done, fail) => { + nextId += 1; + pending.set(nextId, { done, fail }); + ws.send(JSON.stringify({ id: nextId, method, params })); + }); + const evaluate = async (expression) => { + const result = await send('Runtime.evaluate', { expression, awaitPromise: true, returnByValue: true }); + if (result.exceptionDetails) { + throw new Error(result.exceptionDetails.exception?.description ?? result.exceptionDetails.text); + } + return result.result.value; + }; + for (let attempt = 0; attempt < 80; attempt += 1) { + if ((await evaluate('document.readyState')) === 'complete') break; + await sleep(100); + } + return { evaluate, close: () => ws.close() }; + }, + async close() { + proc.kill('SIGTERM'); + await sleep(300); + rmSync(profile, { recursive: true, force: true, maxRetries: 3 }); + }, + }; +} +``` + +`test/fixtures/harness.html`: + +```html + + +harness + +``` + +`test/fixtures/search-card.html` — упрощённая карточка поиска (`ytd-video-renderer`), собранная по структуре, которую я наблюдал на живом YouTube 2026-10-25; блоки VidIQ взяты по образцу из `home-card.html`: + +```html + + +
+ + + +
1:29
+ + +
+
+ + +
+
+ +
+
+ 5,1 млн + 2 г. назад +
+ 111 тыс.подписчики +
+
+
+ +
29 VPH
+
+ 7.5x +
+ + +
+
+
+ +``` + +`test/fixtures/sidebar-card.html` — карточка боковой панели (новый макет, название канала текстом без ссылки, две строки метаданных): + +```html + + +
+ +
+ +
+
+

+ CБОРНИК - В Гостях У Бабули 21-30 серии +

+
+ +
+ Уродская Анимация +
+
+ 1,8 млн + + 3 нед. назад +
+
+ 2,82 млнподписчики +
+ +
+ 12x +
+ + +
+
+
+
+ +``` + +- [ ] **Step 2: Write the failing test** + +`test/card.dom.test.js`: + +```js +import { after, before, test } from 'node:test'; +import assert from 'node:assert/strict'; +import { fileURLToPath } from 'node:url'; +import { CHROME_PATH, launchBrowser, startStaticServer } from '../tools/browser.js'; + +const skip = CHROME_PATH ? false : 'Chrome is not installed'; +let server; +let browser; +let page; + +before(async () => { + if (skip) return; + server = await startStaticServer(fileURLToPath(new URL('..', import.meta.url))); + browser = await launchBrowser(); + page = await browser.openPage(`${server.origin}/test/fixtures/harness.html`); +}); + +after(async () => { + page?.close(); + await browser?.close(); + await server?.close(); +}); + +// Puts a fixture into the page (wrapped in `wrapper`, as YouTube nests it), runs `prepare` +// to tweak the DOM, then scrapes the first video link's card. +const scrape = (fixture, { wrapper = 'ytd-rich-item-renderer', prepare = '', context = null } = {}) => + page.evaluate(`(async () => { + const html = await (await fetch('/test/fixtures/${fixture}')).text(); + document.body.innerHTML = '<${wrapper}>' + html + ''; + ${prepare} + const { scrapeCard, CARD_SELECTOR, LINK_SELECTOR } = await import('/src/lib/card.js'); + const link = document.querySelector(LINK_SELECTOR); + return scrapeCard(link.closest(CARD_SELECTOR), link, ${JSON.stringify(context)}); + })()`); + +test('reads every field from a real home-page card with VidIQ blocks', { skip }, async () => { + const item = await scrape('home-card.html'); + assert.ok(item.thumbnailUrl.startsWith('https://i.ytimg.com/vi/Tts9NHL8D8I/'), item.thumbnailUrl); + assert.deepEqual({ ...item, thumbnailUrl: null }, { + videoId: 'Tts9NHL8D8I', + isShort: false, + title: 'Agar.io - 88k Score on an empty Experimental Server', + thumbnailUrl: null, + channelTitle: 'Wun Wun', + channelUrl: 'https://www.youtube.com/@wunwun9522', + subscribers: 571000, + views: 59000000, + outlierScore: 146.19, + }); +}); + +test('home card without VidIQ blocks still has views but no score or subscribers', { skip }, async () => { + const item = await scrape('home-card.html', { prepare: "document.querySelectorAll('.vidiq-scope').forEach((el) => el.remove());" }); + assert.equal(item.outlierScore, null); + assert.equal(item.subscribers, null); + assert.equal(item.views, 59000000); +}); + +test('badge without a link falls back to the capped badge text', { skip }, async () => { + const item = await scrape('home-card.html', { + prepare: "document.querySelector('[data-hover-popover=\"OutlierPopover\"] a').removeAttribute('href');", + }); + assert.equal(item.outlierScore, 100); +}); + +test('a /shorts/ link marks the item as a short', { skip }, async () => { + const item = await scrape('home-card.html', { + prepare: "document.querySelector('a[href*=\"/watch?v=\"]').setAttribute('href', '/shorts/abcdefghijk');", + }); + assert.equal(item.videoId, 'abcdefghijk'); + assert.equal(item.isShort, true); +}); + +test('search card: old renderer, subscribers inside the metadata line are not mistaken for views', { skip }, async () => { + const item = await scrape('search-card.html', { wrapper: 'div' }); + assert.deepEqual(item, { + videoId: 'pyV68KvUwlw', + isShort: false, + title: 'Minecraft Beaten In 53 Seconds [FWR]', + thumbnailUrl: 'https://i.ytimg.com/vi/pyV68KvUwlw/hq720.jpg?sqp=abc&rs=def', + channelTitle: 'HanabiYaki', + channelUrl: 'https://www.youtube.com/@HanabiYaki', + subscribers: 111000, + views: 5100000, + outlierScore: 7.46, + }); +}); + +test('sidebar card: channel name without a link, views taken from the metadata row', { skip }, async () => { + const item = await scrape('sidebar-card.html', { wrapper: 'div' }); + assert.equal(item.title, 'CБОРНИК - В Гостях У Бабули 21-30 серии'); + assert.equal(item.channelTitle, 'Уродская Анимация'); + assert.equal(item.channelUrl, null); + assert.equal(item.views, 1800000); + assert.equal(item.subscribers, 2820000); + assert.equal(item.outlierScore, 12); +}); + +test('channel page: the channel comes from the page context when the card names none', { skip }, async () => { + const item = await scrape('sidebar-card.html', { + wrapper: 'div', + prepare: "document.querySelector('.ytContentMetadataViewModelMetadataRow').remove();", + context: { channelTitle: 'MrBeast', channelUrl: 'https://www.youtube.com/@MrBeast' }, + }); + assert.equal(item.channelTitle, 'MrBeast'); + assert.equal(item.channelUrl, 'https://www.youtube.com/@MrBeast'); + assert.equal(item.views, 1800000); +}); + +test('thumbnail falls back to a URL built from the video id', { skip }, async () => { + const item = await scrape('sidebar-card.html', { wrapper: 'div', prepare: "document.querySelector('img').remove();" }); + assert.equal(item.thumbnailUrl, 'https://i.ytimg.com/vi/7KXuP7Ws8W8/mqdefault.jpg'); +}); +``` + +- [ ] **Step 3: Run the test to verify it fails** + +Run: `TMPDIR= node --test test/card.dom.test.js` +Expected: tests FAIL (the page cannot import `/src/lib/card.js`: the file does not exist yet). + +- [ ] **Step 4: Write the implementation** + +`src/lib/card.js`: + +```js +import { parseViews } from './numbers.js'; +import { parseVideoLink } from './video-id.js'; +import { readVidiqData } from './vidiq.js'; + +export const LINK_SELECTOR = 'a[href*="/watch?v="], a[href*="/shorts/"]'; +export const CARD_SELECTOR = [ + 'ytd-rich-item-renderer', + 'ytd-video-renderer', + 'ytd-compact-video-renderer', + 'ytd-grid-video-renderer', + 'yt-lockup-view-model', + 'ytm-shorts-lockup-view-model', + 'ytd-reel-item-renderer', +].join(','); + +// Spans that hold the channel name, the view count and the age in the new and old card layouts. +const METADATA_TEXT = '.ytContentMetadataViewModelMetadataText, .inline-metadata-item, #metadata-line span'; +const CHANNEL_LINK = 'a[href^="/@"], a[href^="/channel/"], a[href^="/c/"], a[href^="/user/"]'; +const RELATIVE_TIME = /назад|(^|\s)ago(\s|$)/i; + +// VidIQ puts its own text (subscribers, VPH, score) inside `.vidiq-scope`; it must not be read as YouTube's. +const outsideVidiq = (element) => !element.closest('.vidiq-scope'); + +function readTitle(card) { + const heading = card.querySelector('h3[title]')?.getAttribute('title')?.trim(); + if (heading) return heading; + const oldTitle = card.querySelector('#video-title'); + const oldText = (oldTitle?.getAttribute('title') || oldTitle?.textContent || '').trim(); + if (oldText) return oldText; + return card.querySelector('.ytLockupMetadataViewModelTitle')?.textContent.trim() || null; +} + +function readThumbnail(card, videoId) { + const image = card.querySelector('img[src^="https://i.ytimg.com/vi/"], img[src^="https://i.ytimg.com/vi_webp/"]'); + return image ? image.src : `https://i.ytimg.com/vi/${videoId}/mqdefault.jpg`; +} + +function readViews(card) { + for (const element of card.querySelectorAll(METADATA_TEXT)) { + if (!outsideVidiq(element)) continue; + const views = parseViews(element.textContent) ?? parseViews(element.getAttribute('aria-label')); + if (views !== null) return views; + } + return null; +} + +function absoluteChannelUrl(href) { + try { + const url = new URL(href, 'https://www.youtube.com'); + return url.origin + url.pathname; + } catch { + return null; + } +} + +function readChannel(card, context) { + const link = [...card.querySelectorAll(CHANNEL_LINK)].find((a) => outsideVidiq(a) && a.textContent.trim()); + if (link) return { channelTitle: link.textContent.trim(), channelUrl: absoluteChannelUrl(link.getAttribute('href')) }; + + // New layouts show the channel name as plain text: the first metadata text that is neither a count nor an age. + for (const element of card.querySelectorAll(METADATA_TEXT)) { + const text = element.textContent.trim(); + if (outsideVidiq(element) && text && parseViews(text) === null && !RELATIVE_TIME.test(text)) { + return { channelTitle: text, channelUrl: context?.channelUrl ?? null }; + } + } + return { channelTitle: context?.channelTitle ?? null, channelUrl: context?.channelUrl ?? null }; +} + +// Reads one video card. `context` supplies the channel on channel pages, where cards do not name it. +export function scrapeCard(card, link, context = null) { + const parsed = parseVideoLink(link.getAttribute('href')); + if (!parsed) return null; + const { outlierScore, subscribers } = readVidiqData(card); + const { channelTitle, channelUrl } = readChannel(card, context); + return { + videoId: parsed.videoId, + isShort: parsed.isShort, + title: readTitle(card), + thumbnailUrl: readThumbnail(card, parsed.videoId), + channelTitle, + channelUrl, + subscribers, + views: readViews(card), + outlierScore, + }; +} +``` + +- [ ] **Step 5: Run the test to verify it passes** + +Run: `TMPDIR= node --test test/card.dom.test.js` +Expected: PASS, 8 tests (or all skipped with the reason "Chrome is not installed"). + +If a case fails, fix `card.js` (not the fixture) unless the fixture contradicts the real markup in `home-card.html`. + +- [ ] **Step 6: Commit** + +```bash +git add src/lib/card.js tools/browser.js test/card.dom.test.js test/fixtures +git commit -m "feat: scrape video cards and verify against real markup in headless Chrome" -m "Co-Authored-By: Claude Sonnet 5.5 " +``` + +--- + +### Task 5: Экспорт, настройки и база под новую модель + +**Files:** +- Modify: `src/lib/export.js` (колонки), `src/lib/storage.js` (только `threshold`, `minViews`; без `status`), `src/lib/db.js` (только `outliers`) +- Modify: `test/export.test.js`, `test/storage.test.js` + +**Interfaces:** +- `COLUMNS` = `videoId, title, url, thumbnailUrl, channelTitle, channelUrl, subscribers, views, outlierScore, isShort, foundOn, firstSeenAt, updatedAt` +- `DEFAULT_SETTINGS = { threshold: 5, minViews: 10000 }`; `normalizeSettings(raw?)`, `getSettings()`, `saveSettings(raw)` — без `apiKey`; `getStatus` и `setStatus` удалены +- `db.js` экспортирует `openDb`, `getOutliers(ids): Promise`, `putOutliers(records)`, `listOutliers(limit?)`, `countOutliers()`; функции для `seen` и `channels` удалены + +- [ ] **Step 1: Update the tests so they fail** + +`test/export.test.js` — заменить файл: + +```js +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { COLUMNS, toCsv, toJson } from '../src/lib/export.js'; + +const record = (over = {}) => ({ + videoId: 'dQw4w9WgXcQ', + title: 'Hello', + url: 'https://www.youtube.com/watch?v=dQw4w9WgXcQ', + thumbnailUrl: 'https://i.ytimg.com/vi/dQw4w9WgXcQ/mqdefault.jpg', + channelTitle: 'Chan', + channelUrl: 'https://www.youtube.com/@chan', + subscribers: 1000, + views: 50000, + outlierScore: 50, + isShort: false, + foundOn: 'home', + firstSeenAt: '2026-10-25T10:00:00.000Z', + updatedAt: '2026-10-25T10:00:00.000Z', + ...over, +}); + +test('COLUMNS lists the stored fields in order', () => { + assert.deepEqual(COLUMNS, [ + 'videoId', 'title', 'url', 'thumbnailUrl', 'channelTitle', 'channelUrl', 'subscribers', + 'views', 'outlierScore', 'isShort', 'foundOn', 'firstSeenAt', 'updatedAt', + ]); +}); + +test('toCsv writes a header and one row per record', () => { + assert.equal( + toCsv([record()]), + COLUMNS.join(',') + + '\r\n' + + 'dQw4w9WgXcQ,Hello,https://www.youtube.com/watch?v=dQw4w9WgXcQ,' + + 'https://i.ytimg.com/vi/dQw4w9WgXcQ/mqdefault.jpg,Chan,https://www.youtube.com/@chan,' + + '1000,50000,50,false,home,2026-10-25T10:00:00.000Z,2026-10-25T10:00:00.000Z\r\n', + ); +}); + +test('toCsv quotes commas, quotes and newlines', () => { + const csv = toCsv([record({ title: 'He said "hi", then\nleft' })]); + assert.ok(csv.includes('"He said ""hi"", then\nleft"')); +}); + +test('toCsv writes null and undefined as empty cells and keeps booleans readable', () => { + const csv = toCsv([record({ subscribers: null, isShort: true, channelUrl: undefined })]); + const row = csv.split('\r\n')[1]; + assert.ok(row.includes(',true,')); + assert.ok(row.includes('Chan,,,50000')); // channelUrl and subscribers empty +}); + +test('toCsv of no records is just the header', () => { + assert.equal(toCsv([]), COLUMNS.join(',') + '\r\n'); +}); + +test('toJson pretty-prints and round-trips', () => { + const records = [record(), record({ videoId: 'abcdefghijk' })]; + const text = toJson(records); + assert.ok(text.includes('\n ')); + assert.deepEqual(JSON.parse(text), records); +}); +``` + +`test/storage.test.js` — заменить файл: + +```js +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { DEFAULT_SETTINGS, normalizeSettings } from '../src/lib/storage.js'; + +test('normalizeSettings falls back to defaults for missing input', () => { + assert.deepEqual(DEFAULT_SETTINGS, { threshold: 5, minViews: 10000 }); + assert.deepEqual(normalizeSettings(), DEFAULT_SETTINGS); + assert.deepEqual(normalizeSettings({}), DEFAULT_SETTINGS); +}); + +test('normalizeSettings coerces numeric strings', () => { + assert.deepEqual(normalizeSettings({ threshold: '7.5', minViews: '20000' }), { threshold: 7.5, minViews: 20000 }); +}); + +test('normalizeSettings replaces invalid thresholds with the default', () => { + for (const threshold of [-1, 0, 'abc', '', null, NaN, Infinity]) { + assert.equal(normalizeSettings({ threshold }).threshold, 5, String(threshold)); + } +}); + +test('normalizeSettings replaces invalid minViews with the default and floors decimals', () => { + for (const minViews of [-5, 'abc', '', null, NaN]) { + assert.equal(normalizeSettings({ minViews }).minViews, 10000, String(minViews)); + } + assert.equal(normalizeSettings({ minViews: 1500.7 }).minViews, 1500); + assert.equal(normalizeSettings({ minViews: 0 }).minViews, 0); +}); + +test('normalizeSettings drops fields it does not know (the old apiKey)', () => { + assert.deepEqual(normalizeSettings({ apiKey: 'secret', threshold: 6 }), { threshold: 6, minViews: 10000 }); +}); +``` + +- [ ] **Step 2: Run the tests to verify they fail** + +Run: `node --test test/export.test.js test/storage.test.js` +Expected: FAIL (old columns, `apiKey` still returned). + +- [ ] **Step 3: Update the implementation** + +`src/lib/export.js` — заменить массив `COLUMNS`, остальное не менять: + +```js +export const COLUMNS = [ + 'videoId', + 'title', + 'url', + 'thumbnailUrl', + 'channelTitle', + 'channelUrl', + 'subscribers', + 'views', + 'outlierScore', + 'isShort', + 'foundOn', + 'firstSeenAt', + 'updatedAt', +]; +``` + +`src/lib/storage.js` — заменить файл: + +```js +export const DEFAULT_SETTINGS = { threshold: 5, minViews: 10000 }; + +const toNumber = (value) => (value === null || value === undefined || value === '' ? NaN : Number(value)); + +export function normalizeSettings(raw = {}) { + const threshold = toNumber(raw.threshold); + const minViews = toNumber(raw.minViews); + return { + threshold: Number.isFinite(threshold) && threshold > 0 ? threshold : DEFAULT_SETTINGS.threshold, + minViews: Number.isFinite(minViews) && minViews >= 0 ? Math.floor(minViews) : DEFAULT_SETTINGS.minViews, + }; +} + +export async function getSettings() { + const { settings } = await chrome.storage.local.get('settings'); + return normalizeSettings(settings); +} + +export async function saveSettings(raw) { + const settings = normalizeSettings(raw); + await chrome.storage.local.set({ settings }); + return settings; +} +``` + +`src/lib/db.js` — заменить файл: + +```js +const DB_NAME = 'outlier-saver'; +const DB_VERSION = 1; + +let dbPromise = null; + +export function openDb() { + dbPromise ??= new Promise((resolve, reject) => { + const request = indexedDB.open(DB_NAME, DB_VERSION); + request.onupgradeneeded = () => { + request.result.createObjectStore('outliers', { keyPath: 'videoId' }).createIndex('updatedAt', 'updatedAt'); + }; + request.onsuccess = () => resolve(request.result); + request.onerror = () => reject(request.error); + }).catch((err) => { + dbPromise = null; + throw err; + }); + return dbPromise; +} + +function promisify(request) { + return new Promise((resolve, reject) => { + request.onsuccess = () => resolve(request.result); + request.onerror = () => reject(request.error); + }); +} + +export async function getOutliers(videoIds) { + const db = await openDb(); + const store = db.transaction('outliers', 'readonly').objectStore('outliers'); + const records = await Promise.all(videoIds.map((id) => promisify(store.get(id)))); + return new Map(records.filter(Boolean).map((record) => [record.videoId, record])); +} + +export async function putOutliers(records) { + if (records.length === 0) return; + const db = await openDb(); + await new Promise((resolve, reject) => { + const tx = db.transaction('outliers', 'readwrite'); + const store = tx.objectStore('outliers'); + for (const record of records) store.put(record); + tx.oncomplete = () => resolve(); + tx.onerror = () => reject(tx.error); + tx.onabort = () => reject(tx.error); + }); +} + +// Newest `updatedAt` first. +export async function listOutliers(limit = Infinity) { + const db = await openDb(); + return new Promise((resolve, reject) => { + const results = []; + const request = db.transaction('outliers', 'readonly').objectStore('outliers').index('updatedAt').openCursor(null, 'prev'); + request.onsuccess = () => { + const cursor = request.result; + if (!cursor || results.length >= limit) { + resolve(results); + return; + } + results.push(cursor.value); + cursor.continue(); + }; + request.onerror = () => reject(request.error); + }); +} + +export async function countOutliers() { + const db = await openDb(); + return promisify(db.transaction('outliers', 'readonly').objectStore('outliers').count()); +} +``` + +- [ ] **Step 4: Run the tests and a syntax check** + +Run: `node --test test/export.test.js test/storage.test.js && node --check src/lib/db.js` +Expected: PASS (export 6 tests, storage 5 tests), no syntax errors. + +- [ ] **Step 5: Commit** + +```bash +git add src/lib/export.js src/lib/storage.js src/lib/db.js test/export.test.js test/storage.test.js +git commit -m "refactor: align export, settings and database with the VidIQ record model" -m "Co-Authored-By: Claude Sonnet 5.5 " +``` + +--- + +### Task 6: Service worker, манифест и удаление слоя API + +**Files:** +- Modify (rewrite): `src/background.js`, `manifest.json` +- Delete: `src/lib/api.js`, `src/lib/process.js`, `src/lib/score.js`, `test/api-helpers.test.js`, `test/api-requests.test.js`, `test/process.test.js`, `test/score.test.js` + +**Interfaces:** +- Consumes: `sanitizeItem`, `buildRecord` (record.js); `getSettings` (storage.js); `getOutliers`, `putOutliers` (db.js). +- Produces: обработчик сообщений `{ type: 'cards', items: Item[] }` (до 200 карточек за сообщение); сообщения обрабатываются строго по очереди. + +- [ ] **Step 1: Write `src/background.js`** + +```js +import { getOutliers, putOutliers } from './lib/db.js'; +import { buildRecord, sanitizeItem } from './lib/record.js'; +import { getSettings } from './lib/storage.js'; + +const MAX_ITEMS_PER_MESSAGE = 200; + +async function store(rawItems) { + const items = rawItems.slice(0, MAX_ITEMS_PER_MESSAGE).map(sanitizeItem).filter(Boolean); + if (items.length === 0) return; + const [settings, existing] = await Promise.all([getSettings(), getOutliers(items.map((item) => item.videoId))]); + const now = new Date(); + const records = items.map((item) => buildRecord(item, existing.get(item.videoId), settings, now)).filter(Boolean); + await putOutliers(records); +} + +// Messages are handled one after another so two batches never read-modify-write the same video at once. +let queue = Promise.resolve(); + +chrome.runtime.onMessage.addListener((message) => { + if (message?.type !== 'cards' || !Array.isArray(message.items)) return; + queue = queue.then(() => store(message.items)).catch((err) => console.warn('Outlier Saver: failed to store cards', err)); +}); +``` + +- [ ] **Step 2: Write `manifest.json`** (пока без content script, popup и options остаются как были) + +```json +{ + "manifest_version": 3, + "name": "Outlier Saver", + "version": "0.2.0", + "description": "Сохраняет видео YouTube с высоким outlier score от VidIQ.", + "permissions": ["storage"], + "background": { "service_worker": "src/background.js", "type": "module" }, + "action": { "default_title": "Outlier Saver", "default_popup": "src/popup/popup.html" }, + "options_ui": { "page": "src/options/options.html", "open_in_tab": true }, + "content_scripts": [ + { + "matches": ["https://www.youtube.com/*"], + "js": ["src/content.js"], + "run_at": "document_idle" + } + ], + "web_accessible_resources": [ + { + "resources": ["src/lib/*.js"], + "matches": ["https://www.youtube.com/*"] + } + ] +} +``` + +- [ ] **Step 3: Delete the API layer and its tests** + +Run: `git rm src/lib/api.js src/lib/process.js src/lib/score.js test/api-helpers.test.js test/api-requests.test.js test/process.test.js test/score.test.js` + +- [ ] **Step 4: Verify** + +Run: `node --check src/background.js && python3 -c "import json;json.load(open('manifest.json'))" && npm test` +Expected: syntax OK, manifest valid JSON, all remaining tests pass (video-id, numbers, vidiq, record, export, storage, card.dom). + +- [ ] **Step 5: Commit** + +```bash +git add manifest.json src/background.js +git commit -m "refactor: store VidIQ cards in the service worker and drop the API layer" -m "Co-Authored-By: Claude Sonnet 5.5 " +``` + +--- + +### Task 7: Content script + +**Files:** +- Modify (rewrite): `src/content.js` + +**Interfaces:** +- Consumes: `pageFromPath` (video-id.js); `CARD_SELECTOR`, `LINK_SELECTOR`, `scrapeCard` (card.js) — оба загружаются через динамический `import()` (content script не может быть ES-модулем), поэтому `src/lib/*.js` перечислены в `web_accessible_resources`. +- Produces: сообщения `{ type: 'cards', items: Item[] }`. + +Поведение: сканирует не чаще раза в 500 мс (троттлинг); пропускает карточки без `outlierScore` или `views`; повторно отправляет карточку, только если изменилась подпись `outlierScore|views|subscribers`; после `yt-navigate-finish` сбрасывает подписи и сканирует заново; на странице канала берёт канал из URL и заголовка вкладки; отключается, если расширение перезагрузили. + +- [ ] **Step 1: Write `src/content.js`** + +```js +(async () => { + const { pageFromPath } = await import(chrome.runtime.getURL('src/lib/video-id.js')); + const { CARD_SELECTOR, LINK_SELECTOR, scrapeCard } = await import(chrome.runtime.getURL('src/lib/card.js')); + + const SCAN_INTERVAL_MS = 500; + const MAX_ITEMS_PER_MESSAGE = 200; + + // videoId -> "outlierScore|views|subscribers" of the last message that carried the video. + // VidIQ may draw or update its numbers after the card appears, so a changed signature is sent again. + let sent = new Map(); + let scanTimer = null; + + // Cards on a channel page do not name the channel, so take it from the page itself. + function channelContext() { + const [kind, name] = location.pathname.split('/').filter(Boolean); + const path = kind?.startsWith('@') ? kind : ['channel', 'c', 'user'].includes(kind) && name ? `${kind}/${name}` : null; + return { + channelUrl: path ? `https://www.youtube.com/${path}` : null, + channelTitle: document.title.replace(/^\(\d+\)\s*/, '').replace(/\s*-\s*YouTube$/, '') || null, + }; + } + + function scan() { + scanTimer = null; + const page = pageFromPath(location.pathname); + if (!page) return; + const context = page === 'channel' ? channelContext() : null; + + const found = new Map(); + const visited = new Set(); + for (const link of document.querySelectorAll(LINK_SELECTOR)) { + const card = link.closest(CARD_SELECTOR); + if (!card || visited.has(card)) continue; + visited.add(card); + const item = scrapeCard(card, link, context); + if (!item || item.outlierScore === null || item.views === null) continue; + const signature = `${item.outlierScore}|${item.views}|${item.subscribers}`; + if (sent.get(item.videoId) === signature) continue; + found.set(item.videoId, { item: { ...item, page }, signature }); + } + if (found.size === 0) return; + + const batch = [...found.values()].slice(0, MAX_ITEMS_PER_MESSAGE); + for (const { item, signature } of batch) sent.set(item.videoId, signature); + try { + chrome.runtime.sendMessage({ type: 'cards', items: batch.map(({ item }) => item) }).catch(() => {}); + } catch { + observer.disconnect(); // the extension was reloaded; this script is orphaned + return; + } + if (found.size > batch.length) scheduleScan(); + } + + function scheduleScan() { + if (scanTimer === null) scanTimer = setTimeout(scan, SCAN_INTERVAL_MS); + } + + const observer = new MutationObserver(scheduleScan); + observer.observe(document.documentElement, { childList: true, subtree: true }); + document.addEventListener('yt-navigate-finish', () => { + sent = new Map(); + scheduleScan(); + }); + scheduleScan(); +})(); +``` + +- [ ] **Step 2: Verify** + +Run: `node --check src/content.js && npm test` +Expected: no syntax errors, all tests pass. Работу в браузере проверит Task 9. + +- [ ] **Step 3: Commit** + +```bash +git add src/content.js +git commit -m "feat: scan video cards and send VidIQ-scored ones to the service worker" -m "Co-Authored-By: Claude Sonnet 5.5 " +``` + +--- + +### Task 8: Настройки и popup без ключа и статуса + +**Files:** +- Modify: `src/options/options.html`, `src/options/options.js` +- Modify: `src/popup/popup.html`, `src/popup/popup.css`, `src/popup/popup.js` + +**Interfaces:** +- Consumes: `getSettings`, `saveSettings` (storage.js); `countOutliers`, `listOutliers` (db.js); `toCsv`, `toJson` (export.js). + +- [ ] **Step 1: Options page** + +В `src/options/options.html` удалить `