Social Fetch puts one key and one JSON shape in front of public data from TikTok, Instagram, LinkedIn, X, YouTube and more than a dozen other sites, fetched live on each request. The code is the easy part. Instagram, TikTok, LinkedIn and Facebook work to stop automated reads, so what you pay for is proxies, sessions and parsers that someone repairs when a site changes. An agent can build a clean fetcher in an evening for the platforms with open APIs, such as YouTube, Reddit, Bluesky, GitHub and Hacker News; keeping the hostile ones working is an ongoing job.
Before you buildSocial Fetch gives 100 credits on sign-up and 250 more once you save a card, and a profile lookup is usually 1 credit. Try your real queries there before you build anything.
Build me a personal social-data fetcher to replace Social Fetch for the platforms that allow it. Requirements:
- Python 3.12: FastAPI served by uvicorn on http://127.0.0.1:4900 and a `typer` CLI
(`sfetch youtube-video <url>`, `sfetch reddit-search "<query>"`); both call the same functions.
- One module per provider behind a common interface: YouTube (Data API v3 with YOUTUBE_API_KEY
for channels, videos and comments, plus `youtube-transcript-api` for captions), Reddit (`praw`
with a script app: subreddit posts, threads, search), Bluesky (public XRPC endpoints, no key:
profile, author feed, post thread), GitHub (REST with GITHUB_TOKEN: users, repos, issues) and
Hacker News (the Algolia search API).
- One envelope for every response: {status: found | not_found | private | error, platform, kind,
data, fetchedAt}. Normalise id, url, author, text, createdAt and counts the same way in every
provider and keep the provider's raw payload under `raw`.
- Storage: SQLite in ./data/cache.db with a TTL per kind (default 10 minutes), a log of every
call (provider, kind, status, milliseconds) and a daily quota counter per provider that refuses
a call once the limit set in .env is reached.
- Errors: retry 429 and 5xx up to 3 times with exponential backoff and honour Retry-After; never
retry other 4xx; return the provider's message in the envelope.
- Tests: `pytest` with recorded responses (`vcrpy`) so the suite runs without any key.
- Single user. No accounts, no telemetry, local-first; keys live in .env and the only outbound
calls go to the providers above.
- Out of scope: Instagram, TikTok, LinkedIn, Facebook and X (they block unauthenticated reads and
need rotating proxies and logged-in sessions), media downloads, and proxies of any kind.
- README: how to get each key, the quota per provider, and a plain warning that reading a platform
against its terms is your risk.Opens your agent with the prompt filled in, you press enter. Or copy it and paste it into an empty folder.
The first prompt builds the core. Run these one at a time in the same folder to grow it into your own Social Fetch.
Add account watching to the fetcher in this folder. Requirements: - A `watch.yml` file listing accounts per provider (YouTube channels, Bluesky handles, Reddit users, GitHub users). - A command `sfetch watch run` that fetches each entry, stores a snapshot in SQLite and prints new posts and changes in follower or star counts since the last run. - Respect the daily quota: stop and report which entries were skipped. - An optional `--markdown` flag that writes the report to ./reports/YYYY-MM-DD.md. - Update the README with an example cron line.
Add a yt-dlp provider to the fetcher in this folder. Requirements: - Call `yt-dlp --dump-json --skip-download` through a subprocess with a 60-second timeout; never download the media itself. - Map the output into the shared envelope (title, author, description, duration, view and like counts, upload date) and keep the full JSON under `raw`. - A `subtitles` kind that returns captions as plain text with timestamps. - If yt-dlp is missing or the site is unsupported, return status `error` with a clear message. - Update the README with the install command and a note to keep yt-dlp up to date.
Add CSV export to the fetcher in this folder. Requirements: - A command `sfetch export --kind <kind> --since <date> --out file.csv` that writes cached results of one kind as flat columns (id, url, author, text, createdAt, counts). - Flatten nested fields with a dot in the column name; always quote text and escape newlines. - Skip rows whose status is not `found` unless `--all` is given. - Update the README with two examples.
Teams building trend trackers, research tools and AI features need data from sites that don't want to be read, and running their own proxies and scrapers is a second job. A few hundred lookups a month fit a $14 credit pack with no subscription, and the monthly plans scale the price per credit down as volume grows.
Estimate · The official APIs are free at personal volume and the cache runs on your machine. The saving ends where you would need proxies for the hostile platforms, which is the reason for the verdict. The build is token usage on your agent plan.
| Plan | Monthly | Annual, per month | What you get |
|---|---|---|---|
| Free credits | $0 | $0 | 100 credits on sign-up, plus 250 if you save a card before buying anything. |
| Pay as you go | custom | — | One-time packs: 1,000 credits for $14, 25,000 for $59, 230,000 for $379. One person, no subscription. |
| Starter | $29/workspace | $25.83/workspace | 18,000 credits a month, 1 seat. |
| Growth | $79/workspace | $70.83/workspace | 50,000 credits a month, up to 5 seats. |
| Scale | $225/workspace | $202.50/workspace | 150,000 credits a month, up to 25 seats (more on request). |
| Enterprise | custom | — | For 1M+ credits: invoicing and custom terms. |
Sign-up gives 100 credits with no card; saving a card adds 250 more, and nothing is charged until you buy.
Not really. Teams building trend trackers, research tools and AI features need data from sites that don't want to be read, and running their own proxies and scrapers is a second job. A few hundred lookups a month fit a $14 credit pack with no subscription, and the monthly plans scale the price per credit down as volume grows. The prompt on this page builds the closest honest substitute.
About $29 a month on the Starter plan, $348 a year (checked 2026-10-19).
Live reads of Instagram, TikTok, LinkedIn, Facebook and X without running proxies or logins; The same JSON shape for 20+ platforms, including comments, transcripts and hosted media files; Search, ad libraries and follower lists on platforms that hide them; A vendor that repairs adapters when a platform changes; A published uptime page and a request ID on every response for support. If any of those matter to you, keep paying.
Yes: yt-dlp (open source), Instaloader (open source), gallery-dl (open source).