I read a lot of public pages. Watching what AI vendors ship, researching Codex behavior, gathering seeds for articles, checking sources before publishing.
Reading one page in a browser is fine. The friction starts when that page has to be handed to another AI task, folded into an article outline, and dropped into a comparison table. At that point I am re-extracting the same title, the same key claims, the same source URL and access date, over and over.
The problem was never the reading. It was turning the same public page into something reusable.
So I built public-source-extractor: a small CLI that takes one public URL and converts it into Markdown or JSON that an AI research workflow can actually consume. It started as a personal Python script and grew into something I was willing to publish.
Read this before you use it: the URL you pass is sent to Firecrawl Cloud for extraction. This CLI does not read API keys or cookies, but it is not fully local processing either. It is also not a general-purpose scraper, and its output is not a substitute for reading the original source.
Originally published in Japanese on July 12, 2026, against the alpha releases. This English version was written in September 2026, when v0.1.0 became the current release on both GitHub and PyPI.
The same manual cleanup, every time
When researching an AI tool or feature, the official page is the first stop. If I only need to read it once, nothing is wrong. But as soon as the page becomes a seed for an article, or one of several candidates I am comparing, the same work reappears:
- Confirm the title
- Pull out the central claim
- Record the source URL and when I retrieved it
- Put the key points into Markdown or a table
- Shape it into something the next AI task can take
Individually trivial. Across several pages a day it adds up.
What it does
It extracts exactly one public HTTP or HTTPS page. Markdown by default, JSON when you need fixed fields.
public-source-extractor https://example.com/For comparison tables or article seeds, where I want title, summary, main argument, and key points in a consistent shape:
public-source-extractor https://example.com/ --mode json --prettyTo write to a file instead of stdout, pass --output report.md. To pin a specific version through uvx:
uvx public-source-extractor@0.1.0 --version
uvx public-source-extractor@0.1.0 https://example.com/The second command sends your URL to Firecrawl Cloud. Requires Python 3.11+ and uv.
The goal is not to automate research end to end. It is to get a public page into a consistent shape once, so the next step can start from something structured.
The boundary came before the features
A CLI that accepts URLs invites scope creep. Once it can fetch anything, you stop knowing what is safe to hand it. So I fixed the boundary first.
What it does: extract one public page, convert to Markdown or versioned JSON, write to stdout or a new file.
What it will not do: extract login pages, admin screens, or private pages; use cookies, browser profiles, localStorage, or credentials; crawl a whole site; drive Chrome or perform computer use; judge source reliability automatically; fall back between providers; send telemetry.
URLs are not accepted blindly either. It rejects localhost, private IP ranges, paths that look like login or admin, and queries carrying what appear to be tokens or signatures.
That said, those checks do not fully prevent private URLs or redirects reaching the provider. DNS rebinding, and what the provider ultimately fetches, are outside what this CLI can guarantee. So the rule stays with the user: pass public URLs only. No signed URLs, no URLs containing tokens, no post-login URLs.
Firecrawl Keyless is convenient, and experimental
Extraction uses the firecrawl-keyless provider. As of July 12, 2026, it worked without an API key. I am not going to claim that means “free forever.”
Firecrawl’s Keyless launch post describes keyless access and the credits granted at the time, including direct REST calls without an Authorization header. The current rate limit documentation mentions per-IP daily request and credit limits without publishing specific numbers, and the Scrape API reference shows examples that do use an Authorization header. Reading only the official sources, the sensible conclusion is that the terms of keyless access can change.
The provider is therefore marked experimental. Do not build on the assumption of permanent anonymous REST access or a stable credit allowance.
The other thing worth repeating: the URL you pass is transmitted to Firecrawl Cloud. That is part of why the tool restricts itself to public pages.
And the extracted content is not the source of truth. It can contain errors, and it can contain instruction-shaped text — prompt injection is a real possibility in anything scraped from the open web. Treat the output as untrusted data. Verify important claims against the original URL and primary sources. This CLI does not remove that step; it makes the material easier to handle before you take it.
What using it revealed
After publishing, I ran JSON mode against real research pages and found the boundary needed adjusting in small ways. Provider credit usage and elapsed time were not visible in the output, which made it hard to judge how much of a quota a run had consumed. That went into an issue, then a PR, and shipped in the next alpha.
That is the loop I want: publish something narrow, use it daily, fix only what I have actually hit, and ship what I can verify.
Small tools you actually use
You do not need to build a full AI research system up front. Turn a public page into a reusable artifact. Carry the source URL and retrieval date with it into the next step. That is enough to start.
The same thinking produced codex-healthkit, a local health check CLI for daily Codex use. Both are small, both refuse to do more than they can do safely, and both exist because I got tired of repeating something by hand.
Deciding what you hand to an AI, and how far it goes, is the part worth designing. Once that boundary exists, the research record becomes much easier to keep.