The arXiv API turns a search into preprint metadata as typed rows — for each paper the arXiv id with its version, title, full abstract, up to twenty authors, first-submission and last-update dates, subject categories, the DOI when one has been assigned, a direct PDF link and the abstract-page URL.
arXiv is where machine learning, physics and mathematics publish first, months before the journals catch up, which makes it the feed to watch rather than the archive to visit. Fielded queries pass through untouched — ti: for titles, au: for authors, abs: for abstracts — a category filter pins a run to one subject class like cs.AI, and sort=latest turns the whole thing into a standing what's-new feed. For the peer-reviewed record behind the preprints there is the academic paper API, and for medical research the clinical trials API — same key, same row discipline.
$0.0004 per delivered paper, up to 100 papers per run. Nothing delivered means nothing charged.
One POST reads arXiv's export interface, which speaks Atom XML; the collector parses that into flat rows so nothing XML-shaped ever reaches your code. A plain query searches every field. The choice that shapes a pipeline is sort:
relevance for literature search — the best match to a topic, regardless of age.latest for monitoring — newest submissions first, the mode behind every "papers this week in cs.CL" digest.updated for revision tracking — it surfaces the v2s and v3s of work you may have already read, which relevance ordering hides.Up to 100 papers arrive per run, each with a 1-based rank, and a query that matches nothing costs nothing.
query is the only required input and accepts free text or arXiv's own fielded grammar (ti:, au:, abs:). category narrows to a single subject class — cs.AI, cs.CL, stat.ML — usually the difference between a readable feed and a firehose. max_results takes 1–100 and defaults to 25.
arxiv_id carries the version suffix — 2401.12345v3 — so revision detection is a string comparison, no date arithmetic. abstract is the full abstract text, the substance most screening pipelines actually run on. categories lists the primary class and every cross-list, pdf_url is the direct file link for whatever your side chooses to fetch, and a non-null doi is itself a signal: the preprint has formally landed in a journal.
$0.0004 per delivered paper ($0.40 per 1,000). Nothing delivered means nothing charged, and the $2 monthly free credit covers roughly 5,000 papers here. Volume tiers take up to 30% off.
$0.0004 per delivered paper — 40 cents per 1,000. A daily 100-paper category feed costs four cents a day, about
Empty result sets are never billed, and neither is anything arXiv declines to return.
arXiv's export API is free, keyless and deliberately open — arXiv wants programmatic readers, and if you enjoy Atom XML you can have it directly today. The friction is entirely mechanical: responses are XML feeds, the query string is its own mini-language of search_query clauses joined by URL-encoded ANDs, pagination is manual, and arXiv's usage guidance asks automated clients to identify themselves and pace requests to roughly one every three seconds — reasonable, and one more thing your cron job must get right forever.
Through the collector you write JSON and receive rows; the XML, the encoding, the pacing and the retries sit on our side of the bill. Same honest framing as the rest of this catalogue: the source is free, and the $0.0004 buys the part between the source and your table.
There are decent open-source arXiv clients, and inside a notebook they are the right call. Where DIY frays is the standing pipeline:
Keep the notebook client for exploration — and there is a residential proxy network under this same account for the sources in your pipeline that are far less welcoming than arXiv.
One run per category with sort=latest each morning, dedupe on arxiv_id, and the lab's Slack gets a daily digest of genuinely new work. An au: query does the same for the specific researchers worth following.
Systematic reviews and survey chapters start by triaging hundreds of candidates. Titles, abstracts and categories as rows let the first screening pass run over a spreadsheet — or a classifier — reserving human hours for the papers that survive it.
au: sweeps over a company's known researchers, run on updated sort, show what a rival lab is working on in near real time — and the doi field flags which of those preprints have quietly become peer-reviewed publications.
Rows are metadata and abstracts — the collector never fetches paper bodies. pdf_url is there for your side to use, and for corpus-scale full text arXiv operates dedicated bulk-access channels that are the appropriate tool. One run delivers up to 100 papers with at most one category filter; broader monitoring is multiple runs, parallelised within your plan's request budget — 60 a minute at entry, 1,200 at the top — while the pacing toward arXiv itself stays our responsibility and stays polite.
Worth knowing: new submissions are announced on arXiv's own schedule, roughly one batch per working day, so polling a feed hourly mostly re-reads yesterday. Metadata and abstracts are intentionally open; the papers themselves carry author-chosen licences, so check before republishing full text. arXiv is community infrastructure — treat it accordingly.
arXiv's export interface is free at the source and always has been. This collector layers a $2 free monthly allowance on top — 5,000 delivered papers at $0.0004 each — so a personal research feed typically never generates a bill; you start paying only when volume does.
arXiv asks automated clients to pace themselves — its guidance suggests roughly one request every three seconds. Using the collector, that source-side pacing is handled for you; the ceiling you actually manage is your QuantumProxies plan rate, from 60 requests a minute on pay-as-you-go up to 1,200 on the largest tier.
Use arXiv's own prefixes inside the query — ti:diffusion for titles, au:hinton for authors, abs:contrastive for abstracts — and set the category input to a subject class like cs.CV to fence the run in. Prefixes combine with plain text, and sort picks relevance, latest or updated ordering.
Not through the collector — each row carries pdf_url, the direct link, and the download is yours to make. That split is deliberate: metadata at $0.0004 a row is cheap to sweep broadly, while PDF retrieval at scale belongs on arXiv's bulk-access routes rather than hidden inside an API bill.