How to Build a Job Board Aggregator: Sources, Dedup and Freshness
A good job aggregator is three engineering problems: where to source listings, how to dedupe them, and how to keep them fresh. Solve those and the scraping is the easy part — here is the blueprint.
A job board aggregator is a search engine for job posts: it pulls listings from many sources and shows them in one place. The hard parts are not the HTTP requests — they are choosing the right sources, deduplicating the same job that appears on five sites, and keeping everything fresh so you are not showing roles that filled last month. Get those three right and you have a defensible product; get them wrong and you have a stale, duplicate-ridden feed that erodes trust. This guide is the engineering blueprint: source mix, canonical dedup, freshness, the jobs SERP vertical, and the legal lines to respect. It is informational, not legal advice.
Source tier 1: public ATS endpoints
The cleanest job data on the internet is not on job boards — it is on company career pages powered by applicant tracking systems, most of which expose structured JSON. Greenhouse, Lever, Ashby, BambooHR, iCIMS, Paylocity and Workday all serve listings you can request directly, no HTML parsing. An open-source aggregator that pulls from all seven fetches them in parallel and tunes worker counts to each platform's rate limits — roughly 50 concurrent for Workday, 30 for Greenhouse/Lever/iCIMS, 10 for BambooHR, and 5 for the tightest limiters. This tier gives you canonical titles, locations, salary bands and apply URLs straight from the employer, which makes it the backbone of any serious aggregator.
import httpx
# Greenhouse exposes a public board JSON endpoint per company slug
def fetch_greenhouse(slug: str) -> list[dict]:
url = f"https://boards-api.greenhouse.io/v1/boards/{slug}/jobs?content=true"
r = httpx.get(url, timeout=20)
r.raise_for_status()
jobs = r.json().get("jobs", [])
return [{
"source": "greenhouse",
"external_id": str(j["id"]),
"title": j["title"],
"company": slug,
"location": (j.get("location") or {}).get("name"),
"apply_url": j["absolute_url"],
"updated_at": j.get("updated_at"),
} for j in jobs]
rows = fetch_greenhouse("examplecompany")
print(len(rows), "roles")To find companies to poll, harvest ATS slugs at scale from a web index rather than by hand — scanning Common Crawl's URL archives for ATS domain patterns can surface tens of thousands of company identifiers. That discovery crawl is itself a scraping job; route it through a datacenter proxy pool since the index is lenient and speed matters more than IP reputation there.
Source tier 2: the jobs SERP vertical
ATS feeds miss everything posted only to aggregator sites, regional boards or Google's own jobs experience. The efficient way to cover that long tail without scraping a dozen boards individually is the jobs SERP vertical — a structured query that returns the jobs Google surfaces for a role and location, already normalised. Our SERP API exposes a jobs vertical alongside news, images and shopping, so you can pull role listings by keyword and geo as JSON and fold them into the same pipeline:
import httpx
def fetch_jobs_serp(query: str, location: str) -> list[dict]:
r = httpx.get(
"https://api.quantumproxies.io/serp",
params={"engine": "google_jobs", "q": query,
"location": location, "api_key": "QP_API_KEY"},
timeout=30,
)
jobs = r.json().get("jobs", [])
return [{
"source": "jobs_serp",
"external_id": j.get("job_id"),
"title": j.get("title"),
"company": j.get("company_name"),
"location": j.get("location"),
"apply_url": (j.get("apply_options") or [{}])[0].get("link"),
} for j in jobs]
serp_rows = fetch_jobs_serp("react developer", "Austin, TX")Run the same query across your target cities on a schedule and you have geo coverage no single ATS feed provides. Because SERP results shift by location, geo-target each query — our post on how SERP scraping works in 2026 covers the geo and pagination mechanics.
Pull the jobs vertical from the SERP API
Dedup by canonical key
The same job routinely appears on the company site, two aggregators and a Google jobs card. Showing it four times is the fastest way to look broken. The standard fix is a canonical key: normalise and hash the combination of job title, company, location and — when available — the source's external job ID. Lowercase, strip punctuation and collapse whitespace before hashing so "Sr. Software Engineer" and "senior software engineer" collapse together.
import re, hashlib
def canonical_key(job: dict) -> str:
def norm(s):
return re.sub(r"[^a-z0-9]+", " ", (s or "").lower()).strip()
basis = "|".join([
norm(job.get("title")),
norm(job.get("company")),
norm(job.get("location")),
(job.get("external_id") or ""),
])
return hashlib.sha1(basis.encode()).hexdigest()
def dedupe(rows: list[dict]) -> list[dict]:
seen, out = set(), []
for job in rows:
k = canonical_key(job)
if k not in seen:
seen.add(k)
out.append(job)
return outFreshness is a feature, not an afterthought
A job board's credibility is its freshness. Two mechanics keep it honest: re-fetch each source on a schedule (hourly for high-volume ATS feeds, daily for the long tail), and prune anything you have not re-seen in a rolling window — 30 days is a sensible default, shorter for fast-moving markets. Track a per-source last-seen timestamp and a daily count so you can spot when a source silently breaks; an aggregator that watches its own volume can open an alert the moment a platform's numbers drop off a cliff. Our note on running scrapers as production software covers scheduling, drift detection and alerting.
The legal lines to respect
Job data is where scraping law gets real, because postings can carry personal data (recruiter names, contact details) and detailed descriptions are copyrightable. France's data-protection authority fined the firm KASPR €240,000 for scraping LinkedIn contact data without proper consent; GDPR penalties can reach €20 million or 4% of global turnover, and copyright damages have run to $150,000 per work in US cases. The safer posture is structural: prefer authorised ATS feeds and the jobs SERP over brute-forcing login-walled boards, store factual fields (title, company, location) rather than republishing full copyrighted descriptions, and strip personal data you do not need. Our overview of web scraping legality in 2026 and the LinkedIn compliance saga go deeper.
Where proxies fit
The ATS tier rarely needs residential IPs, but the discovery crawl, the SERP vertical and any direct board scraping do — Google and the bigger boards rate-limit by IP and geo-vary their results. Route those through rotating residential proxies with per-request rotation and geo-targeting so each city query comes from a matching location. Keep the tiers separate: cheap datacenter for the lenient index, residential for the sensitive targets.
Frequently asked questions
What is a job board aggregator?
It is a search engine for job postings that collects listings from many sources — company career pages, ATS platforms, other boards and the jobs SERP — normalises them into one schema, removes duplicates, and presents them to job seekers in a single searchable place. The engineering work is in sourcing, deduplication and freshness, not just the scraping.
How do you dedupe job listings from multiple sources?
Build a canonical key by normalising and hashing job title, company, location and the source's external job ID. Lowercase, strip punctuation and collapse whitespace first so variant spellings collapse together. Keep the first occurrence and drop matches. This catches the same role appearing on the employer site, aggregators and Google jobs.
Is scraping job boards legal?
It depends on the source, the data and your jurisdiction. Public factual listing fields are lower risk than personal data or full copyrighted descriptions, and bypassing login walls raises CFAA-style exposure. Authorised ATS feeds and the jobs SERP are the cleaner routes. Firms have been fined heavily for scraping personal data — get legal advice for commercial use.
How fresh should aggregated job data be?
Re-fetch high-volume sources like ATS feeds hourly and the long tail daily, then prune any listing you have not re-seen within a rolling window — 30 days is a common default. Track a last-seen timestamp per job and monitor per-source volume so you catch a broken source before users see stale roles.
Treat aggregation as three problems — sources, dedup, freshness — and the scraping becomes a supporting detail. Lead with clean ATS feeds and the jobs SERP vertical, dedupe on a canonical key, prune aggressively, and keep the sensitive crawls on rotating residential IPs. That is the difference between a product people trust and a graveyard of dead links.