How to Scrape Public Instagram Data for Marketing Research
Instagram killed public API access in 2020, but the public web endpoints still return profiles, posts and hashtag data as JSON. Here's what's reachable, why mobile IPs matter, and where the line is.
Instagram closed public access to its official API in 2020, which is why every guide to scraping Instagram data now leans on the public web endpoints instead. The good news for marketing research: those endpoints still return profiles, posts, reels and hashtag volumes as clean JSON, no login required. The catch is that Instagram is one of the most aggressive platforms at flagging automated traffic, so the IP you use and the pace you keep decide whether the endpoint stays open. This guide covers what is reachable, the code to reach it, and the ToS and GDPR lines you should not cross.
What public Instagram data you can actually collect
Without an account you can reach a surprising amount, all of it useful for market and influencer research:
- Profiles: full name, bio, follower and following counts, post count, verified and business status.
- Posts and reels: caption, hashtags, mentions, media URLs, like and comment counts, timestamps, and the first few comments.
- Hashtags: total post volume and related-tag clusters - handy for finding less-saturated adjacent tags.
- Places: name, coordinates, address and post count for a location.
What you cannot get without an authenticated session: full follower lists, the list of accounts that liked a post, private accounts and stories. Those endpoints require a session cookie, and that is exactly where account bans and ToS violations concentrate. For research, stay on the public surface.
The public profile endpoint, with code
Instagram's web app fetches profile data from a JSON endpoint that anyone can call, provided you send the public web app id header. Route it through a proxy and you have profile data in a few lines:
import requests
PROXY = "http://USER:PASS@gate.quantumproxies.io:8000" # mobile pool
proxies = {"http": PROXY, "https": PROXY}
HEADERS = {
"x-ig-app-id": "936619743392459", # public web app id
"User-Agent": "Mozilla/5.0 (iPhone; CPU iPhone OS 17_2 like Mac OS X) "
"AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148",
}
def profile(username):
url = "https://www.instagram.com/api/v1/users/web_profile_info/"
r = requests.get(url, params={"username": username},
headers=HEADERS, proxies=proxies, timeout=20)
r.raise_for_status()
return r.json()["data"]["user"]
u = profile("natgeo")
print(u["full_name"], u["edge_followed_by"]["count"])The same payload carries recent posts under edge_owner_to_timeline_media, so one call gives you the profile and its latest engagement together:
for edge in u["edge_owner_to_timeline_media"]["edges"]:
node = edge["node"]
print(node["shortcode"],
node["edge_media_preview_like"]["count"], # likes
node["edge_media_to_comment"]["count"]) # commentsWhy mobile proxies win on Instagram
Instagram's most common failure is a rate-limit block: the endpoint starts returning errors and temporarily bans the IP that hit it. Waiting and lengthening timeouts helps a little, but the durable fix is the IP type. Mobile proxies are the strongest option here because of CGNAT - a single mobile IP is shared by thousands of real subscribers, so Instagram cannot block it without hitting genuine users. That makes mobile exits the hardest to ban and the closest match to how most people actually browse the app. Our explainer on why CGNAT makes mobile IPs trusted covers the mechanism, and mobile proxies vs VPNs explains why a VPN can't do this.
Pair a mobile proxy pool with human pacing - random delays, capped concurrency, and a stop when the block rate climbs - and the public endpoints stay open across long collection runs.
Residential proxies are a reasonable fallback when a mobile pool isn't available - our residential network spans 200+ countries for geo-accurate collection - but on Instagram specifically, mobile exits survive longest. Whatever the IP type, treat a rate-limit response as a signal to slow down, not to retry harder: back off, rotate, and lengthen the gap between calls. A scraper that respects the throttle collects far more over a day than one that hammers through blocks and burns its pool in an hour.
Get mobile proxies built for social platforms
Hashtags and discovery
For trend and influencer discovery, hashtag data beats scraping individual profiles. The hashtag pages expose post volume and related-tag clusters, which let you map how saturated a topic is and find quieter adjacent tags to target. Combine that with per-profile engagement rates - likes and comments over follower count - and you have a defensible shortlist of creators without ever touching gated data. If your research spans platforms, the same discipline applies to TikTok public data, and there's overlap with running social automation across Instagram, TikTok and X.
What to keep, and what to drop
For research, store the aggregate signals and drop the rest. Follower and following counts, posting frequency, average likes and comments, and hashtag usage tell you everything about reach and engagement without retaining a pile of personal detail. Media URLs expire and full captions bloat your storage, so keep stable identifiers like the shortcode plus the metrics, and re-fetch content on demand. Minimising what you hold is both good engineering and the safer side of the GDPR line - a table of engagement rates is far less sensitive than a warehouse of named accounts and their posts.
The ToS and GDPR lines
Two boundaries matter. First, Instagram's terms restrict automated collection, and using a logged-in session to scrape gated data risks the account, not just the IP - keep to public, unauthenticated endpoints for research. Second, follower handles, comment authors and profile details are personal data. Under GDPR and similar regimes you need a lawful basis to collect and store it, you should minimise what you keep, and you should avoid building profiles of identifiable individuals without a clear reason. Aggregate metrics (follower counts, engagement rates, hashtag volumes) are far safer than lists of named people. Our guide to GDPR and scraping personal data covers the specifics - and this is practical guidance, not legal advice.
Frequently asked questions
Can you scrape Instagram data without an account?
Yes, for public data. The web profile endpoint returns bios, follower counts and recent posts as JSON when you send the public x-ig-app-id header, and hashtag and place pages expose volume and metadata. Follower lists, likers and private accounts need an authenticated session and should be avoided for research.
Why does Instagram keep rate-limiting my scraper?
You are hitting the endpoint too fast from an IP Instagram distrusts. Slow down with random delays, cap concurrency, and switch to mobile proxies - a CGNAT mobile IP is shared by thousands of real users, so Instagram is far slower to block it than a datacenter address.
Is scraping Instagram data legal?
Collecting public, non-personal data for research is generally lower-risk, but Instagram's terms restrict automation and personal data is governed by GDPR and similar laws. Stay on public endpoints, minimise the personal data you store, and consult a lawyer for anything involving identifiable individuals at scale. This is not legal advice.
What's the best proxy type for Instagram scraping?
Mobile proxies. Because CGNAT maps one mobile IP to many real subscribers, the platform cannot ban it without collateral damage, making mobile exits the most durable for sustained collection. Residential proxies are a reasonable second choice; datacenter IPs get blocked fastest.
Public Instagram data is very much reachable for marketing research - the profile, post and hashtag endpoints still return clean JSON. The two variables that decide success are the IP (mobile wins) and restraint (public data only, human pace, minimal personal data). Get those right and the endpoint stays open.