← Issues index

3. Low crawl allocation and slower Googlebot response — High (PV)

Evidence (Search Console Crawl stats, 25 Jul – 21 Oct 2026)

PV DH
Googlebot requests/day 237,391 570,023
New URLs/day (sitemaps) ~660 ~210
Crawl requests per new URL ~360 ~2,700
Trend (first → second half) −33% −22%
Avg response time to Googlebot 225 ms 146 ms

Likely link to #2: PV publishes ~3× more URLs/day with far lower demand per article (median 7 impressions vs 1,844 on DH), so Google appears to render a smaller share of PV pages. Not proven.

Update 12 Nov 2026 (Crawl stats exports of 12 Nov, data to 9 Nov; crawl_stats/2026-11-12/pv-summary/, crawl_stats/2026-11-12/dh-summary/). The response/purpose ratios in these exports cover the whole 90 days (14 Aug – 9 Nov), so they still include the pre-fix weeks; the daily chart is the useful part.

Period PV requests/day PV response (ms) DH requests/day DH response (ms)
14 Aug – 13 Sep 276,066 226 609,923 141
14 Sep – 13 Oct 187,748 233 505,672 158
14 – 24 Oct 201,305 228 395,632 166
25 Oct – 9 Nov 219,366 230 368,943 164

Crawl vs traffic (12 Nov 2026; checks/crawl_stats/crawl_vs_traffic.py, output crawl_vs_traffic_2026-11-12.json). Weekly successful crawl vs Search Console clicks, 17 Aug – 8 Nov 2026, PV as the control:

Actions: 🛠️ reduce wasted crawl (#4, #14-type redirects, non-indexable URLs in sitemaps); 🔍 investigate server response time to Googlebot (origin/CDN cache-hit for bots — test responses showed MISS); consider whether low-value print stories need separate handling (sitemap priority / consolidation). File: crawl_stats/.

16 Nov 2026: see #2. Low crawl/render attention per new URL is the likely reason ~76% of PV stories are indexed without text (DH, same code, ~7× more crawl per URL: text indexed). Crawl stats "By file type" exports filed in crawl_stats/2026-11-16/ (JS and JSON ≈ 1% of HTML fetches on both sites; not usable as a rendering measure).

16 Nov 2026 (Crawl stats "By file type", 18 Aug – 14 Nov 2026): PV HTML requests are volatile week to week (282K/day week of 17 Aug → 82K/day week of 21 Sep → 294K/day week of 28 Sep → 155K/day week of 9 Nov), average ~122 KB per HTML fetch, response time steady at ~225–265 ms. PV "Other file type" ran at 22–25K/day until mid-Sep and has been ~1K/day since 21 Sep (no sample URLs in the export, so its source is unknown).

Status (19 Nov 2026): pending re-analysis after #1 + #2 ship on PV. Low crawl per new URL matters mainly because Google then skips rendering PV's body text; once the body is in the server HTML (#1), re-check Crawl stats (By purpose → Discovery, AMP share) and time from publish to first crawl on a sample of new PV articles. Remaining crawl waste (AMP copies) is decided with #26.