3. Low crawl allocation and slower Googlebot response — High (PV)
Evidence (Search Console Crawl stats, 25 Jul – 21 Oct 2026)
| PV | DH | |
|---|---|---|
| Googlebot requests/day | 237,391 | 570,023 |
| New URLs/day (sitemaps) | ~660 | ~210 |
| Crawl requests per new URL | ~360 | ~2,700 |
| Trend (first → second half) | −33% | −22% |
| Avg response time to Googlebot | 225 ms | 146 ms |
Likely link to #2: PV publishes ~3× more URLs/day with far lower demand per article (median 7 impressions vs 1,844 on DH), so Google appears to render a smaller share of PV pages. Not proven.
Update 12 Nov 2026 (Crawl stats exports of 12 Nov, data to 9 Nov; crawl_stats/2026-11-12/pv-summary/, crawl_stats/2026-11-12/dh-summary/). The response/purpose ratios in these exports cover the whole 90 days (14 Aug – 9 Nov), so they still include the pre-fix weeks; the daily chart is the useful part.
| Period | PV requests/day | PV response (ms) | DH requests/day | DH response (ms) |
|---|---|---|---|---|
| 14 Aug – 13 Sep | 276,066 | 226 | 609,923 | 141 |
| 14 Sep – 13 Oct | 187,748 | 233 | 505,672 | 158 |
| 14 – 24 Oct | 201,305 | 228 | 395,632 | 166 |
| 25 Oct – 9 Nov | 219,366 | 230 | 368,943 | 164 |
-
PV: total crawl is up 9% since the AccessType block, even though ~18,000 wasted 401 requests/day disappeared (#4), so crawling of real pages rose by roughly 20% (about 183k → 219k/day; download volume 21 → 27 GB/day). Response time to Googlebot unchanged at ~230 ms: still worth investigating. PV's 200s are now 98% of its crawl (85% in August). In the 12 Nov OK (200) sample (
crawl_stats/2026-11-12/pv-200/, 993 URLs), 453 (46%) are/amp/story/…pages: nearly half of Googlebot's successful PV fetches go to AMP copies of articles rather than the canonical pages. If AMP is no longer needed, dropping it would free that crawl (see also #14 on DH). -
DH: successful crawling has halved since mid-August and is still falling. Breakdown from the 12 Nov "By response" exports (
crawl_stats/2026-11-12/dh-*), requests/day:Period Total 301 404 Other 4XX 5XX Rest (mostly 200) 14–24 Aug 717,730 173,864 7,196 2,530 2,076 532,064 25 Aug – 3 Sep (outage) 532,226 122,985 39,396 37,367 40,563 291,915 4–30 Sep 529,826 107,377 22,420 36,722 8 363,299 1–22 Oct 472,173 100,267 3,876 32,210 7 335,813 23 Oct – 9 Nov 356,801 87,023 3,966 15 4 265,793 The trigger was a server outage on AMP story URLs, 24 Aug – 1 Sep 2026: ~40,000 5XX/day (peak 63,815 on 29 Aug); 980 of 984 sample 5XX URLs are
/amp/story/…articles. Search Console Host status → Server connectivity shows ~25% of requests failing over the same days ("Acceptable fail rate recently, but high in the past"; screenshot 12 Nov); robots.txt fetch and DNS are fine. Google cuts crawling after server errors and restores it slowly; DH's successful crawl has not recovered (532k → 266k/day) even though 5XX have been ~0 since 4 Sep. Response time also rose (141 → ~165 ms; 301s ~230 ms). (Correction: the earlier note that real-page crawling was "roughly flat since mid-October" counted 301s as real pages; it isn't flat.) Next: confirm with tech what failed on AMP 24 Aug – 1 Sep (the AMP middleware in #14 callsnext()afterres.redirect, a possible cause), and watch whether the crawl recovers now that the #14 AMP 301 fix is live (11 Nov 2026; re-checked live 12 Nov: 4/4 DH articles link AMP with plain slashes, 200, no redirect; old%2FURLs still 301 to the plain URL, as they should while Google re-learns them).
Crawl vs traffic (12 Nov 2026; checks/crawl_stats/crawl_vs_traffic.py, output crawl_vs_traffic_2026-11-12.json). Weekly successful crawl vs Search Console clicks, 17 Aug – 8 Nov 2026, PV as the control:
- DH web search: no effect visible. Weekly clicks stayed in the same band (300–450k) and impressions recovered to 24.9M while successful crawl halved; correlation of weekly crawl with web clicks 0.05, with Google News clicks −0.37 (News clicks rose: 95k → 112k/week).
- DH Discover fell ~40% (2.97M → 1.82M clicks/week), but PV's Discover also fell ~25% (2.74M → 2.04M) without a comparable crawl drop, so most of it looks market/seasonal; the ~15-point gap can't be attributed to crawl from this data (correlation 0.33, weak).
- Why total traffic is the wrong comparison: 96% of DH's crawl is refresh (re-fetching known URLs; Purpose table), so Google can cut re-crawling of old pages without touching traffic. Crawl matters for news through discovery speed of new articles. Right measures: Crawl stats → By purpose → Discovery (daily, export for DH), time from publish to first crawl/index for a sample of new articles (URL Inspection), and share of clicks on articles under 48 h old (
crawl_vs_traffic.py --fresh, a large pull). - Conclusion so far: the AMP outage and the crawl drop have no measurable effect on DH's search traffic yet; whether new articles are discovered more slowly is still unmeasured.
Actions: 🛠️ reduce wasted crawl (#4, #14-type redirects, non-indexable URLs in sitemaps); 🔍 investigate server response time to Googlebot (origin/CDN cache-hit for bots — test responses showed MISS); consider whether low-value print stories need separate handling (sitemap priority / consolidation). File: crawl_stats/.
16 Nov 2026: see #2. Low crawl/render attention per new URL is the likely reason ~76% of PV stories are indexed without text (DH, same code, ~7× more crawl per URL: text indexed). Crawl stats "By file type" exports filed in crawl_stats/2026-11-16/ (JS and JSON ≈ 1% of HTML fetches on both sites; not usable as a rendering measure).
16 Nov 2026 (Crawl stats "By file type", 18 Aug – 14 Nov 2026): PV HTML requests are volatile week to week (282K/day week of 17 Aug → 82K/day week of 21 Sep → 294K/day week of 28 Sep → 155K/day week of 9 Nov), average ~122 KB per HTML fetch, response time steady at ~225–265 ms. PV "Other file type" ran at 22–25K/day until mid-Sep and has been ~1K/day since 21 Sep (no sample URLs in the export, so its source is unknown).
Status (19 Nov 2026): pending re-analysis after #1 + #2 ship on PV. Low crawl per new URL matters mainly because Google then skips rendering PV's body text; once the body is in the server HTML (#1), re-check Crawl stats (By purpose → Discovery, AMP share) and time from publish to first crawl on a sample of new PV articles. Remaining crawl waste (AMP copies) is decided with #26.