Crawl budget is real, and it matters for sites in the hundreds of thousands of URLs. Below that threshold it is almost always a misdiagnosis. If you run a few hundred or a few thousand pages and half of them are not indexed, the crawler is not running out of time. It is finding those pages and deciding they are not worth storing.
The usual cause is near-duplication. Faceted filters, session parameters, printer views, paginated archives, and location pages that differ by a single swapped city name all produce documents that a crawler correctly reads as the same document repeated. Google indexes one, treats the rest as alternates, and your coverage report fills with excluded URLs that look mysterious until you diff the HTML.
Start with the Search Console coverage report, but read it against your own inventory rather than in isolation. Export every URL you believe should be indexed, compare it against what actually is, and categorise the gap. In our audits the split is consistently around seventy percent duplication, twenty percent thin content, and ten percent genuine technical blocking.
Google is not rationing attention on your four hundred page site. It is declining to index pages that repeat each other.
Only the last ten percent is fixed with robots directives and canonical tags. The rest requires deciding which pages deserve to exist. Consolidating six weak location pages into one strong one nearly always outperforms keeping all six and hoping the crawler sorts it out.
The uncomfortable conclusion for most sites is that the answer is fewer pages, better made. That is a harder sell internally than a crawl budget fix, which is precisely why crawl budget stays popular as an explanation.
