A four-person practice in San Francisco does not have a large website. It has an old one: roughly a hundred pages, of which perhaps forty describe work the firm stopped doing, prices that changed twice, or a service line invented by an agency nobody remembers hiring.
Almost everything written about crawling assumes the opposite problem — a catalogue growing faster than anyone can index it. That advice does not transfer. When the site is small and half of it is wrong, adding pages is not the fix, and submitting all hundred of them is a way of asking Google to look harder at material you would rather it forgot.
A hundred pages, and forty of them out of date
Look at what a typical professional practice here has actually accumulated. Six or seven pages the founder wrote. A dozen more added as the firm took on new work. Then a stretch of two or three years when an agency was on retainer and produced neighborhood pages, comparison posts and a blog that stopped abruptly mid-sentence in some quarter nobody can name. Then five years of small edits by whoever had access.
None of that is unusual. What makes it consequential in this city is who reads it. A buyer evaluating your firm will find the abandoned service page as readily as the current one, and will not treat it as a historical artifact. They will treat it as a claim you are making today — and if two of your pages disagree about what something costs, they will assume the cheaper one and open the conversation there, or, more often, quietly decide you are disorganized and close the tab.
- Retired services never get deleted. The page stays up because taking it down felt like a decision, and nobody wanted to make it on a Friday.
- Prices age silently. A rate published four years ago still reads as current. Nothing on the page announces when it was written.
- Agency output outlives the agency. Thin neighborhood pages and filler posts remain in the sitemap long after the retainer ended.
- Staff pages become archaeology. Half the people pictured have left. In a city with this much movement, that is the fastest-aging content on the site.
Uploading a file is not the same as being findable
Between a page existing and a page appearing in results there is a sequence, and each step can fail without producing any signal a business owner would notice.
First the address has to be discovered — reached through a link, listed in a sitemap, or submitted directly. Then a crawler has to spend a visit on it. Then the content has to be judged worth storing. Only after all three does the question of ranking arise at all. Sites fail at the first two steps far more often than owners suspect, and a page stalled at step one produces exactly the same experience as a page that ranks badly: silence.
The distinction matters because the remedies are unrelated, and the two cases look identical from the outside.
Never fetched
No bot has arrived. The address is unknown, unreachable, or reachable only through a path nothing links to.
- Missing from the sitemap
- Orphaned: no internal link points at it
- Blocked in robots.txt by an old rule
Fetched and set aside
A bot came, read it, and declined to keep it. Discovery work does nothing here; the page itself is the problem.
- Near-identical to another page you own
- Too thin to be worth storing
- Superseded by a newer version you also published
Treating the second case as though it were the first — resubmitting weekly and waiting — is the most common wasted effort in this whole area, and the per-address visit log is what separates them: either a bot arrived or it did not.
What consumes the attention your site is allotted
Every site receives a rough allowance of crawler attention, proportional to how important and how responsive the domain appears. A hundred-page site has a comfortable allowance in principle. In practice it can spend nearly all of it on addresses that should not exist.
| What eats the visits | How it usually arises | What it costs you |
|---|---|---|
| Chained redirects | Three site rebuilds, each pointing at the last | Several fetches to reach one document |
| Parameter variants | Filters, tracking tags, print versions | The same text fetched a dozen ways |
| Templated location pages | An agency generated one per neighborhood | Attention spent on pages with no reader |
| Slow responses | An oversized theme on shared hosting | Fewer fetches per visit, quietly |
| Dead internal links | Pages removed without cleaning the menu | Errors logged instead of content read |
| Duplicate service pages | Two versions written years apart | Both fetched; the wrong one kept |
The last row is the one that costs small practices the most, and it is worth being blunt about why. When two pages on your domain describe the same service, you are not doubling your chances. You are asking a machine to choose between them on evidence you did not curate, and it frequently chooses the older page, because the older page has been linked to for longer.
Establish what exists before you submit anything
The instinct on opening any indexing tool is to submit everything. Resist it for one afternoon. An inventory pass on a hundred-page site takes about three hours and changes what you would have submitted.
Pull three lists and compare them. What the sitemap claims exists. What the server actually serves. What your analytics account reports as having received impressions in the past year. The gaps between those three lists are the whole story: addresses in the sitemap that no longer resolve, live pages absent from the sitemap entirely, and — the largest group, usually — pages that exist, resolve, are known, and have not been shown to a human being in twelve months.
The classification
Every page lands in exactly one of four categories, and the decision takes under a minute each once you stop deliberating.
- Current and correct: leave it
- Right subject, wrong details: rewrite
- Superseded by another page: redirect
- Serves nobody: remove it
The awkward cases
Two categories cause every argument in the room, and both have a defensible answer.
- Old posts with real inbound links: rewrite, keep the address
- Pages ranking for work you no longer do: redirect to what you do now
- Staff profiles for departed people: remove, do not redirect
- Neighborhood pages with no content: consolidate into one honest page
Do the counting in a spreadsheet, not in your head — the reporting and export tools will hand you the third list as CSV up to ten thousand rows, which for a site this size means the whole thing in one file. The number that matters at the end is not how many pages you have; it is how many you are prepared to have read closely by somebody who compares vendors for a living.
When the outdated page outranks the current one
This is the failure mode worth planning around, and it is not hypothetical. A practice rewrites its services section, publishes a clean new page, and leaves the old one in place because deleting things feels risky. Six months later the old page still holds the position. The new one sits somewhere on the third page of results, and the firm concludes that the rewrite failed.
It did not fail. It was outvoted. The old address has years of accumulated links, an older publication date on record, and a title that matches the query exactly because it was written when that was the fashionable phrasing. None of that describes quality; all of it describes tenure. Meanwhile the visitor arriving on the old page reads a rate that expired two budget cycles ago.
- Redirect, do not merely unpublish. A removed page loses its accumulated value. A redirected one hands that value to the page you want people to read.
- One address per service, permanently. If the rate changes, edit the page. Publishing a second version is how this situation is created in the first place.
- Date the pricing visibly. An effective-from line costs nothing and tells a technical reader that somebody is maintaining this document.
- Check what people actually land on. Sort the page report by impressions and look for addresses you thought were retired. They will be there.
The sitemap is how discovery is directed
A sitemap is treated by most owners as a formality generated by a plugin and never opened. Read properly it is a statement of intent: this is the set of addresses I want examined. Which makes the stale entries in it an active problem rather than clutter — every retired page listed there is a request for attention you did not mean to make.
Submitting a sitemap for parsing
Upload the file or point at its address; nested index files are followed automatically.
- Three levels of recursion. An index file pointing at index files pointing at sitemaps is followed all the way down, which covers every structure a small site will ever have.
- Up to 1,000 sitemaps in one job. Far beyond what a professional practice needs, and the reason a large multi-location structure can be handled in a single pass.
- Two jobs at once, twenty waiting. Parsing runs two jobs concurrently with a queue of up to twenty behind them, so several domains can be lined up in one sitting.
For a hundred-page site the practical use is verification rather than volume. Feed in the sitemap, look at what comes back, and compare that list against the four piles from the audit. Anything appearing in the parsed output that you classified as remove or redirect is a fault in the sitemap, and fixing it there is faster than fixing it anywhere else. The sitemap parsing and URL tracking screens exist for exactly that reconciliation.
A thousand a day, ten thousand to a batch
Direct submission is the second instrument, and its limits shape how the work is scheduled. An account can put forward a thousand addresses per day, and a single batch can carry up to ten thousand for processing across subsequent days. Delivery goes out over IndexNow, which notifies participating crawlers — Google's and Bing's among them — rather than waiting for them to arrive on their own schedule.
Batches, budgets and the per-URL log
What you submitted, what was visited, what failed, and when each of those happened.
- Per-address history. Each URL carries its own record: which bot arrived, at what time, with what status, and the detail behind any failure.
- Live counters on a batch. Submitted, discovered and failed update as the batch works through, so a broken run is visible within minutes rather than at the end.
- Failures are specific. A refusal for an address blocked in robots.txt reads differently from one that timed out, and the two need different fixes.
Read the status of a batch as a report on your server, not on your rankings. The counters answer three questions: did the address resolve, did a bot come, and did anything break. Whether the page was kept is not among them, and never was. That answer arrives later, in the analytics record, as impressions the page did or did not begin collecting.
| What the batch shows | What it means | What it does not mean |
|---|---|---|
| Submitted, no visit yet | The address is queued and waiting | That anything is wrong |
| Visited, status 200 | A bot fetched the page successfully | That the page was stored |
| Visited, status 404 | You submitted an address that is gone | That the sitemap is fine |
| Visited, redirected | The bot was sent elsewhere | That the destination was indexed |
| Failed, blocked | A robots rule refused the request | That the page is low quality |
| Failed, timed out | The server did not answer in time | That resubmitting will help |
Questions from the first cleanup
We deleted the outdated pricing page and traffic dropped. What now?
Put a redirect from the old address to the current page, if it still resolves at all. A deleted page takes its accumulated value with it, which is why removal is the last option rather than the first. Where the old address has genuinely gone and cannot be restored, the replacement will need links pointed at it deliberately, and that takes longer than the deletion did.
Should we submit all hundred pages, or only the ones we changed?
Only what changed, plus anything genuinely new. A site this size sits far inside the daily allowance, so the constraint is not capacity — it is that a submission is a request to look, and you should only request looks at pages you have already decided are correct. Submitting the untouched remainder achieves nothing and re-exposes whatever you have not finished fixing.
How long after submission should a page appear?
A visit from a bot often comes within hours, and the log will show it. Whether the page is stored and starts collecting impressions is a separate question with a much wider range — days to several weeks, and for thin or duplicated pages, never. Watch the analytics record rather than the submission log for that part.
An agency built us forty neighborhood pages. Keep or kill?
Check the page report first. If they have collected no impressions in a year, they are consuming crawl attention and contributing nothing; consolidate them into one page that honestly describes the areas you serve and redirect the rest to it. If two or three of them do collect traffic, keep those, improve them, and treat the remaining thirty-seven as the cleanup.
Our old page still ranks above the new one months later. Is something broken?
Probably not broken — simply left in place. Two pages covering one subject compete, and tenure usually wins. Redirect the old address to the new one, make sure no menu or footer still links to the old, and expect a few weeks before the position transfers. Until you do that, both pages will keep splitting the same signals.
Running the numbers on a hundred-page practice site
Take the case from the opening. A hundred addresses. Sixty current, forty in some state of decay: fifteen describing retired services, ten with prices that have changed twice, fifteen produced by a former agency and never read by anybody.
The submission side of that is trivial. A hundred addresses is a tenth of one day's allowance, and even a full pass across three domains would not reach the limit. What is not trivial is the forty decisions, at three to five minutes each, which is one working day. That day is the entire project. Everything the tooling does afterward — parsing, submitting, logging, reporting — takes an afternoon at most, and none of it improves anything you have not already decided.
That ratio is the argument. On a small professional site the tooling is never the constraint; the constraint is that somebody has to decide, service by service, what the firm currently sells and for how much. Those decisions cannot be automated, and in a market where clients close, relocate or disappear inside a year and a half, they need revisiting more often than anyone plans for. Our written material covers the reading side of that cycle in more detail, and the engagement outlines describe the version where the audit is run for you.
If it helps to see the audit and the submission side in one place rather than in three tools, connect a verified property and open the panel. The sitemap parser, the per-URL log and the analytics that eventually confirm the result all read from the same account, alongside the campaign automation tiers at $149 and $500 per month per domain for firms that would rather delegate the recurring part. Start with the count, though. Everything after it is cheaper than it looks, and everything before it is where the real work sits.