Domain scraping for expired-name research means collecting links or candidates from appropriate sources, extracting the domains and checking their current position. A broken link or missing website is a lead for investigation, not proof that a domain is available to register.
Keep discovery, registration checks and purchase decisions separate. This makes the resulting list easier to verify and avoids spending on names simply because a crawler returned them.
Choose focused sources and a limited first run
Define the subject and intended use before crawling. Select relevant sites or sources and start with a small collection so you can inspect the quality of the results. Record the source URL and collection date.
Domain Hunter Gatherer’s official guide describes keyword hunts and website crawls for candidate discovery. DomainNexus and Domain Hunter Gatherer share a publishing family. Confirm the current tool and plan support your intended workflow rather than assuming every discovery service is a crawler.
Read source access terms and use a crawl rate appropriate to the site. The Robots Exclusion Protocol defines robots.txt controls for automated clients. Respect relevant rules and limits; a public link is not a reason to overwhelm a website.
Do not confuse search-engine queries with website crawling
Use supported data routes when a search service is involved. Google’s machine-generated traffic policy addresses automated queries to Google without express permission. Running more aggressive queries is not a substitute for an appropriate discovery source.
Check the tool’s current documentation for its inputs and source requirements. Avoid copying an old configuration whose source no longer permits or supports that use.
Clean the candidate list
Keep the discovered URL and extracted domain in separate fields. Remove duplicate domains, accidental paths and malformed inputs before using paid lookups. Retain useful old destination paths for later history research.
| Field | Reason to keep it |
|---|---|
| Source and discovered URL | Explain why the candidate entered the list. |
| Domain and collection date | Support consistent current checks. |
| Registration or listing position | Distinguish availability from discovery. |
| History and link observations | Record evidence behind the buying decision. |
The bulk-search workflow helps organise the next checks and leave errors marked unconfirmed.
Verify current registration and sale routes
Check candidates through a supporting registrar or verified auction listing. ICANN’s typical gTLD lifecycle explains why expiry does not immediately produce a new registration opportunity. Country-code registries may follow different rules.
RDAP or DNS gaps are not availability confirmations. Record the actual purchase route, mandatory charges, renewal price and delivery conditions before treating a candidate as purchasable.
Inspect the shortlist manually
Use metrics to prioritise suitable names, then review archived pages and significant current backlink sources. Check topic fit, naming concerns and whether important links remain present. The metrics guide supplies those questions.
Do not assume an auction win requires registering the already held name again through an unrelated service. Follow the acquisition provider’s documented delivery route, then confirm control in the receiving registrar account.
The buying checklist connects discovery to a researched commitment. A useful crawl leaves you with a smaller set of explainable candidates and clear next checks, rather than an unverified promise of valuable domains.