Extract and inventory URLs
Parse XML sitemaps or collect the links found in a public webpage.
4 focused tools
Free SEO Tools
Inventory URLs, inspect internal and external links, and create crawler directives for technical SEO reviews. Export clean results that are ready for spreadsheets, audits or migration planning.
Public URLs are fetched server-side so their sitemap or page links can be analyzed.
No sign-up required
All SEO tools
Choose a tool directly or filter the collection by task.
XML Sitemap Generator
Generate a valid sitemap.xml file from a list of URLs instantly
Sitemap URL Extractor
Extract all URLs from any sitemap.xml file with filtering options
Website URL Extractor
Extract all internal and external links from any webpage URL
Robots.txt Generator
Build a valid robots.txt file with crawler rules, AI bot blocking, and sitemap links
No tools match that search. Try a shorter task or format name.
Find the right tool by task
Start with what you need to accomplish, then open the most specific tool.
Control crawler access
Build robots.txt groups, crawler rules and sitemap declarations.
Generate sitemaps
Create a valid sitemap.xml file from a list of URLs.
Quick pick
Which tool should I use?
Choose the result you need and jump straight to the right tool.
Sitemap URL Extractor
Export every URL from a sitemap or nested sitemap index
Extract all URLs from any sitemap.xml file with filtering options
Website URL Extractor
List internal and external links found on one webpage
Extract all internal and external links from any webpage URL
Robots.txt Generator
Create and download crawler rules for a site root
Build a valid robots.txt file with crawler rules, AI bot blocking, and sitemap links
XML Sitemap Generator
Generate a valid sitemap.xml file from a list of URLs
Generate a valid sitemap.xml file from a list of URLs instantly
Good to know
Make the right choice before you start
01
Start with the source that matches your audit
Use Sitemap URL Extractor when the site already publishes XML sitemaps and you need a clean URL inventory. Use Website URL Extractor when you want to review the links exposed by a particular page. The two lists answer different crawl and architecture questions.
02
Treat robots.txt as crawl guidance, not security
A robots.txt file asks compliant crawlers not to request selected paths; it does not protect private content and does not guarantee removal from search. Validate the generated rules and keep confidential resources behind authentication.
Frequently asked questions
Can the sitemap extractor read a sitemap index?
Yes. It can follow referenced child sitemaps and collect their URLs, subject to the limit selected in the tool.
Does robots.txt remove a page from Google?
Not reliably. It controls crawling, not guaranteed indexing or removal. Use the appropriate indexing controls and authentication for the actual goal.
What is the difference between a sitemap and webpage link inventory?
A sitemap lists URLs intentionally supplied by the site owner. A webpage inventory lists links actually present in the HTML of the page being analyzed.
Related tool collections
Web & Network Tools
Free browser-based web and network utilities: find your IP address, view page source, test iframe embedding, generate passwords and UUIDs, convert timestamps, and more.
Developer Tools
A curated set of free online developer tools: JSON formatter, SQL beautifier, CSS minifier, JWT decoder, regex tester, hash generator, and more. Fast, private, no sign-up.