Extract and inventory URLs
Parse XML sitemaps, collect links from one webpage or scan reachable pages across a website.
6 focused tools
Free SEO Tools
Inventory URLs, inspect internal and external links, and create crawler directives for technical SEO reviews. Export clean results that are ready for spreadsheets, audits or migration planning.
Public URLs are fetched server-side so their sitemap or page links can be analyzed.
No sign-up required
All SEO tools
Choose a tool directly or filter the collection by task.
Meta Tags Checker
Check, edit, and preview website meta tags for social media
XML Sitemap Generator
Generate a valid sitemap.xml file from a list of URLs instantly
Sitemap URL Extractor
Extract all URLs from any sitemap.xml file with filtering options
Website URL Extractor
Scan a website to find reachable pages, status codes and crawl depth
Webpage Link Extractor
Extract links, images, scripts and other resources from one webpage
Robots.txt Generator
Build a valid robots.txt file with crawler rules, AI bot blocking, and sitemap links
No tools match that search. Try a shorter task or format name.
Find the right tool by task
Start with what you need to accomplish, then open the most specific tool.
Control crawler access
Build robots.txt groups, crawler rules and sitemap declarations.
Generate sitemaps
Create a valid sitemap.xml file from a list of URLs.
Analyze Meta Tags
Check, edit, and preview website meta tags for SEO and social sharing.
Quick pick
Which tool should I use?
Choose the result you need and jump straight to the right tool.
Sitemap URL Extractor
Export every URL from a sitemap or nested sitemap index
Extract all URLs from any sitemap.xml file with filtering options
Webpage Link Extractor
List internal, external and resource URLs found on one webpage
Extract links, images, scripts and other resources from one webpage
Website URL Extractor
Scan a website for reachable pages, status codes and crawl depth
Scan a website to find reachable pages, status codes and crawl depth
Robots.txt Generator
Create and download crawler rules for a site root
Build a valid robots.txt file with crawler rules, AI bot blocking, and sitemap links
XML Sitemap Generator
Generate a valid sitemap.xml file from a list of URLs
Generate a valid sitemap.xml file from a list of URLs instantly
Meta Tags Checker
Check, edit, and preview website meta tags for social media
Check, edit, and preview website meta tags for social media
Good to know
Make the right choice before you start
01
Start with the source that matches your audit
Use Sitemap URL Extractor for URLs declared in XML, Webpage Link Extractor for links and resources on one page, or Website URL Extractor to discover pages reachable through a live site crawl. Each list answers a different crawl and architecture question.
02
Treat robots.txt as crawl guidance, not security
A robots.txt file asks compliant crawlers not to request selected paths; it does not protect private content and does not guarantee removal from search. Validate the generated rules and keep confidential resources behind authentication.
Frequently asked questions
Can the sitemap extractor read a sitemap index?
Yes. It can follow referenced child sitemaps and collect their URLs, subject to the limit selected in the tool.
Does robots.txt remove a page from Google?
Not reliably. It controls crawling, not guaranteed indexing or removal. Use the appropriate indexing controls and authentication for the actual goal.
What is the difference between a sitemap and webpage link inventory?
A sitemap lists URLs intentionally supplied by the site owner. A webpage inventory lists links actually present in the HTML of the page being analyzed.
Related tool collections
Web & Network Tools
Free browser-based web and network utilities: find your IP address, view page source, test iframe embedding, generate passwords and UUIDs, convert timestamps, and more.
Developer Tools
A curated set of free online developer tools: JSON formatter, SQL beautifier, CSS minifier, JWT decoder, regex tester, hash generator, and more. Fast, private, no sign-up.