SCM includes a Site Crawler that can start from one or more URLs, load each page in a real browser, collect the links it finds, and follow those links based on your crawl settings.
Use it when you want to build a list of pages from a site, collect article URLs from a blog, find useful source pages for research, or discover internal and external links before starting an SEO content workflow.
What the Site Crawler Does
- Starts from one or more crawl URLs
- Loads pages in a browser so rendered links can be detected
- Follows links up to the crawl depth you choose
- Stops after a set number of links, or runs without a link cap
- Lets you allow only matching URLs, such as
/blog/or/articles/ - Lets you ignore unwanted URLs, such as
wp-content, tags, feeds, checkout pages, or login pages - Saves internal links to the main output file
- Saves external links to a separate external-links file
Basic Site Crawler Workflow
- Open Site Crawler in SCM.
- Enter the domain or starting URLs you want to crawl. Add one URL per line.
- Set Crawl depth. A depth of 1 is good for a quick first pass.
- Set Stop after if you want to limit the number of links collected. Use
-1for no limit. - Add Allow links rules if you only want matching URLs, for example
/blog/. - Add Ignore links rules to skip URLs you do not care about.
- Choose how many crawler threads to run at once.
- Choose the filename and save location for the results.
- Run the task and review the URL files SCM creates.
Example: Crawl Only Blog Posts
If you want to collect article URLs from a competitor blog, start with the blog homepage or category page.
In Allow links, add a pattern that matches the article area, such as:
/blog/
In Ignore links, add patterns for pages you do not want, such as:/tag/
/category/
/author/
wp-content
/feed/
This keeps the crawl focused on useful content pages instead of menus, archives, media files, and technical URLs.
Why SCM Uses a Browser-Based Crawler
Many modern websites add links after the page loads. A simple HTML fetch can miss those links. SCM’s Site Crawler loads the page in a browser, scrolls the page, reads the rendered HTML, and then extracts the links from the final page state.
That makes it more useful for sites where menus, article lists, related posts, or pagination links appear after JavaScript has run.
What to Do With the URLs
Once SCM has collected the URLs, you can use them as a starting point for research, content planning, internal link checks, or other SCM workflows.
For example, you can crawl a site to find article URLs, review the pages worth using, then feed selected URLs into your content workflow as research sources or topic targets.
Older Web Crawler Tutorial Video
The video below shows the older Web Crawler workflow. The interface has changed since this video was recorded, but the core idea is the same: start from a URL, let SCM discover links, then use those URLs in your SEO content process.