Using filters to control the crawler

Settings: Ban URLs

Banned URLs

Here you can place URLs or parts of URLs which should not be included into the sitemap. You do not need to add full URLs, you can just add words, filenames, etc. When these are found anywhere within a link, the link will not be crawled. Some examples would be "download.php" (to not add direct download-links), "redirect" (automatic redirect-links), "no-cookie.htm" (error-pages when cookies are not available, generally add all error-pages), etc.

Settings: Drop parts

Droped parts

When the crawler finds one of these entries within the URL (or parameter) it will just drop that part and try to continue with the new URL. You can use this to drop unnecessary parameters (e.g. "sortorder=asc", "start=0", etc.).

Settings: Remove parameters

Removed parameters

This will remove the parameter and it's value if found in the URL. You generally use this for unknown values, like e.g. a Session-ID passed as parameter. You could add "sessionid", "sid", etc. to the table here, based on the way your site passes / creates a session-id. Session-IDs should NOT be included within a sitemap-file.



© SOFTplus Entwicklungen GmbH · Sitemap · Privacy policy · Terms & conditions · Contact me · About