Bug #38033
Exclude url from sitemap that are in Disallow: lines in robots.txt
| Status: | New | Start date: | 2012-06-14 | |
|---|---|---|---|---|
| Priority: | Should have | Due date: | ||
| Assignee: | Saverio Vigni | % Done: | 0% |
|
| Category: | Sitemap | Spent time: | - | |
| Target version: | - | |||
| TYPO3 Version: | 4.5 | PHP Version: | ||
| Votes: | 0 |
Description
I have this issue in which my sitemap is filled with virtual urls generated by an extension i made that are excluded from robots.txt so iI wrote a little patch to exclude from the sitemap all the urls or part of them, that are disallowed in robots.txt
Please test the patch attached, it's simply a function added to "class.tools.php" that check if the passed url is in any Disallow line of the robots.txt, then this function is called both by the indexer hook and by the typolink hook