Feature #85679
closed
Epic #83559: SEO enhancements in Core
Add robot.txt functionality
0%
Description
TYPO3 should deliver an out-of-the-box robots.txt to prevent certain directories (like eg /typo3) to be indexed. Besides that, it should be possible by integrators to block certain parts of the website to be crawled to prevent using crawl budget while it is not needed to be in the index.
Updated by Richard Haeser about 6 years ago
- Status changed from New to Closed
It is possible to add your robots.txt by a static route. See https://docs.typo3.org/typo3cms/CoreApiReference/ApiOverview/SiteHandling/StaticRoutes.html for more information.
Updated by Richard Haeser almost 6 years ago
It still would be nice if TYPO3 can render a (configurable) robots.txt
Updated by Andreas Kienast almost 6 years ago
What the robots.txt could do:
- Automatically add the sitemap.xml, if configured
Add(discussed with Richard, potentially dangerous as Google doesn't "forget" wrong configs so fast)Disallow: fooif page should not get indexed- Add
typo3temp/var/if in document root - Exclude
typo3/ - Exclude
typo3conf/ext/*/Resources/Private/
Updated by
Updated by Richard Haeser about 5 years ago
A little note why we closed it:
It is best practise to keep your robots.txt as minimal as possible. So that is why we will not include anything in core currently.
For more information see https://yoast.com/wordpress-robots-txt-example as well.