Project

General

Profile

Actions

Feature #85679

closed

Epic #83559: SEO enhancements in Core

Add robot.txt functionality

Added by Richard Haeser over 5 years ago. Updated over 4 years ago.

Status:
Closed
Priority:
Should have
Assignee:
-
Category:
SEO
Target version:
-
Start date:
2018-07-29
Due date:
% Done:

0%

Estimated time:
PHP Version:
Tags:
Complexity:
Sprint Focus:

Description

TYPO3 should deliver an out-of-the-box robots.txt to prevent certain directories (like eg /typo3) to be indexed. Besides that, it should be possible by integrators to block certain parts of the website to be crawled to prevent using crawl budget while it is not needed to be in the index.

Actions #1

Updated by Richard Haeser over 5 years ago

  • Category set to SEO
Actions #2

Updated by Richard Haeser over 5 years ago

  • Status changed from New to Closed

It is possible to add your robots.txt by a static route. See https://docs.typo3.org/typo3cms/CoreApiReference/ApiOverview/SiteHandling/StaticRoutes.html for more information.

Actions #3

Updated by Richard Haeser over 5 years ago

  • Status changed from Closed to New
Actions #4

Updated by Richard Haeser over 5 years ago

It still would be nice if TYPO3 can render a (configurable) robots.txt

Actions #5

Updated by Andreas Kienast over 5 years ago

What the robots.txt could do:

  • Automatically add the sitemap.xml, if configured
  • Add Disallow: foo if page should not get indexed (discussed with Richard, potentially dangerous as Google doesn't "forget" wrong configs so fast)
  • Add typo3temp/var/ if in document root
  • Exclude typo3/
  • Exclude typo3conf/ext/*/Resources/Private/
Actions #6

Updated by Benni Mack over 4 years ago

  • Status changed from New to Closed
Actions #7

Updated by Richard Haeser over 4 years ago

A little note why we closed it:
It is best practise to keep your robots.txt as minimal as possible. So that is why we will not include anything in core currently.

For more information see https://yoast.com/wordpress-robots-txt-example as well.

Actions

Also available in: Atom PDF