Bug #89287

Make linkvalidator crawling polite

Added by Sybille Peters 3 months ago.

Status:
New
Priority:
Should have
Assignee:
-
Category:
Linkvalidator
Target version:
-
Start date:
2019-09-26
Due date:
% Done:

0%

TYPO3 Version:
10
PHP Version:
Tags:
Complexity:
Is Regression:
Sprint Focus:

Description

Currently, linkvalidator does not apply common practice for being "nice" / "polite" when crawling other websites:

  • It should be possible to see what is crawling your site. It is usually standard to add a URL and contact information to the User-Agent or referrer, e.g.
"Mozilla/5.0 (compatible; MetaJobBot; http://www.metajob.de/crawler)" 

"https://www.google.de/" "Mozilla/5.0 (iPhone; CPU iPhone OS 12_4_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.1.2 Mobile/15E148 Safari/604.1" 
  • the crawler should respect the robots.txt
  • the crawler should wait between consecutive requests (Crawl-Delay). A good minimum value is e.g. 1-5 second for requests done on same domain

While linkvalidator is not a full web crawler which downloads the entire pages on the site and it currently uses HEAD by default (and not GET) which does not download the entire page - so this is not as dramatic.

But preferably, I think we should follow these recommendations as well.

Possible solutions

Not just for reducing load on other sites but also on the TYPO3 site, some changes might be made to the crawling process:

  • when crawling external URLs, do not check right away but defer to a separate task which will handle the crawling of external URLs (with reasonable delays in between)
  • do not keep crawling over and over but regularly only recrawl content which was recently modified
  • optional: delegate the checking of external URLs, e.g. use a URLchecking service

This may make it necessary to make more things asynchronous and confine the link checking only to the scheduler.

Resources

About politeness of web crawlers:

URL checking site:

Associated revisions

Revision 2507a32f (diff)
Added by Sybille Peters 28 days ago

[FEATURE] Add additional configuration for external URLs

Additional configuration is added to customize settings for checking
external URLs.

Resolves: #86918
Related: #89287
Releases: master
Change-Id: I1ebfb31fe7760ad5b7c99db3999794c1e363cd17
Reviewed-on: https://review.typo3.org/c/Packages/TYPO3.CMS/+/61801
Tested-by: TYPO3com <>
Tested-by: Chris Müller <>
Tested-by: Benni Mack <>
Reviewed-by: Chris Müller <>
Reviewed-by: Benni Mack <>

Also available in: Atom PDF