Actions
Bug #96746
closed
Make it possible to warmup sitemaps (e.g. sitemap.xml) without generating error messages in log
Status:
Closed
Priority:
Should have
Assignee:
-
Category:
SEO
Target version:
-
Start date:
2022-02-04
Due date:
% Done:
0%
Estimated time:
TYPO3 Version:
12
PHP Version:
Tags:
sitemap, cache warmup
Complexity:
Is Regression:
Sprint Focus:
Description
I regularly get error messages because of timeout for sitemap.xml and sitemap_en.xml etc. in the TYPO3 logs
I already set the timeout to 20s (which is absurdly high). I regularly look at the error messages in the logs and solve them. I can filter out this message, but better would be to be able to warmup the sitemaps after flushing of cache without error messages.
Fri, 04 Feb 2022 10:45:28 +0100 [CRITICAL] request="c5a457a50c3c9" component="TYPO3.CMS.Core.Error.ProductionExceptionHandler": Core: Exception handler (WEB): Uncaught TYPO3 Exception: cURL error 28: Operation timed out after
20001 milliseconds with 0 out of -1 bytes received (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) | GuzzleHttp\Exception\ConnectException thrown in file /var/www/mysite/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php in line 200. Requested URL: https://mysite/sitemap_en.xml - {"TYPO3_MODE":"FE","exception":"GuzzleHttp\\Exception\\ConnectException: cURL error 28: Operation timed out after 20001 milliseconds with 0 out of -1 bytes received (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) in /var/www/mysite/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php:200
Stack trace:
#0 /var/www/mysite/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php(155): GuzzleHttp\\Handler\\CurlFactory::createRejection(Object(GuzzleHttp\\Handler\\EasyHandle), Array)
#1 /var/www/mysite/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php(105): GuzzleHttp\\Handler\\CurlFactory::finishError(Object(GuzzleHttp\\Handler\\CurlHandler), Object(GuzzleHttp\\Handler\\EasyHandle), Object(GuzzleHttp\\Handle
r\\CurlFactory))
#2 /var/www/mysite/vendor/guzzlehttp/guzzle/src/Handler/CurlHandler.php(43): GuzzleHttp\\Handler\\CurlFactory::finish(Object(GuzzleHttp\\Handler\\CurlHandler), Object(GuzzleHttp\\Handler\\EasyHandle), Object(GuzzleHttp\\Handler\\CurlFactory))
#3 /var/www/mysite/vendor/guzzlehttp/guzzle/src/Handler/Proxy.php(28): GuzzleHttp\\Handler\\CurlHandler->__invoke(Object(GuzzleHttp\\Psr7\\Request), Array)
#4 /var/www/mysite/vendor/guzzlehttp/guzzle/src/Handler/Proxy.php(51): GuzzleHttp\\Handler\\Proxy::GuzzleHttp\\Handler\\{closure}(Object(GuzzleHttp\\Psr7\\Request), Array)
#5 /var/www/mysite/vendor/guzzlehttp/guzzle/src/PrepareBodyMiddleware.php(37): GuzzleHttp\\Handler\\Proxy::GuzzleHttp\\Handler\\{closure}(Object(GuzzleHttp\\Psr7\\Request), Array)
#6 /var/www/mysite/vendor/guzzlehttp/guzzle/src/Middleware.php(29): GuzzleHttp\\PrepareBodyMiddleware->__invoke(Object(GuzzleHttp\\Psr7\\Request), Array)
#7 /var/www/mysite/vendor/guzzlehttp/guzzle/src/RedirectMiddleware.php(70): GuzzleHttp\\Middleware::GuzzleHttp\\{closure}(Object(GuzzleHttp\\Psr7\\Request), Array)
#8 /var/www/mysite/vendor/guzzlehttp/guzzle/src/RedirectMiddleware.php(107): GuzzleHttp\\RedirectMiddleware->__invoke(Object(GuzzleHttp\\Psr7\\Request), Array)
#9 /var/www/mysite/vendor/guzzlehttp/guzzle/src/RedirectMiddleware.php(72): GuzzleHttp\\RedirectMiddleware->checkRedirect(Object(GuzzleHttp\\Psr7\\Request), Array, Object(GuzzleHttp\\Psr7\\Response))
#10 /var/www/mysite/vendor/guzzlehttp/promises/src/FulfilledPromise.php(41): GuzzleHttp\\RedirectMiddleware->GuzzleHttp\\{closure}(Object(GuzzleHttp\\Psr7\\Response))
#11 /var/www/mysite/vendor/guzzlehttp/promises/src/TaskQueue.php(48): GuzzleHttp\\Promise\\FulfilledPromise::GuzzleHttp\\Promise\\{closure}()
#12 /var/www/mysite/vendor/guzzlehttp/promises/src/Promise.php(248): GuzzleHttp\\Promise\\TaskQueue->run(true)
#13 /var/www/mysite/vendor/guzzlehttp/promises/src/Promise.php(224): GuzzleHttp\\Promise\\Promise->invokeWaitFn()
#14 /var/www/mysite/vendor/guzzlehttp/promises/src/Promise.php(269): GuzzleHttp\\Promise\\Promise->waitIfPending()
#15 /var/www/mysite/vendor/guzzlehttp/promises/src/Promise.php(226): GuzzleHttp\\Promise\\Promise->invokeWaitList()
#16 /var/www/mysite/vendor/guzzlehttp/promises/src/Promise.php(62): GuzzleHttp\\Promise\\Promise->waitIfPending()
#17 /var/www/mysite/vendor/guzzlehttp/guzzle/src/Client.php(182): GuzzleHttp\\Promise\\Promise->wait()
#18 /var/www/mysite/htdocs/typo3/sysext/core/Classes/Http/RequestFactory.php(55): GuzzleHttp\\Client->request('GET', 'https://mysite/...', Array)
#19 /var/www/mysite/htdocs/typo3/sysext/frontend/Classes/Middleware/StaticRouteResolver.php(131): TYPO3\\CMS\\Core\\Http\\RequestFactory->request('https://mysite/...')
#20 /var/www/mysite/htdocs/typo3/sysext/frontend/Classes/Middleware/StaticRouteResolver.php(186): TYPO3\\CMS\\Frontend\\Middleware\\StaticRouteResolver->getFromUri('https://mysite/...')
#21 /var/www/mysite/htdocs/typo3/sysext/frontend/Classes/Middleware/StaticRouteResolver.php(72): TYPO3\\CMS\\Frontend\\Middleware\\StaticRouteResolver->resolveByType(Object(TYPO3\\CMS\\Core\\Http\\ServerRequest), Object(TYPO3\\CMS\\Core\\Site\\Entity\\Site), 'uri', Array)
#22 /var/www/mysite/htdocs/typo3/sysext/core/Classes/Http/MiddlewareDispatcher.php(172): TYPO3\\CMS\\Frontend\\Middleware\\StaticRouteResolver->process(Object(TYPO3\\CMS\\Core\\Http\\ServerRequest), Object(class@anonymous))
#23 /var/www/mysite/htdocs/typo3/sysext/frontend/Classes/Middleware/SiteBaseRedirectResolver.php(94): class@anonymous->handle(Object(TYPO3\\CMS\\Core\\Http\\ServerRequest))
#24 /var/www/mysite/htdocs/typo3/sysext/core/Classes/Http/MiddlewareDispatcher.php(172): TYPO3\\CMS\\Frontend\\Middleware\\SiteBaseRedirectResolver->process(Object(TYPO3\\CMS\\Core\\Http\\ServerRequest), Object(class@anonymous))
#25 /var/www/mysite/htdocs/typo3conf/ext/secure_downloads/Classes/Middleware/FileDeliveryMiddleware.php(58): class@anonymous->handle(Object(TYPO3\\CMS\\Core\\Http\\ServerRequest))
#26 /var/www/mysite/htdocs/typo3/sysext/core/Classes/Http/MiddlewareDispatcher.php(172): Bitmotion\\SecureDownloads\\Middleware\\FileDeliveryMiddleware->process(Object(TYPO3\\CMS\\Core\\Http\\ServerRequest), Object(class@anonymous))
#27 /var/www/mysite/htdocs/typo3/sysext/adminpanel/Classes/Middleware/AdminPanelInitiator.php(55): class@anonymous->handle(Object(TYPO3\\CMS\\Core\\Http\\ServerRequest))
#28 /var/www/mysite/htdocs/typo3/sysext/core/Classes/Http/MiddlewareDispatcher.php(172): TYPO3\\CMS\\Adminpanel\\Middleware\\AdminPanelInitiator->process(Object(TYPO3\\CMS\\Core\\Http\\ServerRequest), Object(class@anonymous))
#29 /var/www/mysite/htdocs/typo3/sysext/frontend/Classes/Middleware/FrontendUserAuthenticator.php(77): class@anonymous->handle(Object(TYPO3\\CMS\\Core\\Http\\ServerRequest))
#30 /var/www/mysite/htdocs/typo3/sysext/core/Classes/Http/MiddlewareDispatcher.php(172): TYPO3\\CMS\\Frontend\\Middleware\\FrontendUserAuthenticator->process(Object(TYPO3\\CMS\\Core\\Http\\ServerRequest), Object(class@anonymous))
#31 /var/www/mysite/htdocs/typo3/sysext/frontend/Classes/Middleware/BackendUserAuthenticator.php(66): class@anonymous->handle(Object(TYPO3\\CMS\\Core\\Http\\ServerRequest))
#32 /var/www/mysite/htdocs/typo3/sysext/core/Classes/Http/MiddlewareDispatcher.php(172): TYPO3\\CMS\\Frontend\\Middleware\\BackendUserAuthenticator->process(Object(TYPO3\\CMS\\Core\\Http\\ServerRequest), Object(class@anonymous))
#33 /var/www/mysite/htdocs/typo3/sysext/frontend/Classes/Middleware/MaintenanceMode.php(55): class@anonymous->handle(Object(TYPO3\\CMS\\Core\\Http\\ServerRequest))
#34 /var/www/mysite/htdocs/typo3/sysext/core/Classes/Http/MiddlewareDispatcher.php(172): TYPO3\\CMS\\Frontend\\Middleware\\MaintenanceMode->process(Object(TYPO3\\CMS\\Core\\Http\\ServerRequest), Object(class@anonymous))
#35 /var/www/mysite/htdocs/typo3/sysext/frontend/Classes/Middleware/SiteResolver.php(65): class@anonymous->handle(Object(TYPO3\\CMS\\Core\\Http\\ServerRequest))
#36 /var/www/mysite/htdocs/typo3/sysext/core/Classes/Http/MiddlewareDispatcher.php(172): TYPO3\\CMS\\Frontend\\Middleware\\SiteResolver->process(Object(TYPO3\\CMS\\Core\\Http\\ServerRequest), Object(class@anonymous))
#37 /var/www/mysite/htdocs/typo3/sysext/frontend/Classes/Middleware/EidHandler.php(64): class@anonymous->handle(Object(TYPO3\\CMS\\Core\\Http\\ServerRequest))
#38 /var/www/mysite/htdocs/typo3/sysext/core/Classes/Http/MiddlewareDispatcher.php(172): TYPO3\\CMS\\Frontend\\Middleware\\EidHandler->process(Object(TYPO3\\CMS\\Core\\Http\\ServerRequest), Object(class@anonymous))
#39 /var/www/mysite/htdocs/typo3/sysext/core/Classes/Middleware/NormalizedParamsAttribute.php(45): class@anonymous->handle(Object(TYPO3\\CMS\\Core\\Http\\ServerRequest))
#40 /var/www/mysite/htdocs/typo3/sysext/core/Classes/Http/MiddlewareDispatcher.php(172): TYPO3\\CMS\\Core\\Middleware\
ormalizedParamsAttribute->process(Object(TYPO3\\CMS\\Core\\Http\\ServerRequest), Object(class@anonymous))
#41 /var/www/mysite/htdocs/typo3/sysext/frontend/Classes/Middleware/TimeTrackerInitialization.php(58): class@anonymous->handle(Object(TYPO3\\CMS\\Core\\Http\\ServerRequest))
#42 /var/www/mysite/htdocs/typo3/sysext/core/Classes/Http/MiddlewareDispatcher.php(172): TYPO3\\CMS\\Frontend\\Middleware\\TimeTrackerInitialization->process(Object(TYPO3\\CMS\\Core\\Http\\ServerRequest), Object(class@anonymous))
#43 /var/www/mysite/htdocs/typo3/sysext/core/Classes/Http/MiddlewareDispatcher.php(78): class@anonymous->handle(Object(TYPO3\\CMS\\Core\\Http\\ServerRequest))
#44 /var/www/mysite/htdocs/typo3/sysext/core/Classes/Http/AbstractApplication.php(85): TYPO3\\CMS\\Core\\Http\\MiddlewareDispatcher->handle(Object(TYPO3\\CMS\\Core\\Http\\ServerRequest))
#45 /var/www/mysite/htdocs/typo3/sysext/frontend/Classes/Http/Application.php(69): TYPO3\\CMS\\Core\\Http\\AbstractApplication->handle(Object(TYPO3\\CMS\\Core\\Http\\ServerRequest))
#46 /var/www/mysite/htdocs/typo3/sysext/core/Classes/Http/AbstractApplication.php(97): TYPO3\\CMS\\Frontend\\Http\\Application->handle(Object(TYPO3\\CMS\\Core\\Http\\ServerRequest))
#47 /var/www/mysite/htdocs/index.php(25): TYPO3\\CMS\\Core\\Http\\AbstractApplication->run()
#48 /var/www/mysite/htdocs/index.php(26): {closure}()
#49 {main}","context":"FE","url":"https://mysite/sitemap_en.xml"}
Updated by Chris Müller about 2 years ago
The XML sitemap is cached for only 15 minutes:
https://github.com/TYPO3/typo3/blob/main/typo3/sysext/seo/Configuration/TypoScript/XmlSitemap/setup.typoscript#L6
This has to be considered, so a warmup must run also min. every 15 min. Perhaps the bottleneck lies somewhere else.
Do you have custom extensions with translations and a big amount of records? Perhaps this index is missing there:
https://forge.typo3.org/issues/96729
Updated by Oliver Hader about 2 years ago
- Category changed from Caching to SEO
- Status changed from New to Accepted
Updated by Christian Kuhn over 1 year ago
- Related to Task #96729: Auto generate database index "language" added
Updated by Christian Kuhn over 1 year ago
- Status changed from Accepted to Needs Feedback
Updated by Sybille Peters over 1 year ago
- Status changed from Needs Feedback to Closed
@Chris Müller
The XML sitemap is cached for only 15 minutes:
Yes, I noticed that too and increased the cache (to same as page cache).
The situation has now considerably improved for me, I had these 2 problems:
1. loading sitemap initially resulted in 5xx error
2. sitemap cache was expired
no 1 has also improved, regardless of the cache. I suspect that a bug that was fixed recently contributed there but can't say 100%.
For me the situation is sufficiently resolved and I don't see a need for a core feature. One can do a warmup of the sitemaps with simple measures, such as using curl.
I am doing a nightly warmup of sitemaps and pages and that seems to work quite fine now.
I am closing this. If anyone disagrees, can be reopened.
Actions