Bug #92230
closed
Linkvalidator ExternalLinktype preprocessing fails and terminates link crawling with some URLs
100%
Description
I had the scheduler task fail and abort because in one content block there was a URL such as:
<a href="mailto:http://example.org">URL causes exception</a>
This link is not correct, but I think one error should not cause link checking to abort.
This is the error from the log:
Tue, 08 Sep 2020 06:01:49 +0200 [ERROR] request="a0690b01ea8ec" component="TYPO3.CMS.Scheduler.Task.AbstractTask":
Task failed to execute successfully. Class: TYPO3\CMS\Linkvalidator\Task\ValidatorTask, UID: 36, Code: 0,
Argument 1 passed to TYPO3\CMS\Core\Utility\HttpUtility::idn_to_ascii() must be of the type string, null given,
called in /var/www/mysite/htdocs/typo3/sysext/linkvalidator/Classes/Linktype/ExternalLinktype.php on line 225 - {"exception":{},"context":true}
The reason is that the 'host' is not filled on parse_url() for that URL.
For master, catching exceptions was completely removed in ExternalLinktype::preprossURL(). Don't know why.
For TYPO3 9, there is a catch block, but it should be catch Throwable for TYPO3 7, instead of just catch \Exception:
catch (\Exception $e) {
// in case of any error, return empty url.
$this->errorParams['errorType'] = 'exception';
$this->errorParams['exception'] = $e->getMessage();
$this->errorParams['message'] = $this->getErrorMessage($this->errorParams);
return '';
} catch (\Throwable $e) { // For PHP 7
// handle $e ...
return $url;
}
Also, the function should handle empty $parts['hosts'] gracefully.
And if the URL can't be converted or there is some error, it is probably better to return the original URL instead of an empty string. That way we will get useful results when checking this URL.
Releases¶
Error occurs in 9.5.20, 10.4 and master (11).
Updated by Gerrit Code Review over 3 years ago
- Status changed from New to Under Review
Patch set 1 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/65636
Updated by Gerrit Code Review over 3 years ago
Patch set 2 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/65636
Updated by Gerrit Code Review over 3 years ago
Patch set 3 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/65636
Updated by Gerrit Code Review over 3 years ago
Patch set 1 for branch 10.4 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/65610
Updated by Sybille Peters over 3 years ago
- Status changed from Under Review to Resolved
- % Done changed from 0 to 100
Applied in changeset d1e7e8af44b26898f618bae5d7fe439cd0ad9d13.
Updated by
Updated by Rémy DANIEL over 3 years ago
Hi. Thanks for the fix, but why it was not backported to 9.5?
Updated by Sybille Peters over 3 years ago
It was not backported because the patch was created with "Releases: master, 10.4". https://review.typo3.org/c/Packages/TYPO3.CMS/+/65636
That was my decision when I created the patch. Maybe not the best choice. I made it because TYPO3 9 is in priority bugfixing mode. https://typo3.org/cms/roadmap
The patch is merged now. Not sure what the procedure might be now to get it backported to 9.5.
Updated by Rémy DANIEL over 3 years ago
I asked because I've got the error on a 9.5 production server.
I do not know if this fix should be considered a "priority bugfix", but the error kills linkvalidator's task every night on this server.
Updated by Sybille Peters over 3 years ago
@Daniel Hinderink Rémy
I asked in core channel and will create a patch for 9.5
Updated by Gerrit Code Review over 3 years ago
- Status changed from Closed to Under Review
Patch set 1 for branch 9.5 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/66056
Updated by Sybille Peters over 3 years ago
@Daniel Hinderink Rémy
ready for testing ...
btw. did you check to make sure it was this error? Because I applied the fix locally on my installation and the scheduler task still dies somewhere. It is hard to find the actual cause of the error because no log message gets written. But I plan to track this down next.
P.S. I also had problems that scheduler task died somewhere in older versions.
Updated by Rémy DANIEL over 3 years ago
Yes I have the same error.
Here is the backtrace:
[ TypeError ]
Argument 1 passed to TYPO3\CMS\Core\Utility\HttpUtility::idn_to_ascii() must be of the type string, null given, called in /var/www/html/web/typo3/sysext/linkvalidator/Classes/Linktype/ExternalLinktype.php on line 225
Exception trace:
#0 ()
typo3/sysext/core/Classes/Utility/HttpUtility.php:193
#1 TYPO3\CMS\Core\Utility\HttpUtility::idn_to_ascii()
typo3/sysext/linkvalidator/Classes/Linktype/ExternalLinktype.php:225
#2 TYPO3\CMS\Linkvalidator\Linktype\ExternalLinktype->preprocessUrl()
typo3/sysext/linkvalidator/Classes/Linktype/ExternalLinktype.php:90
#3 TYPO3\CMS\Linkvalidator\Linktype\ExternalLinktype->checkLink()
typo3/sysext/linkvalidator/Classes/LinkAnalyzer.php:228
#4 TYPO3\CMS\Linkvalidator\LinkAnalyzer->getLinkStatistics()
typo3/sysext/linkvalidator/Classes/Task/ValidatorTask.php:347
#5 TYPO3\CMS\Linkvalidator\Task\ValidatorTask->checkPageLinks()
typo3/sysext/linkvalidator/Classes/Task/ValidatorTask.php:289
#6 TYPO3\CMS\Linkvalidator\Task\ValidatorTask->execute()
typo3/sysext/scheduler/Classes/Scheduler.php:179
#7 TYPO3\CMS\Scheduler\Scheduler->executeTask()
vendor/helhum/typo3-console/Classes/Console/Command/Scheduler/SchedulerRunCommand.php:121
#8 Helhum\Typo3Console\Command\Scheduler\SchedulerRunCommand->executeScheduledTasks()
vendor/helhum/typo3-console/Classes/Console/Command/Scheduler/SchedulerRunCommand.php:94
#9 Helhum\Typo3Console\Command\Scheduler\SchedulerRunCommand->execute()
vendor/symfony/console/Command/Command.php:255
#10 Symfony\Component\Console\Command\Command->run()
vendor/symfony/console/Application.php:1000
#11 Symfony\Component\Console\Application->doRunCommand()
vendor/helhum/typo3-console/Classes/Console/Mvc/Cli/Symfony/Application.php:169
#12 Helhum\Typo3Console\Mvc\Cli\Symfony\Application->doRunCommand()
vendor/symfony/console/Application.php:271
#13 Symfony\Component\Console\Application->doRun()
vendor/symfony/console/Application.php:147
#14 Symfony\Component\Console\Application->run()
vendor/helhum/typo3-console/Classes/Console/Core/Kernel.php:148
#15 Helhum\Typo3Console\Core\Kernel->handle()
vendor/helhum/typo3-console/Scripts/typo3-console.php:30
#16
{closure}
()
vendor/helhum/typo3-console/Scripts/typo3-console.php:32
#17 require()
vendor/helhum/typo3-console/typo3cms:3
Updated by Gerrit Code Review over 3 years ago
Patch set 1 for branch 9.5 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/66146
Updated by Gerrit Code Review over 3 years ago
Patch set 2 for branch 9.5 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/66146
Updated by Gerrit Code Review over 3 years ago
Patch set 3 for branch 9.5 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/66146
Updated by Sybille Peters over 3 years ago
- Related to Bug #92868: linkvalidator: HttpUtility::idn_to_ascii() throws Exception added
Updated by Gerrit Code Review over 3 years ago
Patch set 4 for branch 9.5 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/66146
Updated by Gerrit Code Review over 3 years ago
Patch set 5 for branch 9.5 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/66146
Updated by Gerrit Code Review over 3 years ago
Patch set 6 for branch 9.5 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/66146
Updated by Sybille Peters over 3 years ago
- Status changed from Under Review to Resolved
Applied in changeset 4126ef182141393a86ad1596341f22565c6a48da.