Bug #92230

Linkvalidator ExternalLinktype preprocessing fails and terminates link crawling with some URLs

Added by Sybille Peters 3 months ago. Updated 12 days ago.

Status:
Resolved
Priority:
Should have
Assignee:
-
Category:
Linkvalidator
Target version:
-
Start date:
2020-09-08
Due date:
% Done:

100%

Estimated time:
TYPO3 Version:
10
PHP Version:
Tags:
Complexity:
Is Regression:
Sprint Focus:

Description

I had the scheduler task fail and abort because in one content block there was a URL such as:

<a href="mailto:http://example.org">URL causes exception</a>

This link is not correct, but I think one error should not cause link checking to abort.

This is the error from the log:

Tue, 08 Sep 2020 06:01:49 +0200 [ERROR] request="a0690b01ea8ec" component="TYPO3.CMS.Scheduler.Task.AbstractTask": 
Task failed to execute successfully. Class: TYPO3\CMS\Linkvalidator\Task\ValidatorTask, UID: 36, Code: 0, 
Argument 1 passed to TYPO3\CMS\Core\Utility\HttpUtility::idn_to_ascii() must be of the type string, null given, 
called in /var/www/mysite/htdocs/typo3/sysext/linkvalidator/Classes/Linktype/ExternalLinktype.php on line 225 - {"exception":{},"context":true}

The reason is that the 'host' is not filled on parse_url() for that URL.

For master, catching exceptions was completely removed in ExternalLinktype::preprossURL(). Don't know why.

For TYPO3 9, there is a catch block, but it should be catch Throwable for TYPO3 7, instead of just catch \Exception:

catch (\Exception $e) {
    // in case of any error, return empty url.
    $this->errorParams['errorType'] = 'exception';
    $this->errorParams['exception'] = $e->getMessage();
    $this->errorParams['message'] = $this->getErrorMessage($this->errorParams);
    return '';
} catch (\Throwable $e) { // For PHP 7
    // handle $e ...
    return $url;
}

Also, the function should handle empty $parts['hosts'] gracefully.

And if the URL can't be converted or there is some error, it is probably better to return the original URL instead of an empty string. That way we will get useful results when checking this URL.

Releases

Error occurs in 9.5.20, 10.4 and master (11).


Related issues

Related to TYPO3 Core - Bug #92868: linkvalidator: HttpUtility::idn_to_ascii() throws ExceptionClosedSybille Peters2020-11-18

Actions
#1

Updated by Sybille Peters 3 months ago

  • Description updated (diff)
#2

Updated by Gerrit Code Review 3 months ago

  • Status changed from New to Under Review

Patch set 1 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/65636

#3

Updated by Sybille Peters 3 months ago

  • Description updated (diff)
#4

Updated by Gerrit Code Review 3 months ago

Patch set 2 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/65636

#5

Updated by Sybille Peters 3 months ago

  • Description updated (diff)
#6

Updated by Sybille Peters 3 months ago

  • Description updated (diff)
#7

Updated by Gerrit Code Review 3 months ago

Patch set 3 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/65636

#8

Updated by Gerrit Code Review 3 months ago

Patch set 1 for branch 10.4 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/65610

#9

Updated by Sybille Peters 3 months ago

  • Status changed from Under Review to Resolved
  • % Done changed from 0 to 100
#10

Updated by Benni Mack 2 months ago

  • Status changed from Resolved to Closed
#11

Updated by DANIEL Rémy about 2 months ago

Hi. Thanks for the fix, but why it was not backported to 9.5?

#12

Updated by Sybille Peters about 2 months ago

It was not backported because the patch was created with "Releases: master, 10.4". https://review.typo3.org/c/Packages/TYPO3.CMS/+/65636

That was my decision when I created the patch. Maybe not the best choice. I made it because TYPO3 9 is in priority bugfixing mode. https://typo3.org/cms/roadmap

The patch is merged now. Not sure what the procedure might be now to get it backported to 9.5.

#13

Updated by DANIEL Rémy about 2 months ago

I asked because I've got the error on a 9.5 production server.
I do not know if this fix should be considered a "priority bugfix", but the error kills linkvalidator's task every night on this server.

#14

Updated by Sybille Peters about 2 months ago

Daniel Hinderink Rémy

I asked in core channel and will create a patch for 9.5

#15

Updated by Gerrit Code Review about 2 months ago

  • Status changed from Closed to Under Review

Patch set 1 for branch 9.5 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/66056

#16

Updated by Sybille Peters about 2 months ago

Daniel Hinderink Rémy

ready for testing ...

btw. did you check to make sure it was this error? Because I applied the fix locally on my installation and the scheduler task still dies somewhere. It is hard to find the actual cause of the error because no log message gets written. But I plan to track this down next.

P.S. I also had problems that scheduler task died somewhere in older versions.

#17

Updated by DANIEL Rémy about 2 months ago

Yes I have the same error.
Here is the backtrace:

[ TypeError ]
Argument 1 passed to TYPO3\CMS\Core\Utility\HttpUtility::idn_to_ascii() must be of the type string, null given, called in /var/www/html/web/typo3/sysext/linkvalidator/Classes/Linktype/ExternalLinktype.php on line 225

Exception trace:
#0 ()
typo3/sysext/core/Classes/Utility/HttpUtility.php:193
#1 TYPO3\CMS\Core\Utility\HttpUtility::idn_to_ascii()
typo3/sysext/linkvalidator/Classes/Linktype/ExternalLinktype.php:225
#2 TYPO3\CMS\Linkvalidator\Linktype\ExternalLinktype->preprocessUrl()
typo3/sysext/linkvalidator/Classes/Linktype/ExternalLinktype.php:90
#3 TYPO3\CMS\Linkvalidator\Linktype\ExternalLinktype->checkLink()
typo3/sysext/linkvalidator/Classes/LinkAnalyzer.php:228
#4 TYPO3\CMS\Linkvalidator\LinkAnalyzer->getLinkStatistics()
typo3/sysext/linkvalidator/Classes/Task/ValidatorTask.php:347
#5 TYPO3\CMS\Linkvalidator\Task\ValidatorTask->checkPageLinks()
typo3/sysext/linkvalidator/Classes/Task/ValidatorTask.php:289
#6 TYPO3\CMS\Linkvalidator\Task\ValidatorTask->execute()
typo3/sysext/scheduler/Classes/Scheduler.php:179
#7 TYPO3\CMS\Scheduler\Scheduler->executeTask()
vendor/helhum/typo3-console/Classes/Console/Command/Scheduler/SchedulerRunCommand.php:121
#8 Helhum\Typo3Console\Command\Scheduler\SchedulerRunCommand->executeScheduledTasks()
vendor/helhum/typo3-console/Classes/Console/Command/Scheduler/SchedulerRunCommand.php:94
#9 Helhum\Typo3Console\Command\Scheduler\SchedulerRunCommand->execute()
vendor/symfony/console/Command/Command.php:255
#10 Symfony\Component\Console\Command\Command->run()
vendor/symfony/console/Application.php:1000
#11 Symfony\Component\Console\Application->doRunCommand()
vendor/helhum/typo3-console/Classes/Console/Mvc/Cli/Symfony/Application.php:169
#12 Helhum\Typo3Console\Mvc\Cli\Symfony\Application->doRunCommand()
vendor/symfony/console/Application.php:271
#13 Symfony\Component\Console\Application->doRun()
vendor/symfony/console/Application.php:147
#14 Symfony\Component\Console\Application->run()
vendor/helhum/typo3-console/Classes/Console/Core/Kernel.php:148
#15 Helhum\Typo3Console\Core\Kernel->handle()
vendor/helhum/typo3-console/Scripts/typo3-console.php:30
#16

{closure}
()
vendor/helhum/typo3-console/Scripts/typo3-console.php:32
#17 require()
vendor/helhum/typo3-console/typo3cms:3
#18

Updated by Gerrit Code Review about 2 months ago

Patch set 1 for branch 9.5 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/66146

#19

Updated by Gerrit Code Review about 1 month ago

Patch set 2 for branch 9.5 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/66146

#20

Updated by Gerrit Code Review about 1 month ago

Patch set 3 for branch 9.5 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/66146

#21

Updated by Sybille Peters 13 days ago

  • Related to Bug #92868: linkvalidator: HttpUtility::idn_to_ascii() throws Exception added
#22

Updated by Gerrit Code Review 13 days ago

Patch set 4 for branch 9.5 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/66146

#23

Updated by Gerrit Code Review 13 days ago

Patch set 5 for branch 9.5 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/66146

#24

Updated by Gerrit Code Review 12 days ago

Patch set 6 for branch 9.5 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/66146

#25

Updated by Sybille Peters 12 days ago

  • Status changed from Under Review to Resolved

Also available in: Atom PDF