Task #90143

Redirects: Poor performance of redirect matching for large redirects table

Added by Mario Lubenka almost 2 years ago. Updated about 1 month ago.

Status:
Under Review
Priority:
Should have
Assignee:
-
Category:
Link Handling, Site Handling & Routing
Start date:
2020-01-17
Due date:
% Done:

0%

Estimated time:
TYPO3 Version:
9
PHP Version:
7.2
Tags:
Complexity:
medium
Sprint Focus:

Description

Problem description

The current redirect matching retrieves all redirects from the database table and stores it in an internal cache.
This is then fetched and filtered for the route that matches the request.

Related files:

With a large redirects table this takes a pretty long time.
I created 350.000 unique redirects and it took the page about 5 seconds to redirect to the target page.
Before you say "350.000 will never be reached": On a long term a heavily edited website might reach that number at some point in time (assuming this feature will continue to exist with further TYPO3 versions).

Suggested solution

I suggest that the redirect matching is mainly done via the database query.
Most of the logic can be moved into a WHERE clause. Operations which are not available via database operations would then be applied to the remaining records.
With that the above 5 seconds of loading time could be reduced to merely 1 second.

This might also imply that the redirect cache via RedirectCacheService is not needed anymore.
Databases already come with caching mechanisms, as to such I fail to understand why a PHP cache is also needed.

This is a perfomance improvement which should be applied to TYPO3 9.5 as well.


Files

redirect-measure-time.ods (17.5 KB) redirect-measure-time.ods Complete runtime measurement Mario Lubenka, 2020-01-20 15:24

Related issues

Related to TYPO3 Core - Bug #93287: Avoid creation of unnecessary redirects on Slug changesNew2021-01-14

Actions
Related to TYPO3 Core - Epic #93547: Collection of problems with large sitesAccepted2021-02-19

Actions
Related to TYPO3 Core - Bug #91792: Automatically created redirects are not created using the DataHandlerNew2020-07-13

Actions
#1

Updated by Gerrit Code Review almost 2 years ago

  • Status changed from New to Under Review

Patch set 1 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/62956

#2

Updated by Gerrit Code Review almost 2 years ago

Patch set 2 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/62956

#3

Updated by Gerrit Code Review almost 2 years ago

Patch set 3 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/62956

#4

Updated by Mario Lubenka almost 2 years ago

Adding some performance numbers. ;)

Test Setup

I installed a TYPO3 10.3.0-dev with the Introduction package and created multiple redirects by adding the following code after Bootstrap::init in index.php and reloaded the page ONCE. After that I removed the code.

    /** @var \TYPO3\CMS\Core\Database\Connection $conn */
    $conn = GeneralUtility::makeInstance(ConnectionPool::class)
        ->getConnectionForTable('sys_redirect');
    $n = 400000; // 400k redirects
    for ($i = 1; $i <= $n; $i++) {
        $conn->insert('sys_redirect', [
            'source_host' => '*',
            'source_path' => '/test' . $i,
            'target' => 't3://page?uid=80'
        ]);
    }

I did 6 sets with a different number of redirect records in the database (1k up to 400k).
Before each set, the typo3temp folder was removed to make sure all code caches are regenerated.
During each set, the page load time until redirect header was taken 6 times.
The first value was discarded because Cache had to regenerate, leaving us with 5 values for the page load time.

The time was measured using the following command, returning the time in seconds.

curl -w %{time_total} --url https://t3-contribution.ddev.site/test10

Results

The full data is in the attached redirect-measure-time.ods file.
The table below shows the average response time until the redirect happens in miliseconds.

# redirects Current implementation Proposed implementation
1000 61ms 53ms
10000 130ms 68ms
50000 426ms 105ms
100000 701ms 141ms
250000 1560ms 263ms
400000 2384ms 346ms

As you can see, the proposed implementation is a lot faster when many redirect records in the database.

With large TYPO3 websites, the current implementation may cause page load times up to 5 seconds, which is what we discovered in our real project.

#5

Updated by Mario Lubenka almost 2 years ago

No additional indexes needed.

#6

Updated by Gerrit Code Review over 1 year ago

Patch set 4 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/62956

#7

Updated by Gerrit Code Review over 1 year ago

Patch set 5 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/62956

#8

Updated by Gerrit Code Review over 1 year ago

Patch set 6 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/62956

#9

Updated by Gerrit Code Review over 1 year ago

Patch set 7 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/62956

#10

Updated by Gerrit Code Review over 1 year ago

Patch set 8 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/62956

#11

Updated by Sybille Peters 9 months ago

  • Related to Bug #93287: Avoid creation of unnecessary redirects on Slug changes added
#12

Updated by Sybille Peters 8 months ago

  • Related to Epic #93547: Collection of problems with large sites added
#13

Updated by Daniel Goerz 7 months ago

  • Related to Bug #91792: Automatically created redirects are not created using the DataHandler added
#14

Updated by Gerrit Code Review about 2 months ago

Patch set 9 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/62956

#15

Updated by Gerrit Code Review about 1 month ago

Patch set 10 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/62956

#16

Updated by Gerrit Code Review about 1 month ago

Patch set 11 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/62956

#17

Updated by Gerrit Code Review about 1 month ago

Patch set 12 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/62956

#18

Updated by Gerrit Code Review about 1 month ago

Patch set 13 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/62956

#19

Updated by Gerrit Code Review about 1 month ago

Patch set 14 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/62956

#20

Updated by Gerrit Code Review about 1 month ago

Patch set 15 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/62956

Also available in: Atom PDF