Bug #89997

Unnecessary fetching of duplicate records in Typo3DatabaseBackend

Added by Markus Klösges about 1 month ago. Updated about 17 hours ago.

Status:
Resolved
Priority:
Should have
Assignee:
-
Category:
-
Target version:
-
Start date:
2019-12-19
Due date:
% Done:

100%

TYPO3 Version:
10
PHP Version:
Tags:
Complexity:
Is Regression:
Sprint Focus:

Description

A (possible) bug in EXT:staticfilecache (reported at https://github.com/lochmueller/staticfilecache/issues/233) generates massive amounts of duplicated identifier entries in tables managed by the caching framework.

This revealed that the Typo3DatabaseBackend will fetch each and every identifier in there, just to deduplicate them in php-space afterwards.
That may lead to OutOfMemory-situations, where a massive amount of rows is fetched, to be thrown away again.

The DBMS is able to do that more efficiently using a DISTINCT/Group By. Note that most other functions in Typo3DatabaseBackend already use this method.

Associated revisions

Revision 2ad25481 (diff)
Added by Markus Klösges about 17 hours ago

[BUGFIX] Reduce memory consumption in findIdentifiersByTag()

When searching cache-identifiers by tag, the Typo3DatabaseBackend now
utilizes DBMS functions for de-duplication of identifiers instead
of fetching all records and de-duplicate them afterwards in php space.
This dramatically reduces memory usage, when many duplicated identifiers
are in a cache table for any given tag.
Note that there is no noticable impact whenever there are no duplicates
at all, as the DBMS handles the de-duplication.

Releases: master, 9.5
Resolves: #89997
Change-Id: Ia28deee1e02eec9792dc4b0d4ce41e2177fcadf4
Reviewed-on: https://review.typo3.org/c/Packages/TYPO3.CMS/+/62749
Tested-by: TYPO3com <>
Tested-by: Jonas Eberle <>
Tested-by: Susanne Moog <>
Reviewed-by: Jonas Eberle <>
Reviewed-by: Daniel Goerz <>
Reviewed-by: Susanne Moog <>

Revision ab6cc198 (diff)
Added by Markus Klösges about 16 hours ago

[BUGFIX] Reduce memory consumption in findIdentifiersByTag()

When searching cache-identifiers by tag, the Typo3DatabaseBackend now
utilizes DBMS functions for de-duplication of identifiers instead
of fetching all records and de-duplicate them afterwards in php space.
This dramatically reduces memory usage, when many duplicated identifiers
are in a cache table for any given tag.
Note that there is no noticable impact whenever there are no duplicates
at all, as the DBMS handles the de-duplication.

Releases: master, 9.5
Resolves: #89997
Change-Id: Ia28deee1e02eec9792dc4b0d4ce41e2177fcadf4
Reviewed-on: https://review.typo3.org/c/Packages/TYPO3.CMS/+/62976
Tested-by: TYPO3com <>
Tested-by: Susanne Moog <>
Reviewed-by: Susanne Moog <>

History

#1 Updated by Gerrit Code Review about 1 month ago

  • Status changed from New to Under Review

Patch set 1 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/62749

#2 Updated by Gerrit Code Review about 1 month ago

Patch set 2 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/62749

#3 Updated by Gerrit Code Review about 1 month ago

Patch set 3 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/62749

#4 Updated by Gerrit Code Review about 1 month ago

Patch set 4 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/62749

#5 Updated by Gerrit Code Review 12 days ago

Patch set 5 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/62749

#6 Updated by Gerrit Code Review 3 days ago

Patch set 6 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/62749

#7 Updated by Gerrit Code Review about 17 hours ago

Patch set 1 for branch 9.5 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/62976

#8 Updated by Anonymous about 17 hours ago

  • Status changed from Under Review to Resolved
  • % Done changed from 0 to 100

Also available in: Atom PDF