Bug #86028

getTreeList inserts duplicate keys in cache_treelist

Added by Olaf Döring 7 months ago. Updated 3 months ago.

Status:
Closed
Priority:
Must have
Category:
Caching
Start date:
2018-08-29
Due date:
% Done:

100%

TYPO3 Version:
8
PHP Version:
7.1
Tags:
Complexity:
Is Regression:
Sprint Focus:

Description

In my installation the getTreeList() function from TYPO3\CMS\Frontend\ContentObject\ContentObjectRenderer tries to insert a duplicate entry into cache_treelist.
I think this is possible on large sites with many getTreeList usage.

I think it will help to encapsulate the following block (line 6978-6986) into a try/catch construct:

GeneralUtility::makeInstance(ConnectionPool::class)->getConnectionForTable('cache_treelist')->insert(
                'cache_treelist',
                [
                    'md5hash' => $requestHash,
                    'pid' => $id,
                    'treelist' => implode(',', $theList),
                    'tstamp' => $GLOBALS['EXEC_TIME']
                ]
            );

If it fails inserting, there is an entry inserted just microseconds before, so we can ignore that.
The problem occures every 2-3 days.


Related issues

Related to TYPO3 Core - Bug #87139: Regression: getTreeList inserts duplicate keys in cache_treelist Resolved 2018-12-12
Duplicated by TYPO3 Core - Bug #86491: Duplicate entry for PRIMARY key in cache_treelist Closed 2018-10-01

Associated revisions

Revision b63f03d6 (diff)
Added by Alexander Schnitzler 4 months ago

[BUGFIX] Remove expired cache_treelist entries during runtime

When \TYPO3\CMS\Frontend\ContentObject\ContentObjectRenderer::getTreeList
checked for an existing cache_treelist entry, the given md5hash and the
expiry timestamp had been compared. As caches do not expire at all by
default, there a very few cases when an entry is actually expired.

However, if a cache entry has been expired, the cache entry hasn't been
removed and therefore the creation of a new cache entry with the same
md5hash identifier resulted in a duplicate entry exception.

To solve this, the affected, expired entry will be removed during runtime.

Releases: master, 8.7
Resolves: #86028
Resolves: #86491
Change-Id: If1a907607db29f7edd0fa77a8bb47a69bdfc0df9
Reviewed-on: https://review.typo3.org/58951
Tested-by: TYPO3com <>
Reviewed-by: Benni Mack <>
Tested-by: Benni Mack <>
Reviewed-by: Markus Klein <>
Tested-by: Markus Klein <>

Revision 7e83f87f (diff)
Added by Alexander Schnitzler 4 months ago

[BUGFIX] Remove expired cache_treelist entries during runtime

When \TYPO3\CMS\Frontend\ContentObject\ContentObjectRenderer::getTreeList
checked for an existing cache_treelist entry, the given md5hash and the
expiry timestamp had been compared. As caches do not expire at all by
default, there a very few cases when an entry is actually expired.

However, if a cache entry has been expired, the cache entry hasn't been
removed and therefore the creation of a new cache entry with the same
md5hash identifier resulted in a duplicate entry exception.

To solve this, the affected, expired entry will be removed during runtime.

Releases: master, 8.7
Resolves: #86028
Resolves: #86491
Change-Id: If1a907607db29f7edd0fa77a8bb47a69bdfc0df9
Reviewed-on: https://review.typo3.org/59031
Tested-by: TYPO3com <>
Reviewed-by: Markus Klein <>
Tested-by: Markus Klein <>

History

#1 Updated by Olaf Döring 7 months ago

Some additions:
I upgraded my TYPO3 from 6.2 to 8.7, there was no problem with 6.2. Maybe the insertion of a duplicate entry into a key field in 6.2 did not throw such an exception.

Our TYPO3 intallation is a web- and intranetsite, used by over 1000 visitors daily (website) and over 1000 employees (intranet). Our pagetree is very large, the site is online since 2001, and so we have hundreds of new pages every year. The large pagetree is a problem since many years, it slows down the menu-creation and the handling in backend.

#2 Updated by Rens Admiraal 6 months ago

Quick FYI without digging deep: we run into the same issue, but not with a lot of pages (there's 262 pages in the list of the failing query)

#3 Updated by Alexander Schnitzler 4 months ago

  • Status changed from New to Accepted
  • Assignee set to Alexander Schnitzler
  • Target version set to Candidate for patchlevel

#4 Updated by Alexander Schnitzler 4 months ago

  • Duplicated by Bug #86491: Duplicate entry for PRIMARY key in cache_treelist added

#5 Updated by Gerrit Code Review 4 months ago

  • Status changed from Accepted to Under Review

Patch set 1 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/58951

#6 Updated by Alexander Schnitzler 4 months ago

Short explanation why this issue occurres:

By default all new cache_treelist entries do not expire. This means, that if there is a cache entry for a given hash, you will get that entry.

There is just one exception though. If the endtime of a page gets updated, the cache entry gets said endtime as expiry timestamp. The problem with that is, that the method \TYPO3\CMS\Frontend\ContentObject\ContentObjectRenderer::getTreeList didn't check for expired entries and also didn't delete them.
The method "thought", there is no entry at all and that it is fine to create a new one, which obviously failes.

#7 Updated by Gerrit Code Review 4 months ago

Patch set 2 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/58951

#8 Updated by Gerrit Code Review 4 months ago

Patch set 1 for branch TYPO3_8-7 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/59031

#9 Updated by Anonymous 4 months ago

  • Status changed from Under Review to Resolved
  • % Done changed from 0 to 100

#10 Updated by Gerrit Code Review 4 months ago

  • Status changed from Resolved to Under Review

Patch set 1 for branch TYPO3_8-7 of project Teams/Security/TYPO3v4-Core has been pushed to the review server.
It is available at https://review.typo3.org/59040

#11 Updated by Stefan P 3 months ago

We also have this issue. According to the changelog this patch is in 8.7.21 -- but we still experience it with this version. Completly random the Frontend is populated with occassional "Ooops, an error occured" and the log always says "Duplicate entry 'hash' for key 'PRIMARY' in..."

#12 Updated by Guido Schmechel 3 months ago

Same here with 8.7.21

#13 Updated by Christian Toffolo 3 months ago

Same here after yesterday I upgraded from 8.7.20 to 8.7.21, in the various "typo3_xxxxxxxxxxx.log" files of my installations I have many errors related to `MysqliException: Duplicate entry` in table `cache_treelist`.

I have a script that crawl my websites and this errors happen completely randomly.

I reverted https://review.typo3.org/#/c/58951/ and after many runs of my crawler the errors didn't show up again.

#14 Updated by Alexander Schnitzler 3 months ago

  • Related to Bug #87139: Regression: getTreeList inserts duplicate keys in cache_treelist added

#15 Updated by Alexander Schnitzler 3 months ago

  • Status changed from Under Review to Closed

Also available in: Atom PDF