Bug #86028

getTreeList inserts duplicate keys in cache_treelist

Added by Olaf Döring about 3 years ago. Updated almost 3 years ago.

Status:
Closed
Priority:
Must have
Category:
Caching
Start date:
2018-08-29
Due date:
% Done:

100%

Estimated time:
TYPO3 Version:
8
PHP Version:
7.1
Tags:
Complexity:
Is Regression:
Sprint Focus:

Description

In my installation the getTreeList() function from TYPO3\CMS\Frontend\ContentObject\ContentObjectRenderer tries to insert a duplicate entry into cache_treelist.
I think this is possible on large sites with many getTreeList usage.

I think it will help to encapsulate the following block (line 6978-6986) into a try/catch construct:

GeneralUtility::makeInstance(ConnectionPool::class)->getConnectionForTable('cache_treelist')->insert(
                'cache_treelist',
                [
                    'md5hash' => $requestHash,
                    'pid' => $id,
                    'treelist' => implode(',', $theList),
                    'tstamp' => $GLOBALS['EXEC_TIME']
                ]
            );

If it fails inserting, there is an entry inserted just microseconds before, so we can ignore that.
The problem occures every 2-3 days.


Related issues

Related to TYPO3 Core - Bug #87139: Regression: getTreeList inserts duplicate keys in cache_treelistClosedAlexander Schnitzler2018-12-12

Actions
Has duplicate TYPO3 Core - Bug #86491: Duplicate entry for PRIMARY key in cache_treelistClosedAlexander Schnitzler2018-10-01

Actions
#1

Updated by Olaf Döring about 3 years ago

Some additions:
I upgraded my TYPO3 from 6.2 to 8.7, there was no problem with 6.2. Maybe the insertion of a duplicate entry into a key field in 6.2 did not throw such an exception.

Our TYPO3 intallation is a web- and intranetsite, used by over 1000 visitors daily (website) and over 1000 employees (intranet). Our pagetree is very large, the site is online since 2001, and so we have hundreds of new pages every year. The large pagetree is a problem since many years, it slows down the menu-creation and the handling in backend.

#2

Updated by Rens Admiraal about 3 years ago

Quick FYI without digging deep: we run into the same issue, but not with a lot of pages (there's 262 pages in the list of the failing query)

#3

Updated by Alexander Schnitzler about 3 years ago

  • Status changed from New to Accepted
  • Assignee set to Alexander Schnitzler
  • Target version set to Candidate for patchlevel
#4

Updated by Alexander Schnitzler about 3 years ago

  • Has duplicate Bug #86491: Duplicate entry for PRIMARY key in cache_treelist added
#5

Updated by Gerrit Code Review about 3 years ago

  • Status changed from Accepted to Under Review

Patch set 1 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/58951

#6

Updated by Alexander Schnitzler about 3 years ago

Short explanation why this issue occurres:

By default all new cache_treelist entries do not expire. This means, that if there is a cache entry for a given hash, you will get that entry.

There is just one exception though. If the endtime of a page gets updated, the cache entry gets said endtime as expiry timestamp. The problem with that is, that the method \TYPO3\CMS\Frontend\ContentObject\ContentObjectRenderer::getTreeList didn't check for expired entries and also didn't delete them.
The method "thought", there is no entry at all and that it is fine to create a new one, which obviously failes.

#7

Updated by Gerrit Code Review about 3 years ago

Patch set 2 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/58951

#8

Updated by Gerrit Code Review almost 3 years ago

Patch set 1 for branch TYPO3_8-7 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/59031

#9

Updated by Anonymous almost 3 years ago

  • Status changed from Under Review to Resolved
  • % Done changed from 0 to 100
#10

Updated by Gerrit Code Review almost 3 years ago

  • Status changed from Resolved to Under Review

Patch set 1 for branch TYPO3_8-7 of project Teams/Security/TYPO3v4-Core has been pushed to the review server.
It is available at https://review.typo3.org/59040

#11

Updated by Stefan P almost 3 years ago

We also have this issue. According to the changelog this patch is in 8.7.21 -- but we still experience it with this version. Completly random the Frontend is populated with occassional "Ooops, an error occured" and the log always says "Duplicate entry 'hash' for key 'PRIMARY' in..."

#12

Updated by Guido Schmechel almost 3 years ago

Same here with 8.7.21

#13

Updated by Christian Toffolo almost 3 years ago

Same here after yesterday I upgraded from 8.7.20 to 8.7.21, in the various "typo3_xxxxxxxxxxx.log" files of my installations I have many errors related to `MysqliException: Duplicate entry` in table `cache_treelist`.

I have a script that crawl my websites and this errors happen completely randomly.

I reverted https://review.typo3.org/#/c/58951/ and after many runs of my crawler the errors didn't show up again.

#14

Updated by Alexander Schnitzler almost 3 years ago

  • Related to Bug #87139: Regression: getTreeList inserts duplicate keys in cache_treelist added
#15

Updated by Alexander Schnitzler almost 3 years ago

  • Status changed from Under Review to Closed

Also available in: Atom PDF