Bug #90977
closed
possible race condition in indexedsearch
0%
Description
During artificially high load on a dev system (9.5.15) with 6 php-fpm threads I am able to provoke
An exception occurred while executing 'INSERT INTO `index_words` (`wid`, `baseword`, `metaphone`) VALUES (?, ?, ?)' with params [139691952, "gelungenen", "99353181"]: Duplicate entry '139691952' for key 'PRIMARY'
Backtrace:
...
at TYPO3\CMS\Core\Database\Connection->insert('index_words', array('wid' => 139691952, 'baseword' => 'gelungenen', 'metaphone' => '99353181'))
in /var/www/html/src/public/typo3/sysext/indexed_search/Classes/Indexer.php line 2170
at TYPO3\CMS\IndexedSearch\Indexer->checkWordList(array('unkomplizierte' => array('first' => 1, 'hash' => 19668563, 'metaphone' => '1229927', 'count' => 1), 'flügelkonzept' => array('first' => 5, 'hash' => 180729245, 'metaphone' => '146113107', 'count' => 1), 'neuesten' => array('first' => 10, 'hash' => 188782041, 'metaphone' => '67006256', 'count' => 1), 'erkenntnisse' => array('first' => 11, 'hash' => 15462739, 'metaphone' => '180751463', 'count' => 4), 'forschung' => array('first' => 13, 'hash' => 152791788, 'metaphone' => '99883556', 'count' => 1), 'langjährigen' => array('first' => 16, 'hash' => 82744380, 'metaphone' => '128523246', 'count' => 1), 'mflugzeugen' => array('first' => 23, 'hash' => 38128782, 'metaphone' => '7174348', 'count' => 1), 'resultat' => array('first' => 25, 'hash' => 209439804, 'metaphone' => '167262399', 'count' => 2), 'harmlose' => array('first' => 32, 'hash' => 251103929, 'metaphone' => '204428637', 'count' => 1), 'gleitleistung' => array('first' => 36, 'hash' => 132977964, 'metaphone' => '31057351', 'count' => 1), 'offenen' => array('first' => 44, 'hash' => 215076010, 'metaphone' => '7547654 ...
This seems to be a race condition as it does not show itself under low loads. Yet it is very seldom.
Updated by Xavier Perseguers about 3 years ago
Didn't investigate with load (unsure there's a high load at all) but can reproduce this problem under TYPO3 v10.
Updated by Jonas Eberle about 3 years ago
I suspected it was about 2 threads concurrently indexing and racing checks for "term exists already" vs. persisting, but it might also only depend on a specific current state + newly indexed terms.
I would love to have a reproducible case but wasn't able to come up with one.
Updated by Xavier Perseguers about 3 years ago
- Related to Bug #84541: Uncaught TYPO3 Exception in indexed_search: duplicate key (hash) error added
Updated by Gerrit Code Review over 2 years ago
- Status changed from New to Under Review
Patch set 2 for branch main of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/74688
Updated by Sybille Peters over 2 years ago
@Jonas I do not think it is a race condition and has nothing to do with load on the system. The error message "Duplicate entry" - looks like it is the same as #84541. - hash collison. I may be wrong - but I excessively looked into this problem and what you wrote above very much looks like it.
You can also look in the stack trace.
The nature of this error is that the hash collision depends on if there is a hash collision (which depends on which words are indexed). The error will only occur when the page is indexed (which is usually on the initial load). If you change the page and reload, it will be indexed (depends on configuration). After another reload, it will not - so you will not see the error the next time you reload the page.
If I am right, this issue could be closed as duplicate.
Updated by Sybille Peters over 2 years ago
For a reproducible test case, look at : https://forge.typo3.org/issues/84541#Reproduce
I have not tested this for a while, but should still give you the same result.
Updated by Jonas Eberle over 2 years ago
- Status changed from Under Review to Closed
Yes, I am fine with closing as duplicate.
I could not reproduce (after truncating indexed_search tables) when indexing with only 1 thread but this is now 2 versions in the past and I cannot reproduce with this exact site any more.
Updated by Jonas Eberle over 2 years ago
- Related to deleted (Bug #84541: Uncaught TYPO3 Exception in indexed_search: duplicate key (hash) error)
Updated by Jonas Eberle over 2 years ago
- Is duplicate of Bug #84541: Uncaught TYPO3 Exception in indexed_search: duplicate key (hash) error added
Updated by Gerrit Code Review over 2 years ago
- Status changed from Closed to Under Review
Patch set 3 for branch main of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/74688
Updated by Gerrit Code Review over 2 years ago
Patch set 4 for branch main of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/74688
Updated by Gerrit Code Review over 2 years ago
Patch set 5 for branch main of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/74688
Updated by Gerrit Code Review over 2 years ago
Patch set 6 for branch main of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/74688
Updated by Gerrit Code Review over 2 years ago
Patch set 7 for branch main of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/74688
Updated by Gerrit Code Review over 2 years ago
Patch set 8 for branch main of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/74688
Updated by Gerrit Code Review over 2 years ago
Patch set 9 for branch main of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/74688
Updated by Gerrit Code Review over 2 years ago
Patch set 10 for branch main of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/74688
Updated by Gerrit Code Review over 1 year ago
Patch set 11 for branch main of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/74688
Updated by Gerrit Code Review over 1 year ago
Patch set 12 for branch main of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/74688
Updated by Stefan Bürk over 1 year ago
- Related to Bug #101249: Prevent exception caused by hash collisions in indexed_search added
Updated by Gerrit Code Review over 1 year ago
Patch set 13 for branch main of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/74688
Updated by Gerrit Code Review over 1 year ago
Patch set 14 for branch main of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/74688
Updated by Gerrit Code Review over 1 year ago
Patch set 15 for branch main of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/74688
Updated by Gerrit Code Review over 1 year ago
Patch set 16 for branch main of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/74688
Updated by Sybille Peters over 1 year ago
- Status changed from Under Review to Closed
Updated by Stefan Bürk 10 months ago
- Related to Task #102975: Use full md5 hash for `indexed_search` added
Updated by Gerrit Code Review 10 months ago
- Status changed from Closed to Under Review
Patch set 17 for branch main of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/74688
Updated by Gerrit Code Review 10 months ago
Patch set 18 for branch main of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/74688
Updated by Gerrit Code Review 10 months ago
Patch set 19 for branch main of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/74688
Updated by Gerrit Code Review 10 months ago
Patch set 20 for branch main of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/74688
Updated by Gerrit Code Review 10 months ago
Patch set 21 for branch main of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/74688
Updated by Stefan Bürk 10 months ago
- Status changed from Under Review to Resolved
Applied in changeset 3ad5b4535b6d49758acc004e3202309e5479f642.
Updated by