Bug #84541

Updated by Sybille Peters over 1 year ago

I get an Exception with HTTP status code 503 on loading a page with the Exception message:

Oops, an error occurred!
An exception occurred while executing 'INSERT INTO `index_words` (`wid`, `baseword`, `metaphone`) VALUES (?, ?, ?)' with params [55602717, "images\/botschaften", "66411504"]: Duplicate entry '55602717' for key 'PRIMARY'


h2. System

* TYPO3 version 8.7.17
* current master

h2. Severity of error

Even if Configuration preset "Live" is selected, an error is still displayed in the frontend on first load of page. The page is not rendered. Doesn't happen for all pages, only for pages where the hash collision occurs. Still, it's pretty ugly.


h2. Reproduce

# Activate the extension indexed_search
# Add static includes for indexed_search
# Add the following words to a new or existing content element: graf gettogethers abfluss erworbener
# Load the page in the frontend (you may have to do that twice)

h3. h2. Expected result

Page is displayed

h3. h2. Actual result

Exception message:

Uncaught TYPO3 Exception
An exception occurred while executing 'INSERT INTO `index_words` (`wid`, `baseword`, `metaphone`) VALUES (?, ?, ?)' with params [186135449, "gettogethers", "199699927"]: Duplicate entry '186135449' for key 'PRIMARY'


h2. Reproduce with test code

You can also reproduce the problem with this simple test script, which includes the original hash function from "IndexedSearchUtility":https://github.com/TYPO3/TYPO3.CMS/blob/master/typo3/sysext/indexed_search/Classes/Utility/IndexedSearchUtility.php and some test strings

<pre><code class="php">

$str = ['graf', 'gettogethers', 'abfluss', 'erworbener'];

function md5inthash($stringToHash)
return hexdec(substr(md5($stringToHash), 0, 7));

foreach ($str as $s) {
print("string=$s hash=" . md5inthash($s) . "\n");
The first 2 strings and the third and forth string produce the same hash (which causes the collisions):

<pre><code class="text">
string=graf hash=186135449
string=gettogethers hash=186135449
string=abfluss hash=211412923
string=erworbener hash=211412923


h2. Additional information

If you need to reproduce this several times, you can delete the index for the page by selecting the "Indexing" module, select the page with the content you inserted earlier, then select "Detailed statistics" and press the "Delete" garbage can.

You can also check the sys_log for similar errors:

<pre><code class="sql">
SELECT uid,FROM_UNIXTIME(tstamp),details FROM sys_log WHERE details LIKE '%index_word%' and error = 2 ORDER BY uid DESC LIMIT 10;

h2. Cause of error

possibly: https://github.com/TYPO3/TYPO3.CMS/blob/8124407655ae73656bf6c21f6bc8841b8e1d2023/typo3/sysext/indexed_search/Classes/Indexer.php#L2134

(the link goes to a specific commit which may no longer reflect the current codebase)

hash collision?

h2. Affected fields

These are the fields in the DB, that get filled using the current hash algorithm:

(all affected fields by looking at DB schema and content and guessing):

* index_debug.phash
* index_fulltext.phash
* index_grlist.phash
* index_grlist.phash_x
* index_grlist.hash_gr_list
* index_phash.phash
* index_phash.phash_grouping
* index_phash.contentHash
* index_rel.phash
* index_rel.wid
* *index_section.phash* (see https://forge.typo3.org/issues/79802)
* index_section.phash_t3
* *index_words.wid* (see this issue)
* *index_words.metaphone*

They should be changed from int to char(32), and the hash algorithm changed to simple PHP md5() (without truncation).