Project

General

Profile

Actions

Bug #59349

closed

Update Reference Index failed by SQL errors (INSERT INTO sys_refindex )

Added by Johannes C. Laxander over 10 years ago. Updated about 6 years ago.

Status:
Closed
Priority:
Should have
Assignee:
-
Category:
File Abstraction Layer (FAL)
Target version:
-
Start date:
2014-06-04
Due date:
% Done:

0%

Estimated time:
TYPO3 Version:
6.2
PHP Version:
Tags:
Complexity:
Is Regression:
No
Sprint Focus:

Description

I'll update a website from 4.7.17 to 6.2.3. By updating the reference Index I get this errors:

76054 records from 57 tables were checked/updated.
Record backend_layout:1 had 1 added indexes and 0 deleted indexes
Record backend_layout:3 had 1 added indexes and 0 deleted indexes
Record tx_news_domain_model_media:1 had 1 added indexes and 0 deleted indexes ...
Record tx_news_domain_model_file:1 had 1 added indexes and 0 deleted indexes ...
Record tx_gridelements_backend_layout:1 had 1 added indexes and 0 deleted indexes ...

As a result of this I have set debugSQL to true and get these SQL errors:

INSERT INTO sys_refindex (tablename,recuid,field,flexpointer,softref_key,softref_id,sorting,deleted,workspace,ref_table,ref_uid,ref_string,hash) VALUES ('tx_news_domain_model_media','119','image','','','','0','0','0',NULL,NULL,'','238eecf9c5c65e14bb7bbbb39f1ba839')
debug_backtrace require(.2.3/typo3/sysext/lowlevel/dbint/index.php),.2.3/typo3/mod.php#45 // TYPO3\CMS\Lowlevel\View\DatabaseIntegrityView->main#39 // TYPO3\CMS\Lowlevel\View\DatabaseIntegrityView->func_refindex#221 // TYPO3\CMS\Core\Database\ReferenceIndex->updateIndex#318 // TYPO3\CMS\Core\Database\ReferenceIndex->updateRefIndexTable#958 // TYPO3\CMS\Core\Database\DatabaseConnection->exec_INSERTquery#143 // TYPO3\CMS\Core\Database\DatabaseConnection->debug#215

OR

INSERT INTO sys_refindex (tablename,recuid,field,flexpointer,softref_key,softref_id,sorting,deleted,workspace,ref_table,ref_uid,ref_string,hash) VALUES ('tx_gridelements_backend_layout','15','icon','','','','0','0','0',NULL,NULL,'','903729ceee40d48661c4b5ce1f3e8e82')

These records are in affect to the 'image' or 'icon' field with 'group' configuration:

'image' => array(
    'exclude' => 0,
    'l10n_mode' => 'copy',
    'label' => $ll . 'tx_news_domain_model_media.media',
    'config' => array(
        'type' => 'group',
        'internal_type' => 'file',
        'allowed' => $GLOBALS['TYPO3_CONF_VARS']['GFX']['imagefile_ext'],
        'max_size' => $GLOBALS['TYPO3_CONF_VARS']['BE']['maxFileSize'],
        'uploadfolder' => 'uploads/tx_news',
        'show_thumbs' => 1,
        'size' => 1,
        'minitems' => 1,
        'maxitems' => 1,
    )
),

'icon' => array(
    'exclude' => 1,
    'label' =>
'LLL:EXT:gridelements/Resources/Private/Language/locallang_db.xml:tx_gridelements_backend_layout.icon',
    'config' => array(
        'type' => 'group',
        'internal_type' => 'file',
        'allowed' => 'jpg,gif,png',
        'uploadfolder' => 'uploads/tx_gridelements',
        'show_thumbs' => 1,
        'size' => 5,
        'minitems' => 0,
        'maxitems' => 2,
    )
),

Files

Dokument-0.pdf (83.3 KB) Dokument-0.pdf Update Reference Index - Vorgehensweise und Erkenntnisse Johannes C. Laxander, 2014-08-15 19:10
Dokument-1.pdf (83.5 KB) Dokument-1.pdf sys_refindex für ausgewählten Datensatz unter 4.7.19 vor Umstellung Johannes C. Laxander, 2014-08-15 19:10
Dokument-2.pdf (88.4 KB) Dokument-2.pdf sys_refindex für Tabelle 'tx_news_domain_model_file' vor Referenz Index Update unter 6.2.4 Johannes C. Laxander, 2014-08-15 19:10
Dokument-3.pdf (202 KB) Dokument-3.pdf Ergebnis Referenz Index CHECK unter TYPO3 6.2.4 Johannes C. Laxander, 2014-08-15 19:10
Dokument-4.pdf (286 KB) Dokument-4.pdf Referenz Index UPDATE - Alle SQL INSERT Error Johannes C. Laxander, 2014-08-15 19:10
Dokument-5.pdf (162 KB) Dokument-5.pdf Ergebnis Referenz Index CHECK vor Upgrade Wizard (Extra Test) Johannes C. Laxander, 2014-08-15 19:10
Dokument-6.pdf (61 KB) Dokument-6.pdf sys_refindex - Keine deleted records nach vor Check (Extra Test) Johannes C. Laxander, 2014-08-15 19:10
screenshot-01.pdf (44.5 KB) screenshot-01.pdf Upgrade Wizard - Migrate all file links of RTE enabled fields to FAL Johannes C. Laxander, 2014-08-16 17:45
screenshot-02.pdf (97.4 KB) screenshot-02.pdf Update failed Johannes C. Laxander, 2014-08-16 17:45
Update failed - Migrate all file links of RTE-enabled fields to FAL .txt (3.23 KB) Update failed - Migrate all file links of RTE-enabled fields to FAL .txt Johannes C. Laxander, 2014-08-16 17:47
screenshot-03.pdf (102 KB) screenshot-03.pdf DB Check: records with uids specified in error messages Johannes C. Laxander, 2014-08-17 13:29
DB-Check-uids-in-error messages.pdf (199 KB) DB-Check-uids-in-error messages.pdf Shows the bodytext field of the checked records Johannes C. Laxander, 2014-08-17 13:29

Related issues 1 (0 open1 closed)

Related to TYPO3 Core - Bug #64253: Fix incorrect calculation of file references in workspacesClosed2015-01-13

Actions
Actions #1

Updated by Markus Klein over 10 years ago

What is the actual SQL error message?

Actions #2

Updated by Johannes C. Laxander over 10 years ago

exec_INSERTquery

caller: TYPO3\CMS\Core\Database\DatabaseConnection::exec_INSERTquery
ERROR: Column 'ref_uid' cannot be null

Actions #3

Updated by Markus Klein over 10 years ago

Thanks.

You reported this for 6.2, but I guess you ran the ref index update on 4.7. So the bug is on 4.7 then, right?

Actions #4

Updated by Jo Hasenau over 10 years ago

That's what I assumed too, but according to the discussion in the German list, the update has been done on 6.2.3 after the migration tool gave the advice to run a reference index update before.

Actions #5

Updated by Markus Klein over 10 years ago

Well I guess we have to rephrase this text.
You should update the reference index BEFORE upgrading in the old version.

Actions #6

Updated by Jo Hasenau over 10 years ago

It doesn't make a difference though, since this is still a bug in the reference index updater, which should not insert NULL values but the correct defaults.

Actions #7

Updated by Jo Hasenau over 10 years ago

BTW: Here is the discussion http://forum.typo3.org/index.php/t/204087/

It seems that the reference index had been updated with 4.7.x and everything was reported to be OK.
Then the upgrade was done using "Migrate all file links of RTE-enabled fields to FAL" which produced the report "There are currently 52 links to update. Update failed!"
There was a hint to do a reference index upgrade, which is why the update has been run on 6.2.3 again, which produced the SQL errors mentioned above.

IMHO the problem is, that there is neither a reference table nor a reference uid, when the reference points to a non FAL based group field of the internal type "file".

Actions #8

Updated by Markus Klein over 10 years ago

I guess there are three issues actually.

  • First one is that internal_type 'file' seems to be not handled correctly.
  • Second one is that there's a missing check somewhere that the ref_table is not valid
  • Third one is that the upgrade wizard obviously does not work in all cases, since it should be able to convert all links in the first run.

So we need a DB dump or somebody debugging the upgrade wizard on this instance to figure out why it says the update fails. "What's wrong with these links".

Actions #9

Updated by Johannes C. Laxander over 10 years ago

Both is possible. It is a test instance, and so you can got an admin login by email.
Unless you'd prefer a DB Dump?

Actions #10

Updated by Markus Klein over 10 years ago

Hi!
Well admin sounds good already, but how many content elements are in this instance?
If it is over 100, I'd prefer if you could do a real debugging session.

Actions #11

Updated by Johannes C. Laxander over 10 years ago

tt_content has ca. 1800 records.

Sorry I do not understand what you mean:
"if you could do a real debugging session" ?

Actions #12

Updated by Markus Klein over 10 years ago

debuggin with your IDE via xdebug to find out which records are affected by this problem.

Actions #13

Updated by Johannes C. Laxander over 10 years ago

I'm not a developer, and so I have no IDE installed.

Actions #14

Updated by Markus Klein over 10 years ago

Ok, do you know if your server has xdebug enabled for remote debugging?

Actions #15

Updated by Johannes C. Laxander over 10 years ago

No, xdebug.so is not pre-installed, but it is possible to do this. Provider ist jweiland.net and from support I#ve got this answer:

Ja, die Nutzung von xdebug ist möglich. In den php Einstellung in der Domainkonfiguration muss die domainspezifische Einstellung aktiviert werden. Dann muss unter Zend-Extensions der Pfad zu einer lokalen xdebug.so eintragen werden.

Unter den sonstigen Einstellungen müsste dann noch ein Eintrag

xdebug.remote_enable = on

vorgenommen werden. Alternativ können die Einstellungen auch in lokale php.ini

Dateien gepackt werden, die dann in das jeweilige PHP Quellcode Verzeichnis kopiert werden.

Eine Anleitung zur generellen Aktivierung von weiteren PHP-Erweiterungen finden Sie unter dem nachfolgenden Link:

http://jweiland.net/typo3-hosting/anleitung/php/weitere-php-module.html

Actions #16

Updated by Johannes C. Laxander over 10 years ago

What is the status? Did you wait for something?

Actions #17

Updated by Markus Klein over 10 years ago

Sorry, I it's currently not possible for me to invest more time into getting debugging running on your hosting provider.
If you want, you can send me a complete DB dump and filesystem dump of the 4.7.17 instance, so I can try an upgrade locally on my dev-server, where I've all necessary tools at hand.

Updated by Johannes C. Laxander over 10 years ago

Ich verfasse diesen Beitrag in deutsch und bitte um Verständnis und Nachsicht, da meine Sprachkenntnisse nicht ausreichen, diesen Beitrag in englischer Sprache zu schreiben.

Ich habe noch einmal einen Versuch unternommen das Upgrade von TYPO3 4.7.19 auf Version 6.2.4 zu machen. Leider mit demselben Ergebnis wie schon oben berichtet. Mein Vorgehen und die Erkenntnisse habe ich Dokument-0.pdf" zusammengefasst.

Kurz-Fazit:
Der Referenz-Index ist unter TYPO3 4.7.19 noch OK. Und direkt nach der Umstellung auf den TYPO3-Source 6.2.4 tritt das Problem auf.

Ich sollte die Website eigentlich schon umgestellt haben, komme hier aber einfach nicht weiter. Habe ich die falsche Vorgehensweise? Ist es möglicherweise ein Bug im TYPO3 Code 6.2.x? Wobei kann ich evtl. helfen, um hier möglichst schnell zu einem Ergebnis zu kommen, das mich in die Lage versetzt, die Umstellung zu Ende zu bringen?

Danke für jede Unterstützung.

Actions #19

Updated by Markus Klein over 10 years ago

  • Status changed from New to Needs Feedback

Why do you update the reference index again during the update steps??

Please continue on the mailing lists since this is the bugtracker. (e.g. german mailing list - I'm not reading this list though, just the english ones)
Once you sorted out the issue, you can update this ticket here with the precise reproduction instructions.
I'm changing the status of the ticket meanwhile.
Thank you.

Updated by Johannes C. Laxander over 10 years ago

Update the reference index again during the update steps: please have a look to screenshot-01.pdf
When I do this not the update on wizard step "Migrate all file links of RTE-enabled fields to FAL" failed. Please have a look to screenshot-02.pdf (All 20 records are shown in file "Update failed - Migrate all file links of RTE-enabled fields to FAL.txt". Go back to the updade wizard bring me back to the step "Migrate all file links of RTE-enabled fields to FAL" as shown in file "screenshot-01.pdf.

Actions #22

Updated by Markus Klein over 10 years ago

Questions:
  • Are the files reported missing really missing or are they present?
  • Are those files really used in the the 4.7 installation? (Please check the DB with the uids specified in the error messages)

We need to find out, if the data is inconsistent on 4.7 or if the update wizards do something wrong.

Updated by Johannes C. Laxander over 10 years ago

Are the files reported missing really missing or are they present?

Yes, the files are really missing, except the jpg-files in folder fileadmin/user_upload/bilder/kairos/: kabine-A.jpg and kabine-B.jpg (last 4 messages)

Are those files really used in the the 4.7 installation? (Please check the DB with the uids specified in the error messages)

Yes, the files are really used in the 4.7 installation. Some of the records are hidden. - See screenshot-03.pdf and file DB-Check-uids-in-error messages.pdf

Actions #24

Updated by Jo Hasenau about 10 years ago

I just pointed out, what the actual problem is, but maybe this has been overlooked somehow ;-)

IMHO the problem is, that there is neither a reference table nor a reference uid, when the reference points to a non FAL based group field of the internal type "file".

This results in insert queries with NULL values, which are throwing errors with fields set to NOT NULL.

Actions #25

Updated by Alexander Opitz almost 10 years ago

  • Category set to File Abstraction Layer (FAL)
  • Status changed from Needs Feedback to New
Actions #26

Updated by Markus Klein over 9 years ago

Is this still an issue?

Actions #27

Updated by Jo Hasenau over 9 years ago

Seems this has been fixed together with #64253. In line 198 of typo3/sysext/core/Classes/Database/ReferenceIndex.php there is now a check that should prevent the NULL values from being inserted.

foreach ($relations as $k => $datRec) {
    if (!is_array($relations[$k])) {
        continue;
    }
...

Actions #28

Updated by Markus Klein over 9 years ago

  • Status changed from New to Resolved
Actions #29

Updated by Benni Mack about 6 years ago

  • Status changed from Resolved to Closed
Actions

Also available in: Atom PDF