TYPO3 Core

Hey Andreas,
in the upload folder are 18567 files.

Same problem here, got a huge website with 200 GB of images in the uploaded folder. I run out of memory altough memory_limit is currently 1 GB.

I suggest to only migrate up to 4000 records per run.
The wizard should have a hint that it requires multiple runs. (auto reloading via redirect?)

Klaus Hinum wrote:

Same problem here, got a huge website with 200 GB of images in the uploaded folder. I run out of memory altough memory_limit is currently 1 GB.

I've also got the same problem. Working on a migration of a big web page from 4.5 to 6.2 on a testserver. For the last run with 6.2.4 (PHP 5.4.30) the wizard run completely. Now I started over again with 6.2.5 and PHP 5.4.33 and I'm getting this error. The memory limit can't be the issue (tried around with up to 3GB) since its far from the allocation boundaries which the error gives. Also Downgraded PHP back to 5.4.30 with no effect.

Detected Fatal Error
Out of memory (allocated 107741184) (tried to allocate 86507504 bytes) in E:\wwwroot\typo3\sysext\install\Classes\Updates\TceformsUpdateWizard.php on line 336

Also "just to run it over and over again" has no effect. The sys_file_reference table constantly stays on 50 entries, although it should slowly go up to something around 100,000.

Thanks for your report Joshua.
Does it work if you downgrade to 6.2.4 (only replace the source) just for running the wizard?
If that is the case, we have some problem in the wizard introduced with 6.2.5.

Particularly I'm thinking of #57575, which introduced a big try...catch.

Thanks for your fast reply.

I've tried that and got the same error. The only other difference I had to previous attempts was some lowlevel_cleaner DB-cleaning (afaik everything was still running under 4.5 then), the new php 5.4.33 version (downgrading back had no effect) and 6.2.5 instead of 6.2.4. (downgrading back had no effect either). So probably the lowlever_cleaner makes the difference, still though I guess this error should not appear?

I'm not sure what to try next. Either I start again with 6.2.5 and leave out the lowlevel_cleaner or I do the lowlevel_cleaner but use 6.2.4 and then upgdate to 6.2.5 after running the update wizard... Any other ideas?

Basically you should always use the latest TYPO3 version to upgrade, as it usually contains also the fixes for upgrade problems.
Changing the PHP version shouldn't cause massive memory leaks.

What exactly did you do with the lowlevel_cleaner? Did you clean deleted records?

Markus Klein wrote:

Basically you should always use the latest TYPO3 version to upgrade, as it usually contains also the fixes for upgrade problems.

Yep, that's why I did so when I started from scratch and a newer version was out.

What exactly did you do with the lowlevel_cleaner? Did you clean deleted records?

Yes I did. Did the following lowlevel tasks (in this order):

orphan_records
versions
tx_templavoila_unusedce
deleted
missing_relations

left out the other tasks for I didnt see a point after a dryrun. Especially the deleted records was the reason why I did it since the database is very bloated with records that are flagged as deleted and I hoped to speed up things by removing them (also some of the deleted records referenced missed files/deleted files, which caused problems)

Ok, so you cleaned away the deleted ones, that's already a good step.
Did update the reference index after the lowlevel commands as well?

Yes and a DB compare in the install tool.

I'll give a short description of how I went on after that (I hope this gives hints on what i'm doing wrong):

After RefIndex update (integrity perfect) and DB compare (tables and field definitions are ok) I deleted "uninstalled extensions", updated those extensions that have a compatibility version (up to this point the 4.5 installation ran perfectly in the BE and FE), deleted all DAM-Extensions, updated the Reference Index again, solved all smoothmigration issues, updated the typo3 core, adjusted the "folder structure" (install tool) updated the reference index again, ran the update wizwards up to the wizard "rewrite binary file permissions into detailed list", filled sys_file by using the FAL scheduler task in the BE (otherwise "migrate all file relations from tt_content.image and pages.media" never worked for me) and updated the reference index again, then ran "migrate all file relations from tt_content.image and pages.media". --> given error

In previous attempts this worked fine so I sucessfully finished the update wizards and went on with DAM->FAL and our extension adjustment (which is what I was working on before I restarted). There I got some problems (double relations due to "pre-DAM"-fallbacks in the 4.5 Setup etc... just case-specific issues that made me believe it would be easier to first delete "deleted records", especially since a new 6.2.5 was out anyway) that caused me to start again and now i'm a bit stuck.

Uff... that's somehow weird.

From my experience I can only say: Never ever update the reference index after switching the source.

I've no experience with DAM, so I don know if it's necessary to uninstall the extension, but I guess you'll need to.

These steps should actually suffice:

• Uninstall extensions
• Clean deleted records
• DB compare
• RefIndex update
• Update source
• (fix folder structure)
• Run upgrade wizards

Whenever you do anything content/file related while the FAL tables are not migrated/filled correctly, you will end up in a data mess.

Well, the problem is, that if I uninstall DAM (and all the other DAM related extensions) before I clean deleted records/do a db compare the DAM-references in the DB get messed up and I can't migrate them after the update wizards anymore. Besides that it also seems impossible to not do a RefIndex update after the source update: sys_file does not fill up automatically (50,000entries) so I need to run the scheduler FAL-task, but this one doesn't run through without a ref-Index update. And without sys_file being filled it is impossible to run the "migrate all file relations from tt_content.image and pages.media" update wizard. Therefore RefIndex -> Scheduler Task -> file relation wizard seemed the most logical order and worked so far, I had no data mess in the end.

Over the weekend I tried to set up everything the way I did before (old php/mysql version, no deleted record removal) and it is still getting the same error although it ran through that way before I restarted. Therefor I think it is not actually Typo3-migration related but rather a server issue that I am just not aware of. With updating mysql/php something must have changed that by resetting them to the old versions doesnt reset. Checked weather the right my.ini is loaded, checked phpinfo(), watched what happens to the memory when running the wizard (not even running full), checked what happens to the DB when running the wizard (sys_file_references always stops running full at entry uid 50, after like 5 seconds, while the error in the install tool only appears after 10 minutes! In these 10 minutes I have no idea whatsoever what the wizard is doing)

We're thinking of starting it again on another local mashine, but if this works it would unfortunatelly still not help to get rid of this general bug or to understand better in what cases it occurs.

Edit: It might be important to mention that we're running on a Windows-Server and therefore also on Windows locally.

I found the reason in our case: In tt_content.image we had some "NULL" strings instead of NULL-values which caused the wizard to fail. I do not know though, why this didn't cause a wizard crash before since these values must have been in there before(maybe due to a mysql update?). Now the wizard ran through. I'm not sure if weird tt_content.image values should cause the wizard to crash with a "memory allocation" error but I guess its fair.

Thanks again for your help.

This is indeed strange and it is even stranger, how these stringified NULL values came into the DB.
I've no clue though, why the script runs out of memory in such a case, because missing images are reported in the logfile usually.