Bug #43725

Task #57578: TceformsUpdateWizard improvements (Migrate all file relations from tt_content.image and pages.media)

TceformsUpdateWizard exceeds memory_limit/max_execution_time

Added by Scanlitho no-lastname-given over 9 years ago. Updated over 6 years ago.

Status:
Closed
Priority:
Must have
Assignee:
-
Category:
Install Tool
Target version:
-
Start date:
2012-12-07
Due date:
% Done:

0%

Estimated time:
TYPO3 Version:
6.2
PHP Version:
5.3
Tags:
Complexity:
Is Regression:
No
Sprint Focus:

Description

Hey,

I have a very very big Website which I tried to Update to Typo3 6.0.
My test server throws an PHP Fatal error:
"Allowed memory size of 67108864 bytes exhausted."

Server settings:
max_execution_time = 30
max_input_time = 60
memory_limit = 64M

The refindex has 73,532 entrys and is 14,3 MB big.

Maybe it would be better to let the function do only a limited number of query at each run? Is that even possible?

Greetings

Affected wizard: Migrate all file relations from tt_content.image and pages.media


Related issues

Related to TYPO3 Core - Bug #57393: Endless Loop in TceformsUpdateWizardClosed2014-03-27

Actions
#1

Updated by Scanlitho no-lastname-given over 9 years ago

  • Project changed from TYPO3 Core to 1401
#2

Updated by Andreas Wolf over 9 years ago

  • Status changed from New to Needs Feedback

Sorry for not replying so long, but this tracker is only monitored from time to time now that FAL is part of the core.

Regarding your problem: 64 MB is not too much for administrative tasks, so that might be one part of the problem.

Could you give an estimate on the number of images that have to be migrated (i.e. how many files are in your uploads/ folder)? The refindex size does not give a clue here.

#3

Updated by Andreas Wolf over 9 years ago

  • Project changed from 1401 to TYPO3 Core
#4

Updated by Scanlitho no-lastname-given over 9 years ago

Hey Andreas,
in the upload folder are 18567 files.

#5

Updated by Klaus Hinum about 9 years ago

Same problem here, got a huge website with 200 GB of images in the uploaded folder. I run out of memory altough memory_limit is currently 1 GB.

#6

Updated by Alexander Opitz almost 9 years ago

  • Status changed from Needs Feedback to New
#7

Updated by Markus Klein over 8 years ago

  • Subject changed from Migrate all file relations from tt_content.image and pages.media / Memory size to TceformsUpdateWizard exceeds memory_limit/max_execution_time
  • Category set to Install Tool
  • Status changed from New to Accepted
  • Target version set to next-patchlevel
  • Parent task set to #57578
  • TYPO3 Version changed from 6.1 to 6.2
  • Is Regression set to No
#8

Updated by Markus Klein over 8 years ago

I suggest to only migrate up to 4000 records per run.
The wizard should have a hint that it requires multiple runs. (auto reloading via redirect?)

#9

Updated by Joshua Schäuble almost 8 years ago

Klaus Hinum wrote:

Same problem here, got a huge website with 200 GB of images in the uploaded folder. I run out of memory altough memory_limit is currently 1 GB.

I've also got the same problem. Working on a migration of a big web page from 4.5 to 6.2 on a testserver. For the last run with 6.2.4 (PHP 5.4.30) the wizard run completely. Now I started over again with 6.2.5 and PHP 5.4.33 and I'm getting this error. The memory limit can't be the issue (tried around with up to 3GB) since its far from the allocation boundaries which the error gives. Also Downgraded PHP back to 5.4.30 with no effect.


Detected Fatal Error
Out of memory (allocated 107741184) (tried to allocate 86507504 bytes) in E:\wwwroot\typo3\sysext\install\Classes\Updates\TceformsUpdateWizard.php on line 336

Also "just to run it over and over again" has no effect. The sys_file_reference table constantly stays on 50 entries, although it should slowly go up to something around 100,000.

#10

Updated by Markus Klein almost 8 years ago

Thanks for your report Joshua.
Does it work if you downgrade to 6.2.4 (only replace the source) just for running the wizard?
If that is the case, we have some problem in the wizard introduced with 6.2.5.

Particularly I'm thinking of #57575, which introduced a big try...catch.

#11

Updated by Joshua Schäuble almost 8 years ago

Thanks for your fast reply.

I've tried that and got the same error. The only other difference I had to previous attempts was some lowlevel_cleaner DB-cleaning (afaik everything was still running under 4.5 then), the new php 5.4.33 version (downgrading back had no effect) and 6.2.5 instead of 6.2.4. (downgrading back had no effect either). So probably the lowlever_cleaner makes the difference, still though I guess this error should not appear?

I'm not sure what to try next. Either I start again with 6.2.5 and leave out the lowlevel_cleaner or I do the lowlevel_cleaner but use 6.2.4 and then upgdate to 6.2.5 after running the update wizard... Any other ideas?

#12

Updated by Markus Klein almost 8 years ago

Basically you should always use the latest TYPO3 version to upgrade, as it usually contains also the fixes for upgrade problems.
Changing the PHP version shouldn't cause massive memory leaks.

What exactly did you do with the lowlevel_cleaner? Did you clean deleted records?

#13

Updated by Joshua Schäuble almost 8 years ago

Markus Klein wrote:

Basically you should always use the latest TYPO3 version to upgrade, as it usually contains also the fixes for upgrade problems.

Yep, that's why I did so when I started from scratch and a newer version was out.

What exactly did you do with the lowlevel_cleaner? Did you clean deleted records?

Yes I did. Did the following lowlevel tasks (in this order):

orphan_records
versions
tx_templavoila_unusedce
deleted
missing_relations

left out the other tasks for I didnt see a point after a dryrun. Especially the deleted records was the reason why I did it since the database is very bloated with records that are flagged as deleted and I hoped to speed up things by removing them (also some of the deleted records referenced missed files/deleted files, which caused problems)

#14

Updated by Markus Klein almost 8 years ago

Ok, so you cleaned away the deleted ones, that's already a good step.
Did update the reference index after the lowlevel commands as well?

#15

Updated by Joshua Schäuble almost 8 years ago

Yes and a DB compare in the install tool.

I'll give a short description of how I went on after that (I hope this gives hints on what i'm doing wrong):

After RefIndex update (integrity perfect) and DB compare (tables and field definitions are ok) I deleted "uninstalled extensions", updated those extensions that have a compatibility version (up to this point the 4.5 installation ran perfectly in the BE and FE), deleted all DAM-Extensions, updated the Reference Index again, solved all smoothmigration issues, updated the typo3 core, adjusted the "folder structure" (install tool) updated the reference index again, ran the update wizwards up to the wizard "rewrite binary file permissions into detailed list", filled sys_file by using the FAL scheduler task in the BE (otherwise "migrate all file relations from tt_content.image and pages.media" never worked for me) and updated the reference index again, then ran "migrate all file relations from tt_content.image and pages.media". --> given error

In previous attempts this worked fine so I sucessfully finished the update wizards and went on with DAM->FAL and our extension adjustment (which is what I was working on before I restarted). There I got some problems (double relations due to "pre-DAM"-fallbacks in the 4.5 Setup etc... just case-specific issues that made me believe it would be easier to first delete "deleted records", especially since a new 6.2.5 was out anyway) that caused me to start again and now i'm a bit stuck.

#16

Updated by Markus Klein almost 8 years ago

Uff... that's somehow weird.

From my experience I can only say: Never ever update the reference index after switching the source.

I've no experience with DAM, so I don know if it's necessary to uninstall the extension, but I guess you'll need to.

These steps should actually suffice:

  • Uninstall extensions
  • Clean deleted records
  • DB compare
  • RefIndex update
  • Update source
  • (fix folder structure)
  • Run upgrade wizards

Whenever you do anything content/file related while the FAL tables are not migrated/filled correctly, you will end up in a data mess.

#17

Updated by Joshua Schäuble over 7 years ago

Well, the problem is, that if I uninstall DAM (and all the other DAM related extensions) before I clean deleted records/do a db compare the DAM-references in the DB get messed up and I can't migrate them after the update wizards anymore. Besides that it also seems impossible to not do a RefIndex update after the source update: sys_file does not fill up automatically (50,000entries) so I need to run the scheduler FAL-task, but this one doesn't run through without a ref-Index update. And without sys_file being filled it is impossible to run the "migrate all file relations from tt_content.image and pages.media" update wizard. Therefore RefIndex -> Scheduler Task -> file relation wizard seemed the most logical order and worked so far, I had no data mess in the end.

Over the weekend I tried to set up everything the way I did before (old php/mysql version, no deleted record removal) and it is still getting the same error although it ran through that way before I restarted. Therefor I think it is not actually Typo3-migration related but rather a server issue that I am just not aware of. With updating mysql/php something must have changed that by resetting them to the old versions doesnt reset. Checked weather the right my.ini is loaded, checked phpinfo(), watched what happens to the memory when running the wizard (not even running full), checked what happens to the DB when running the wizard (sys_file_references always stops running full at entry uid 50, after like 5 seconds, while the error in the install tool only appears after 10 minutes! In these 10 minutes I have no idea whatsoever what the wizard is doing)

We're thinking of starting it again on another local mashine, but if this works it would unfortunatelly still not help to get rid of this general bug or to understand better in what cases it occurs.

Edit: It might be important to mention that we're running on a Windows-Server and therefore also on Windows locally.

#18

Updated by Joshua Schäuble over 7 years ago

I found the reason in our case: In tt_content.image we had some "NULL" strings instead of NULL-values which caused the wizard to fail. I do not know though, why this didn't cause a wizard crash before since these values must have been in there before(maybe due to a mysql update?). Now the wizard ran through. I'm not sure if weird tt_content.image values should cause the wizard to crash with a "memory allocation" error but I guess its fair.

Thanks again for your help.

#19

Updated by Markus Klein over 7 years ago

This is indeed strange and it is even stranger, how these stringified NULL values came into the DB.
I've no clue though, why the script runs out of memory in such a case, because missing images are reported in the logfile usually.

#20

Updated by Mathias Schreiber over 7 years ago

  • Target version changed from next-patchlevel to 7.4 (Backend)
#21

Updated by Markus Klein over 7 years ago

  • Target version changed from 7.4 (Backend) to next-patchlevel

Setting target version back to next-patchlevel as this code does not exist in CMS 7 anymore.
Once there is some dedicated target version for 6.2, please change it again.

#22

Updated by Mathias Schreiber over 6 years ago

  • Target version deleted (next-patchlevel)
#23

Updated by Wouter Wolters over 6 years ago

  • Status changed from Accepted to Closed

Over a year no action here. Meanwhile we released version 7.6 a few months ago and I think this can be closed now.
If there is still interest in fixing this, please come up with a patch for it.

Also available in: Atom PDF