TYPO3 Core

The "@ini_set" parts were introduced by Sebastian in revision 970 on 2006-01-04.

Surprisingly, changing the memory_limit at runtime seems to work quite well at many hosters, but I agree that it's not a perfect solution.

However, it seems nobody was willing yet to work on the impexp stuff. Will you?

In fact, the import/export could quite take some time. But changing the limits at runtime might not work in all situations. A better solution would by a step-by-step processing, e.g. impexp works for a defined time of processor time (lower than max_execution_time), stores the result temporarily to database and recalls itself using a HTTP location header and continues until the whole process is finished.

But I fear this could not be solved "on the fly" and has to wait until 4.2. What do you think?

As impexp works on quite a large dataset it will always use up memory on a grand scale even if you can improve the implementation. Using a chained set of single steps (by using HTTP refresh and/or JS) seems a possible solution. But this may lead to a leaking of temp files if anything goes wrong (or the user kills the browers while waiting).

May it is possible to use AJAX to do the real stuff in ome steps in the background with a nice progressbar in the live UI.

I do not undestand that this issue is opened since such a long time, is there nobody to fix this?

You can solve it easily by raising memory_limit in php.ini to whatever is needed. However, you should reset it after successful import.

The fact that impexp consumes so much memory cannot be solved easily, and it seems like nobody finds it important to change (probably because there is a workaround).

But of course it would be highly appreciated if you can try to fix this...

The fix for this would look like something like this :

@ini_set('memory_limit', $TYPO3['SYS']['impexp']['memory_limit]);
@ini_set('max_execution_time', $TYPO3['SYS']['impexp'][max_execution_time]);

And then you set these values in localconf.php.

As this is a sysext I think a core team member could spend those 2 minutes to implement this and 2 more to documentate it. As not many people ever will use impexp or event the memory limit of 256M may be enough for lots of project, maybe there should just be a predefined value of 256M and 600 secs set at the installation of typo3.

The way impexp writes data has to be changed. Now complete data is prepared in memory before write to disc or send. of course this result in high memory load.

In future this should be done sequentially, written to a temp file, so we can save a lot of memory and still able to export a complete site.

Hey Tizian,
your thoughts are nice but it will not work. Just try to add these variables to t3lib/config_default.php and you will see that even the workaround will not work (because editable $TYPO3_CONF_VARS should be 2- not 3-dimensional). So again, it's not as easy as it seems.

Some time ago the memory_limit was hardcoded in class.tx_impexp.php but that had to be removed because it overrided custom settings from php.ini (which may have been higher).

The ideal compromise for now should be to hardcode the settings to 512M/600s unless the current values are already higher.

Again, you are very welcome to work on this issue and provide a patch. See http://typo3.org/teams/core/ on how you can help.

Greetings, Michael

Better idea: Use $TYPO3_CONF_VARS['EXT']['extConf']['impexp'] and add the options to EXT:impexp/ext_conf_template.txt. This will keep the values editable.

About 2 years ago I already proposed a solution and ready to run patch on the core list:

http://lists.typo3.org/pipermail/typo3-team-core/2008-July/017829.html

The feedback was the usual "respect the CGL" nitpicking and other worries. My code worked well and did not have many side effects. Altough exports which have been made before can probably not get imported using the new code, it would not be a big problem storing some kind of export-format version number in the generated file.

My solution consits of a patch against the trunk of this time and a T3X implementing the feature as extension (overruling the impexp sys-ext)

For those who do not want to read the whole thread: This message explains how my solution works:

http://lists.typo3.org/pipermail/typo3-team-core/2008-July/017914.html

Hi Bernhard,

I understand your frustration about the CGL nitpicking, but it looks to me like this was not the main topic of the discussion. Actually those comments came from only one person, while the rest was all about the functionality.

Suddenly the thread ends not because of CGL nitpickings but more likely because you didn't answer the last mail [1] where it turns out that functionality is getting removed. And remember that reminders are sometimes needed, especially for not-so-simple patches like this one. It needs quite some time to understand it and test it.

My suggestion for you is to revive the topic, reimplement XML support (I've never used it but it seems important to me, even if only from a marketing perspective) and get this finished.

To me it looks like you solved 95% of the problem, and only a very small effort is needed to finish it. So I'm really looking forward to see this done!

Greetings, Michael

[1] http://lists.typo3.org/pipermail/typo3-team-core/2008-July/017925.html

The support for XML export is NOT removed. It simply works the old way. So if you export an XML file you will get into the same memory exhausting problem - but it still works. The new way of exporting files will only work for the binary T3D format.

The only thing I noticed in the meantime which should get fixed is the incompability between the old T3D format and those generated after the patch. The new version should be able to recognize old-style format and import it - currently you can only import T3D files when they have been exported with the patch applied.

i've had this message (memory exhausted) now for the second time.
the question is: why does impexp use up the memory for?
in both my cases it was because of excessive files in uploads/tx_???

is there an option to ignore these files and thus stay below the "normal" 64M or lower?

R

It uses the memory for storing the files which get referenced from content elements or other exported recoreds. The impexp module builds up one huge array in memory. And when finished it serializes this array (so memory requirement == data-size X 2) and then writes the serialized array to a file.

You can either apply the patch from this bugnote:
http://bugs.typo3.org/view.php?id=8649

But I guess it wont apply cleanly against current T3 versions. The other option (to exclude referenced files) can get achieved by marking the checkbox in the export preview for files which should not get included. Or in the second tab of the impexp module you have an option: "Max size of files to include (kb):". If you set this to "0" no files will get included in the export.

yes Bernhard, this second case was a page that held three zip files
when i exclude this one page, the export function works excellent !

thanks much for adding the notion that filesize should be zero
best regards, Ron

Patch set 2 of change I3e690eea1a079ed6cadd548230afc405e005b115 has been pushed to the review server.
It is available at http://review.typo3.org/2610