Bug #83451

DatabaseRowsUpdateWizard Out of Memory on big tables

Added by Alexander Opitz over 1 year ago. Updated 9 months ago.

Status:
Needs Feedback
Priority:
Should have
Assignee:
-
Category:
Install Tool
Start date:
2018-01-02
Due date:
% Done:

0%

TYPO3 Version:
8
PHP Version:
Tags:
Complexity:
Is Regression:
Sprint Focus:

Description

The "database migrations on single rows" wizard selects complete tables, which leads to memory exhausted on large tables (on my system 1,6 million rows) as all rows get selected. Normally the rows should only be read from database while one row is processed after the other one (as we don't use something like fetchAll) but there are PHP issues around the buffering on MySQLi and pdo_mysql drivers.
See https://secure.php.net/manual/en/mysqlinfo.concepts.buffering.php about this (And special one in MySQLi see https://github.com/doctrine/dbal/blob/master/lib/Doctrine/DBAL/Driver/Mysqli/MysqliStatement.php#L192).

So we should move from select all to select only 1(0).000 rows at once. This will be slower but helps in this OoM issue.


Related issues

Related to TYPO3 Core - Task #86200: Migrate RowUpdater wizards Closed 2018-09-08

History

#1 Updated by Alexander Opitz over 1 year ago

Shortest complete answer: https://stackoverflow.com/questions/13728106/unexpectedly-hitting-php-memory-limit-with-a-single-pdo-query

But we can't switch from buffered to unbuffered read, as we do queries while iterating over the rows.

#2 Updated by Christian Kuhn 9 months ago

  • Status changed from New to Needs Feedback

unsure on how to proceed. any ideas? note: in v9, the row updater can be called via cli, this may relax the issue a bit.

#3 Updated by Christian Kuhn 9 months ago

  • Related to Task #86200: Migrate RowUpdater wizards added

Also available in: Atom PDF