Bug #96042
open
Row update wizard may consume too much memory for tables with many records and content
0%
Description
Reproduce¶
I did not reproduce it exactly like this but it should be able to be reproducible:
1. Use a row updater
2. Apply this on a table with lots of rows and fields with lots of content (e.g. tt_content).
When the QueryBuilder Statement is constructed, it will out of memory.
See also patch set 1 where a test was used to create a number of rows: https://review.typo3.org/c/Packages/TYPO3.CMS/+/72243/1
The test has now been removed because not really necessary for patch, just to check that out of memory is possible with current code.
Possible todos¶
1. Fix this. Can be done with chunking, but how to define the size of chunks?
2. Think about how to fix these things in general and find other places
Updated by Sybille Peters about 3 years ago
- Related to Epic #93547: Collection of problems with large sites added
Updated by Gerrit Code Review about 3 years ago
- Status changed from New to Under Review
Patch set 1 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/72243
Updated by Sybille Peters about 3 years ago
- Tags changed from memory hog, memory, out of memory, slurping to memory hog, memory, out of memory, large-site
Updated by Gerrit Code Review about 3 years ago
Patch set 2 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/72243
Updated by Gerrit Code Review about 3 years ago
Patch set 3 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/72243
Updated by Gerrit Code Review about 3 years ago
Patch set 4 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/72243
Updated by Gerrit Code Review about 3 years ago
Patch set 5 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/72243
Updated by Gerrit Code Review about 3 years ago
Patch set 6 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/72243
Updated by Gerrit Code Review about 3 years ago
Patch set 7 for branch main of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/72243
Updated by Sybille Peters almost 3 years ago
- Status changed from Under Review to Closed
Is abandoned.
Updated by Philipp Kitzberger almost 2 years ago
@Sybille Peters, had a similar out-of-memory issue with a custom console command using "QueryBuilder->update()" within a loop, iterating over 4000 records. Which is not that many.
I ended up changing it to "Connection->update()" to bring down runtime from 20 minutes (!!!) to 1-2 seconds.
Maybe we should put some hint into the docs?
Updated by Jigal van Hemert about 1 month ago
Currently we run into this problem with an upgrade to v12 (PHP 8.3, MariaDB 10.11) both on a local ddev system but also on a real server. The row updaters stop after the sys_file_reference table. Next up after checking the code seems to be tt_content (600k records).
Wizard was started from CLI to prevent issues with max execution time.
PHP Fatal error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 2900456 bytes) in /xxx/http/vendor/doctrine/dbal/src/Driver/Mysqli/Result.php on line 64
This line contains $this->statement->store_result(); . The PHP docs ( https://www.php.net/manual/en/mysqli.store-result.php ) mention "Although it is always good practice to free the memory used by the result of a query using the mysqli_free_result() function, when transferring large result sets using the mysqli_store_result() this becomes particularly important."
Although the tests shown by Stefan Bürk showed very little peak memory usage the query to fetch the entire tt_content table seems to cause PHP to run out of memory. What is different between his system and the system where I encounter the issue?