Bug #96042
closed
Row update wizard may consume too much memory for tables with many records and content
0%
Description
Reproduce¶
I did not reproduce it exactly like this but it should be able to be reproducible:
1. Use a row updater
2. Apply this on a table with lots of rows and fields with lots of content (e.g. tt_content).
When the QueryBuilder Statement is constructed, it will out of memory.
See also patch set 1 where a test was used to create a number of rows: https://review.typo3.org/c/Packages/TYPO3.CMS/+/72243/1
The test has now been removed because not really necessary for patch, just to check that out of memory is possible with current code.
Possible todos¶
1. Fix this. Can be done with chunking, but how to define the size of chunks?
2. Think about how to fix these things in general and find other places
Updated by Sybille Peters over 2 years ago
- Related to Epic #93547: Collection of problems with large sites added
Updated by Gerrit Code Review over 2 years ago
- Status changed from New to Under Review
Patch set 1 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/72243
Updated by Sybille Peters over 2 years ago
- Tags changed from memory hog, memory, out of memory, slurping to memory hog, memory, out of memory, large-site
Updated by Gerrit Code Review over 2 years ago
Patch set 2 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/72243
Updated by Gerrit Code Review over 2 years ago
Patch set 3 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/72243
Updated by Gerrit Code Review over 2 years ago
Patch set 4 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/72243
Updated by Gerrit Code Review over 2 years ago
Patch set 5 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/72243
Updated by Gerrit Code Review over 2 years ago
Patch set 6 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/72243
Updated by Gerrit Code Review over 2 years ago
Patch set 7 for branch main of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/72243
Updated by Sybille Peters about 2 years ago
- Status changed from Under Review to Closed
Is abandoned.
Updated by Philipp Kitzberger about 1 year ago
@Sybille Peters, had a similar out-of-memory issue with a custom console command using "QueryBuilder->update()" within a loop, iterating over 4000 records. Which is not that many.
I ended up changing it to "Connection->update()" to bring down runtime from 20 minutes (!!!) to 1-2 seconds.
Maybe we should put some hint into the docs?