Bug #89845

TYPO3 Extbase: clearState not clearing much

Added by Sven Burkert about 2 months ago. Updated 3 days ago.

Status:
Rejected
Priority:
Should have
Assignee:
-
Category:
Extbase
Target version:
-
Start date:
2019-12-04
Due date:
% Done:

0%

TYPO3 Version:
9
PHP Version:
7.2
Tags:
typo3, extbase, memory
Complexity:
Is Regression:
Sprint Focus:

Description

I have an import job for scheduler written with Extbase and memory usage is really high. In TYPO3 7.6, I was able to free memory with

$this->objectManager->get(\TYPO3\CMS\Extbase\Persistence\Generic\PersistenceManager::class)->clearState();

But after upgrading to TYPO3 9.5, "clearState" seems not to work like before, this function just frees a tiny amount of memory.

I've written a simple example code which demonstrates the problem. This example can run in a Extbase Controller, no need to run it in Scheduler context.

var_dump(memory_get_usage());
$v1 = $this->frontendUserRepository->findAll();
var_dump(memory_get_usage());
foreach($v1 as $vv1) {
    $a1 = $this->frontendUserRepository->findOneByUid($vv1->getUid());
}
var_dump(memory_get_usage());
$this->objectManager->get(\TYPO3\CMS\Extbase\Persistence\Generic\PersistenceManager::class)->clearState();
var_dump(memory_get_usage());

Output is int(49190752) int(49191568) int(566560256) int(544548736): Retrieving users from the database by "findOneByUid" consumes much memory, because the objects are stored in some PHP variable and I like to free this memory. In TYPO3 7, this worked with function "clearState".

History

#1 Updated by Susanne Moog about 2 months ago

  • Status changed from New to Needs Feedback

`clearState` did not change since 2015 - please also note that it is annotated as an internal function and not meant to be used by 3rd party code.

If you can, try to find out, what exactly is consuming that memory?

#2 Updated by Sven Burkert about 2 months ago

In my example, $this->frontendUserRepository->findOneByUid() consumes the memory. 12552 records are retrieved this way, this leads to 566 MB of memory usage.

#3 Updated by Alexander Schnitzler 12 days ago

  • Status changed from Needs Feedback to Rejected

I used your code to perform manual tests and I benchmarked it with blackfire.

Maybe there is a misunderstanding in what clearState() does. It removes all references to entities inside the persistence manager and the persistence session.
This does not necessarily mean that the php garbage collection is triggered. All entities remain in the memory as long as they are used in your code.

What happens here is the following:

$v1 = $this->frontendUserRepository->findAll();

This creates a QueryResultInterface, which at this point in time does not allocate memory for entities.

foreach($v1 as $vv1) {

This triggers $v1 to instantiate all entities and it binds them in it's internal property \TYPO3\CMS\Extbase\Persistence\Generic\QueryResult::$queryResult.

$a1 = $this->frontendUserRepository->findOneByUid($vv1->getUid());

This triggers a new sql query and the instantiation of new objects. I'd consider this a bug because as long as an existing, unchanged entity exists in the session, the database should not be queried again.
As `$a1` is not outside the loop, the memory consumption only increases shortly before the garbage collection kicks in.

$this->objectManager->get(\TYPO3\CMS\Extbase\Persistence\Generic\PersistenceManager::class)->clearState();

This actually works. All references to existing objects are removed.


To have reliable numbers, you should add the following to your test snippet:

unset($v1);
var_dump(memory_get_usage());

This brings your memory consumption down to about the same level it had in the beginning.
If you prove me wrong, I will dig deeper but for now I consider this bug report invalid.

#4 Updated by Sven Burkert 3 days ago

Hi Alexander,
thank you for your input.
I added

unset($v1);var_dump(memory_get_usage());

but this didn't lower memory usage:
int(50027840) int(50028656) int(575807288) int(553503120) int(553105632)

Also available in: Atom PDF