Bug #102050
closed
SOLVED: Version 12.4.6 increased our server load tenfold
0%
Description
UPDATE
We rolled out version 12.4.6 again this morning, including the latest updates to the third party extensions we use as well as our own extensions. The problem did not occur again. As the core version hasn't changed in the meantime it seems safe to assume that this was not an issue with the 12.4.6 version after all. Sorry for the confusion caused, we were very confused ourselves and the core update seemed as the most likely candidate at the time.
------------- original description ------------
We updated our product from 12.4.5 to 12.4.6 today. As a result of the update, the CPU and network traffic load on our server increased by about a factor 10 (see attached Grafana screenshots*). In htop, we were able to see a large number of php-fpm child processes with a runtime of up to four minutes. php-fpm also reguarly logged timeouts in seemingly random files, as well as warnings for too many child processes being created. The sys_log table didn't log any irregularities.
We are hosting TYPO3 on a 22.04.3 Ubuntu system, using NGINX and php8.1.
Honestly, I have no clue why it broke but I'll be happy to provide more information about the scenario. We didn't notice this at all on our staging environment, this seems to have been exacerbated by actual user traffic.
Files
Updated by Leonie Krenzer about 1 year ago
I noticed my image descriptions were a little imprecise: We decreased the version to 12.4.5 at about 2:30 PM, at 2:55 we published the changes to our own extensions again, but this time with an updated composer.lock to keep the system on version 12.4.5 while still including our changes - so we were able to rule out immediate side effects from our own changes.
Updated by Leonie Krenzer about 1 year ago
- Subject changed from Version 12.4.6 increased our server load tenfold to SOLVED: Version 12.4.6 increased our server load tenfold
- Description updated (diff)
Updated by Stefan Bürk about 1 year ago
- Status changed from New to Closed
I was in contact with Leonie over slack direct messages. This isse could
not narrowed down to any core commits, and also first conclusions related
to 3rd party extensions did not showed up again.
The upgraded project has been re-deployed, without having the raised values
in the monitoring. So it could be that 3rd party extension issue has been
fixed as new versions has been used partly for the re-deployment.
For now, there are now hints that this is core issue, and as no other reports
pointing to a similar isse has been reported I'm going to close this issue now.
If something changes feel to ask for reopening / reopen it.