Project

General

Profile

Actions

Bug #102050

closed

SOLVED: Version 12.4.6 increased our server load tenfold

Added by Leonie Krenzer about 1 year ago. Updated about 1 year ago.

Status:
Closed
Priority:
Should have
Assignee:
-
Category:
Performance
Target version:
-
Start date:
2023-09-27
Due date:
% Done:

0%

Estimated time:
TYPO3 Version:
12
PHP Version:
8.1
Tags:
Complexity:
Is Regression:
Sprint Focus:

Description

UPDATE
We rolled out version 12.4.6 again this morning, including the latest updates to the third party extensions we use as well as our own extensions. The problem did not occur again. As the core version hasn't changed in the meantime it seems safe to assume that this was not an issue with the 12.4.6 version after all. Sorry for the confusion caused, we were very confused ourselves and the core update seemed as the most likely candidate at the time.

------------- original description ------------

We updated our product from 12.4.5 to 12.4.6 today. As a result of the update, the CPU and network traffic load on our server increased by about a factor 10 (see attached Grafana screenshots*). In htop, we were able to see a large number of php-fpm child processes with a runtime of up to four minutes. php-fpm also reguarly logged timeouts in seemingly random files, as well as warnings for too many child processes being created. The sys_log table didn't log any irregularities.

We are hosting TYPO3 on a 22.04.3 Ubuntu system, using NGINX and php8.1.

Honestly, I have no clue why it broke but I'll be happy to provide more information about the scenario. We didn't notice this at all on our staging environment, this seems to have been exacerbated by actual user traffic.


Files

CPU_usage.png (104 KB) CPU_usage.png you can see the time on the horizontal axis in both images. We deployed 12.4.6 at 7:15 AM, and rolled it back to 12.4.5 at about 2:55 PM (and paniced in between). Leonie Krenzer, 2023-09-27 13:36
network_traffic.png (98.5 KB) network_traffic.png you can see the time on the horizontal axis in both images. We deployed 12.4.6 at 7:15 AM, and rolled it back to 12.4.5 at about 2:55 PM (and paniced in between). Leonie Krenzer, 2023-09-27 13:36
Actions

Also available in: Atom PDF