Bug #51116

Task #52304: Performance issues

Massive speed problem from TYPO3 version 6.x at the first page request after long period

Added by Harald no-lastname-given about 9 years ago. Updated almost 4 years ago.

Status:
Closed
Priority:
Won't have this time
Assignee:
-
Category:
Performance
Target version:
-
Start date:
2013-08-16
Due date:
% Done:

100%

Estimated time:
TYPO3 Version:
6.0
PHP Version:
5.3
Tags:
Complexity:
Is Regression:
No
Sprint Focus:

Description

Dear ladies and gentlemen!

I want to tell you that I have a massive connection problems since TYPO3 version 6.x.

If my page after a period of about 10 hours (in the meantime the site was never called!!!) is called for the first time, often takes several of seconds to a browser can open the page. Then the connection speeds are fine.
The page can be used properly? Everything goes very fast!

To my "calming" I could now confirm that the same phenomenon have other people. And my provider could confirm that some of his customers have the same problems.

I use NO fluid templating engine!
I have built my page on automaketemplate.

I do not write them out of desperation. Rather, I would like to point out or maybe can confirm the TYPO3 User hereby further.

Thank you very much

Harald


Related issues

Related to TYPO3 Core - Bug #52125: Saving records takes ages to completeClosed2013-09-19

Actions
Related to TYPO3 Core - Bug #52235: Timeout when copying pages recursively due to cache-clearing overloadClosed2013-09-24

Actions
Related to TYPO3 Core - Task #53747: Change TCA cache from php code to serialized arrayClosedThomas Maroschik2013-11-18

Actions
#1

Updated by Harald no-lastname-given about 9 years ago

I wanted to add that this is "NOT" a cache issue is and other people have the same problem from the 6.x version of TYPO3

Harald

#2

Updated by Martin Borer almost 9 years ago

Hi Harald,

I've had similar problems, but it was not a Typo3 issue. It was a configuration issue of MySQL. Which version of MySQL do you use? I've had problems after an Upgrade from MySQL 5.1 to 5.5. Insert Queries to innodb tables were very slow!

Had to set:

    innodb_flush_log_at_trx_commit = 0

in my.cnf under [mysqld] part.

#3

Updated by Harald no-lastname-given almost 9 years ago

Hi Martin!

Thank you for your recommendation. I started the project with MySQL 5.5!

Unfortunately I have no way to access the mysql.ini. This is managed by my TYPO3-hoster.

This hoster in a TYPO3 specialist and only hosts TYPO3 projects. He is also well known. I think it will fit regarding this setting.

Thank you very much

Harald

#4

Updated by Henrik Ziegenhain almost 9 years ago

Hi Martin,

thank you for the recommendation regarding MySQL settings.
We are also affected by this issue.
The Hoster disabled the option and the page generation is much faster.

So, is this an TYPO3 Issue that should be solved or a suboptimal configuration?

#5

Updated by Martin Borer almost 9 years ago

Henrik Ziegenhain wrote:

So, is this an TYPO3 Issue that should be solved or a suboptimal configuration?

Hi Henrik,

Glad that your problems are solved now! I've had to spend a lot of time, to figure this out. ;)

No it's more a suboptimal/wrong configuration than a Typo3 Error! Typo3 4.5 instances on such systems are also very slow.

I've had this problem on a debian Wheezy server. It's possible that only this systems are 'affected'... I don't know.

#6

Updated by Harald no-lastname-given almost 9 years ago

  • Target version set to 6.0.9

Dear Ladies and Gentlemen!

I have found an interesting post on typo3 forge, which may have a link with my problem described?

http://forge.typo3.org/issues/50231

Many Thanks

Harald

#7

Updated by Henrik Ziegenhain almost 9 years ago

Hi Martin,

our hoster said, it has to do with ext4 filesystem.
If I remember right: every database action is stored in the filesystem for a better data integrity.
Generating a non-cached page took ~30sec, a cached page was rendered in about 300ms.

#8

Updated by Martin Borer almost 9 years ago

Henrik Ziegenhain wrote:

Hi Martin,

our hoster said, it has to do with ext4 filesystem.
If I remember right: every database action is stored in the filesystem for a better data integrity.
Generating a non-cached page took ~30sec, a cached page was rendered in about 300ms.

Ohh sound's interesting! So the main problem is not the MySQL Setting innodb_flush_log_at_trx_commit?
Do you know, what has to be done to fix the ext4 issue?

#9

Updated by Henrik Ziegenhain almost 9 years ago

I think the combination of ext4 and innodb_flush_log_at_trx_commit was the problem.

Sorry, but I am not so familiar with those technical issues.

#10

Updated by Ernesto Baschny almost 9 years ago

  • Category set to Performance
  • Status changed from New to Needs Feedback
  • Priority changed from Should have to Won't have this time
  • Target version deleted (6.0.9)

So nothing TYPO3 can do except that we could document the settings or system configuration somewhere, right? Maybe you could add this somewhere to this (first hit in Google for "TYPO3 performance"): http://wiki.typo3.org/Performance_tuning

If that's okey, we would then close this bug issue in our tracker.

#11

Updated by Martin Borer almost 9 years ago

Yes.. for me, it's not a Typo3 Issue!

As Hanrik pointed out, it's a ext4 issue.
Truncating table queries were also very slow! Clear Typo3 Cache took very long.
Had to disable write barriers of the mounted ext4 hd to fix this (https://wiki.archlinux.org/index.php/Ext4).

So changing MySQL Setting innodb_flush_log_at_trx_commit fixes only one part of the Problem.

#12

Updated by Stefan Froemken almost 9 years ago

Hello,

do you really know WHAT innodb_flush_log_at_trx_commit does? It stops writing the log for just 1 second. So...if something gets wrong your data gets lost if you set this value to 0. I know...this options makes MySQL about 50% faster with inserts. This is pretty cool and works only with InnoDB (Mainly Cache tables of TYPO3). So this option is NOT the problem. This options gives you some more performance in general, but not only regarding a special TYPO3-Version. With this option you can speed up TYPO3 4.6, also.

So please: Do NOT turn off this options. It is for safety reasons and has nothing to do with this problem here. Sure, it speeds up your webpage, but it does NOT solve your problem.

Stefan

#13

Updated by Ernesto Baschny almost 9 years ago

  • Is Regression set to No

The compromise setting innodb_flush_log_at_trx_commit anyway is "2" which makes the transaction commit write to the log file, but sync to disk only each second. This makes up the best compromise as even on a mysql crash this stands (of course not ACID compliant).

Usually you are much interested in high performance all the time, and do not care about the very unlikely situations of a power outage or a kernel panic where you could loose "1 sec of data". In these situations you probably have other worries than this potential data lost (cache tables, logs ..).

The performance benefits are so tremendous so I cannot recommend anything else (except if your website is performing highly delicate bank transactions or you are obliged by law to ACID compliance).

Back to topic: I think the issue with the "first hit after a long time" has more to do with the the operating system not having important parts of the file system (i.e. mysql files) in RAM anymore, and mysql also having deallocated vital buffers already. This happen even more visible if your hoster is hosting multiple sites on the same machine which were hit in the meantime.

I also don't think this is a TYPO3 issue but something to tune on server side, and very specific to the specific setup the hoster is caring to run it (NFS, remote MySQL servers, etc).

#14

Updated by Stefan Froemken almost 9 years ago

To say "This is not a TYPO3 problem" is easy to say. But in my opinion there are many changes within TYPO3 just at starting period. Just one example: You have changed the load of TCA. Please check your profiling against Utility\ArrayUtility::arrayExport($GLOBALS['TCA']); This takes 1,1 Second on my local machine with SSD. What do you think what will it take on a HDD? 3 or 4 seconds?
I will create a patch for this here in this ticket and you all can check if it helps a little bit.

Stefan

#15

Updated by Gerrit Code Review almost 9 years ago

  • Status changed from Needs Feedback to Under Review

Patch set 1 for branch master has been pushed to the review server.
It is available at https://review.typo3.org/23969

#16

Updated by Alexander Opitz almost 9 years ago

The output of "cat /proc/meminfo" may help to see if this is more a problem of the server.

#17

Updated by Ernesto Baschny almost 9 years ago

@Stefan, don't get me wrong, I am not opposing working on performance in the 6.x branch, as I am sure we have still lot to do in this area which was a bit neglected lately. The original reporter mentioned that the problem was ext4 + the mysql setting, which is what "TYPO3 cannot do anything about it" (except maybe document some "known pitfals").

Great that you took some time to work on the performance front by profiling and getting into detail, Stefan. I support this work and hope others can jump in. I'll try to work on that area too a bit.

#18

Updated by Martin Borer almost 9 years ago

@Stefan,

Don't know if Herald's performance issues are solved now. I've had similar problems and for me it was obvious that the problem was on MySQL/Server side. If a simple insert query takes some seconds instead of milliseconds, there has to be something wrong with the server configuration/setup. It's possible that disabling writing barriers on ext4 systems will do the trick and you don't have to disable innodb_flush_log_at_trx_commit anymore. Have not tested this..

But improving Typo3 performance in general is always a good idea!.. ;)

#19

Updated by Gerrit Code Review almost 9 years ago

Patch set 2 for branch master has been pushed to the review server.
It is available at https://review.typo3.org/23969

#20

Updated by Gerrit Code Review almost 9 years ago

Patch set 3 for branch master has been pushed to the review server.
It is available at https://review.typo3.org/23969

#21

Updated by Gerrit Code Review almost 9 years ago

Patch set 4 for branch master has been pushed to the review server.
It is available at https://review.typo3.org/23969

#22

Updated by Christian Kuhn almost 9 years ago

+1 on Ernesto's analysis innodb_flush_log_at_trx_commit = 2 , we set this, too.

#23

Updated by Gerrit Code Review almost 9 years ago

Patch set 1 for branch TYPO3_6-1 has been pushed to the review server.
It is available at https://review.typo3.org/24051

#24

Updated by Markus Klein almost 9 years ago

  • Status changed from Under Review to Resolved
  • % Done changed from 0 to 100
#25

Updated by Gerrit Code Review almost 9 years ago

  • Status changed from Resolved to Under Review

Patch set 1 for branch TYPO3_6-0 has been pushed to the review server.
It is available at https://review.typo3.org/24052

#26

Updated by Markus Klein almost 9 years ago

  • Status changed from Under Review to Resolved
#27

Updated by Christian Kuhn almost 9 years ago

  • Parent task set to #52304
#28

Updated by Benni Mack almost 4 years ago

  • Status changed from Resolved to Closed

Also available in: Atom PDF