Bug #76581
closed
Backend Performance Flaws
100%
Description
Hello,
i did some profling on the TYPO3 backend lately and i witnessed two huge performance flaws.
My focus has been on the page-speed of loading an editing-page for a generic content-element (type textmedia in my case), i.e. on the url:
/typo3/alt_doc.php?returnUrl=%2Ftypo3%2Fsysext%2Fcms%2Flayout%2Fdb_layout.php%3Fid%3D63%23element-tt_content-48&edit[tt_content][48]=edit
Here is what i've found:
1. When loading an editing-page TYPO3 builds a tree-structure (don't aks me what it stores) and sorts children of tree-nodes. This sorting is implemented with TYPO3\CMS\Backend\Tree\SortedTreeNodeCollection. When append() is called on the collection it first literally appends the element to the tail of the list and then restores the order with a call to $this->asort(), followed by call to $this->normalize() which iterates the entire list again. The first screenshot shows a callee-map which was generated by QCacheGrind. You can easily see that approximately 50% of execution time are spent in this sorting-method and its sub-routines. I was able to eliminate this bottleneck by pre-sorting the input-list to SortedTreeNodeCollection and then pushing the elements directly to the collection instead of calling append. This has the further advantage that the internal-order is always consistent.
2. After this bottleneck was resolved the TypoScript-Parser made up the next big block. It consumes roughly 36% of the remaining 50% execution time. The parsing-code is absolutely messy and i didn't want to invest much time in there fixing the actual algorithms. Instead i build a cache around the parser.
The two bottlenecks some up to 68% of overall execution time. The second screenshot shows the callee-map after i did some optimization. The appended diff-file entails all the tweaks i have done. This reduced the time-to-first-bit from roughly 4s to 1.3s for one single request which is still frustrating but better than nothing. I'm not into the official TYPO3 workflow for contributions, but i'm open to do some polishing on this changeset if somebody can guide me through the official workflow.
Regards
Raoul
Files
Updated by Christian Kuhn over 8 years ago
Thanks for this analysis!
The tree stuff is indeed a huge bottleneck when opening records via "formEngine" (that's how the "edit a record" code stuff is called, the codebase went through a huge refactoring in 7). I guess the tree operations are triggered by TCA category tree elements in page and/or tt_content records, and this part is probably still mostly identical to the 6.2 codebase.
For the patch, we will not change that in 6.2 anymore since 6.2 is in "critical bug fixes only" mode already. However, the patches could be pushed to master and depending on the impact and risk we may decide to merge it into 7.6, too - this will be sorted out during the review process. I'd suggest to split the patch into two parts, though.
The tree part of the patch should get a solid review and analysis (this part of the code base is hard to understand and tends to break rather easily) and should be pushed as one patch part (could anyone step in here to help Raoul or push directly?).
For the TypoScriptParser part, I'm a bit more reluctant: The parser is used in many different scopes in both backend and frontend and just adding some cache around may lead to various issues we may not see in the first place. If that patch part makes it, it will probably go to master only, if at all. We may easily step into issues here since the TypoScript parser also relies on various global settings and has other nasty dependencies. It will be very hard to validate a simple cache in front is regression-free.
Updated by Markus Klein over 8 years ago
Regarding the TS I want to point you to the findings of Elmar Hinz, who is about to rewrite the parser from scratch.
See http://typoscript-parser.readthedocs.io/en/latest/
I'm actually a bit irritated by your numbers though, since TS parsing in FE makes up only a few percent. I've no clue why it should be so much worse in BE. Moreover I would expect the FormEngine code being the most expensive part, since it queries the DB a lot and is also the major component that is active for these requests.
Updated by
Updated by Sybille Peters almost 4 years ago
- Tags changed from performance, backend to performance, backend, large-site
Updated by Sybille Peters almost 4 years ago
- Related to Epic #93547: Collection of problems with large sites added
Updated by Gerrit Code Review about 2 years ago
- Status changed from New to Under Review
Patch set 5 for branch main of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/76443
Updated by
Updated by Gerrit Code Review about 2 years ago
Patch set 6 for branch main of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/76443
Updated by Gerrit Code Review about 2 years ago
Patch set 7 for branch main of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/76443
Updated by Gerrit Code Review about 2 years ago
Patch set 1 for branch 11.5 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/76473
Updated by Benni Mack about 2 years ago
- Status changed from Under Review to Resolved
- % Done changed from 0 to 100
Applied in changeset 971be9000b455391c314cfee251f8b3fe254008e.
Updated by Gerrit Code Review about 2 years ago
- Status changed from Resolved to Under Review
Patch set 2 for branch 11.5 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/76473
Updated by Benni Mack about 2 years ago
- Status changed from Under Review to Resolved
Applied in changeset bdeeb0cf96a9f86eeaf60db8f8beafc26718978c.
Updated by Christian Kuhn about 2 years ago
- Related to Bug #99007: Workspace Overlays are slow when having multiple records queried at once added
Updated by Benni Mack almost 2 years ago
- Status changed from Resolved to Closed