Epic #88474

Page tree performance in 9.5

Added by Tymoteusz Motylewski over 1 year ago. Updated about 2 months ago.

Status:
New
Priority:
Must have
Assignee:
-
Category:
Pagetree
Target version:
-
Start date:
2018-11-16
Due date:
% Done:

86%

Sprint Focus:

Description

The new page tree delivered in v.9.5 has performance issues with bigger instalations e.g. 100k pages
Because of processing the whole page tree in one go.
We should make it possible to fetch one page tree level at once and process only these pages on PHP side.

For the discussion please join the slack channel

 #page-tree-performance 

https://typo3.slack.com/messages/CK20S7EKS


Subtasks

Bug #86945: Performance PageTree in WorkspaceClosedAlexander Opitz

Bug #88097: Page tree data fetching is using a huge amount of memory in PHPNew

Bug #88098: Page tree XHR is fetching huge JSON dataClosed

Bug #88943: Pagetree taking extremely long to load for editorsClosed

Bug #89687: Page tree sends unnecessary dataClosed

Bug #91036: Do not calculate beuser page permissions for every pageClosedTymoteusz Motylewski

Task #91037: Tune permission check in page treeClosedTymoteusz Motylewski


Related issues

Related to TYPO3 Core - Task #90104: Introduce prepared statement for BackendUtility::getPageForRootline Closed 2020-01-13
Related to TYPO3 Core - Task #90105: Allow full row to be passed into isInWebMount() Closed 2020-01-13
Related to TYPO3 Core - Bug #91883: Filtering in the page tree shows translated pages alongside the L=0 ones Rejected 2020-07-29
Related to TYPO3 Core - Bug #91884: Page tree filter has no delete "X", needs hitting enter and has no filter info box anymore, a big backwards step regarding usability Closed 2020-07-29
Related to TYPO3 Core - Bug #91885: Page-Tree filter does not reset Rejected 2020-07-29
Related to TYPO3 Core - Bug #91887: Page-Tree filter does not indicate its state New 2020-07-29
Related to TYPO3 Core - Bug #91902: Page tree not expanding deeper from initial view for editors Closed 2020-07-30
Related to TYPO3 Core - Bug #91728: Very slow tree loading if you have multiple workspaces TYPO3 v10 Closed 2020-06-30
Related to TYPO3 Core - Bug #90056: Selection not open after after leaving filtered mode New 2020-01-06
Related to TYPO3 Core - Bug #92027: Reset page tree filter after deletion of page record Accepted 2020-08-17

History

#1 Updated by Wolfangel Cyril over 1 year ago

Two idéas :

If i remember well, the 4.5 -> 6.2 team already addressed a similar issue and worked on the pagetree performance. It's probably a regression.

Other thing, maybe we could remove the loading animation would immediatly solve the bug as the 7.X version is doing the action in the background and nobody complained ?

#2 Updated by Johannes Wagner over 1 year ago

I have a large and deeply nested page tree. I want to install it in Typo9.5.

You could build the page tree asynchronously as in the previous Typo3 version. There, my page tree is quickly set up using the getNextTreeLevel method.

related question: https://stackoverflow.com/questions/56628254/how-can-the-typo3-backend-page-tree-be-constructed-asynchronously-in-typo3-v9-x

#3 Updated by Tymoteusz Motylewski over 1 year ago

  • Description updated (diff)

#4 Updated by Tymoteusz Motylewski over 1 year ago

  • Description updated (diff)

#5 Updated by Sybille Peters about 1 year ago

  • Sprint Focus set to On Location Sprint

Issues with performance for pagetree can be reproduced with less than 100 000 pages. Of course, this depends on resources used. But a page tree of about 25 000 pages can make backend pretty unmanageable requiring several seconds wait time after inserting a new page, changing the page title or just logging in and trying to access the page tree.

I tested this with an initial 9.5.9 installation with Introduction Package where about 23 000 pages were added (recursively).

A lot of time is spent in here:

TreeController::fetchDataAction():

...
foreach ($entryPoints as $page) {
  $items = array_merge($items, $this->pagesToFlatArray($page, (int)$page['uid']));
}

https://github.com/TYPO3/TYPO3.CMS/blob/3cb5ff17ad734e6677a4fc4f9dc26e2733d4e813/typo3/sysext/backend/Classes/Controller/Page/TreeController.php#L196

or specifically in the pagesToFlatArray function.

The items array is filled for all existing pages.

Loading the pages asynchronously would help a lot. If most parts of the page tree are collapsed, the page information is not really required (yet).

It also seems that costly operations are performed for minor changes, e.g. changing the page title of one page and saving.

#6 Updated by Christian Eßl about 1 year ago

  • Category set to Pagetree

#7 Updated by Susanne Moog 10 months ago

  • Sprint Focus deleted (On Location Sprint)

#8 Updated by Markus Klein 10 months ago

  • Related to Task #90104: Introduce prepared statement for BackendUtility::getPageForRootline added

#9 Updated by Markus Klein 10 months ago

  • Related to Task #90105: Allow full row to be passed into isInWebMount() added

#10 Updated by Markus Klein 10 months ago

With #90104 and #90105 I managed to reduce the loading time for a full page tree (refresh tree button) with 11000 pages by 60%.

#11 Updated by Marko Nix 9 months ago

I can add some investigation on that issue I think.

Setup

  • TYPO3 v9.5.13 and for testing switched to 9.5.14-dev, all results remain the same.
  • About 23k entries in pages table.
  • Normally there is "be_acl" loaded and used, for investigation deactivated the extension.
  • There are granular permissions set normally, to find an error all pages got an owner (we call him "owner") and is assigned to group "editors" with the permission bits: user 31, group 1, all 1.
  • All editors have the root page as mount point and additional mount points for pages where they have write access. Everything else is read-only... "Page show" for them.

Problem

Page tree loading runs into maximum execution tim (set to 240 seconds).
Using Backend user module to switch over to
  • "owner": no page tree
  • any "editors": no page tree

As administrator there is no problem, page tree loads immediately.

Ideas for solving - tried patches

https://review.typo3.org/c/Packages/TYPO3.CMS/+/62904 and https://review.typo3.org/c/Packages/TYPO3.CMS/+/62868 (same content?)
https://review.typo3.org/c/Packages/TYPO3.CMS/+/62867
https://review.typo3.org/c/Packages/TYPO3.CMS/+/62086

Some looks in the code behind

typo3/sysext/backend/Classes/Controller/Page/TreeController.php (approx line 671)

Changing

$entryPoint = $repository->getTree($entryPoint, function ($page) use ($backendUser) {
// check each page if the user has permission to access it
return $backendUser->doesUserHaveAccess($page, Permission::PAGE_SHOW);
});

to
$entryPoint = $repository->getTree($entryPoint, function ($page) use ($backendUser) {
// check each page if the user has permission to access it
return true;
});

"fixes" the problem, page tree loads immediately.

Digged a bit deeper and found the point where it may ends, but as far as i Know that is already known.

typo3/sysext/core/Classes/Authetication/BackendUserAuthentication.php (approx line 671)

if ($id > 0) {
        $wM = $this->returnWebmounts();
        $rL = BackendUtility::BEgetRootLine($id, ' AND ' . $readPerms);
        foreach ($rL as $v) {
        if ($v['uid'] && in_array($v['uid'], $wM)) {
           return $v['uid'];
        }
    }
}

As far as i can see for all 23k pages the complete rootline is loaded. I figured out that one number can change everything. But at that point I am out, I have no idea to "fix" it.

typo3/sysext/backend/Classes/Utility/BackendUtitlity.php (approx line 383)

    $output = $beGetRootLineCache[$ident];
} else {
    $loopCheck = *100*;
    $theRowArray = [];
    while ($uid != 0 && $loopCheck) {

Changing following let me load an incomplete page tree in a normal time.

    $output = $beGetRootLineCache[$ident];
} else {
    $loopCheck = *3*;
    $theRowArray = [];
    while ($uid != 0 && $loopCheck) {

May this helps.

#12 Updated by Uwe Trotzek 9 months ago

I'm currently upgrading an 8.7 instance to version 10.2 and I'm also experiencing the described performance issues with the page tree.
I have a page tree with ~11k pages with a max nesting level of 7 levels.

Even with the applied patch of https://review.typo3.org/c/Packages/TYPO3.CMS/+/62086 the page tries to validate the permissions of every page.

After debugging a bit I could find out that my setup obviously completely bypass the "load pages on demand solution".

/**
 * Returns JSON representing page tree
 *
 * @param ServerRequestInterface $request
 * @return ResponseInterface
 */
public function fetchDataAction(ServerRequestInterface $request): ResponseInterface
{
    $this->initializeConfiguration();

    $items = [];
    if (!empty($request->getQueryParams()['pid'])) {
        // Fetching a part of a page tree
        $entryPoints = $this->getAllEntryPointPageTrees((int)$request->getQueryParams()['pid']);
        $mountPid = (int)($request->getQueryParams()['mount'] ?? 0);
        $parentDepth = (int)($request->getQueryParams()['pidDepth'] ?? 0);
        $this->levelsToFetch = $parentDepth + 2;
        foreach ($entryPoints as $page) {
            $items = array_merge($items, $this->pagesToFlatArray($page, $mountPid, $parentDepth));
        }
    } else {
        $entryPoints = $this->getAllEntryPointPageTrees();
        foreach ($entryPoints as $page) {
            $items = array_merge($items, $this->pagesToFlatArray($page, (int)$page['uid']));
        }
    }

    return new JsonResponse($items);
}

In my setup the condition "!empty($request->getQueryParams()['pid'])" is always false, which leads to a loading of the entire pagetree. I don't know when the parameter "$request->getQueryParams()['pid']" is set.

I can provide some details to the setup which we are using:

  • We have only ONE mount point for all users. They all start at a root page.
  • To restrict access to certain subpages, we configure the visibility for certain pages via the access module only for a defined group
  • User who should "see" a page, are member of one or more groups.

I don't know if the "one mount point for all" setup is exotic, but this worked for us for many years in a bunch of projects.

#13 Updated by Markus Klein 9 months ago

Please comment your findings about a certain patch directly on Gerrit to the code. Thanks.

"one mount point for all" is definitely not exotic.

#14 Updated by Tymoteusz Motylewski 9 months ago

@Uwe
Did you make sure your browser is not caching the JS files?
The JS prt was modified to pass the pid param.
Please ensure you see the changed js file in your browser.

#15 Updated by Uwe Trotzek 9 months ago

Sorry, I was not at the office yesterday.
I just "moved" my comment to gerrit. Thanks for your responses so far.

#16 Updated by Christian Kuhn 3 months ago

  • Related to Feature #90931: Add option to disable version highlighting in the page tree added

#17 Updated by Christian Kuhn 3 months ago

  • Related to deleted (Feature #90931: Add option to disable version highlighting in the page tree)

#18 Updated by Stefan P 3 months ago

  • Related to Bug #91883: Filtering in the page tree shows translated pages alongside the L=0 ones added

#19 Updated by Stefan P 3 months ago

  • Related to Bug #91884: Page tree filter has no delete "X", needs hitting enter and has no filter info box anymore, a big backwards step regarding usability added

#20 Updated by Ingo Fabbri 3 months ago

  • Related to Bug #91885: Page-Tree filter does not reset added

#21 Updated by Ingo Fabbri 3 months ago

  • Related to Bug #91887: Page-Tree filter does not indicate its state added

#22 Updated by Oliver Hader 3 months ago

  • Related to Bug #91902: Page tree not expanding deeper from initial view for editors added

#23 Updated by Danilo Caccialanza 3 months ago

  • Related to Bug #91728: Very slow tree loading if you have multiple workspaces TYPO3 v10 added

#24 Updated by Riccardo De Contardi 2 months ago

  • Related to Bug #90056: Selection not open after after leaving filtered mode added

#25 Updated by Riccardo De Contardi 2 months ago

  • Related to Bug #92027: Reset page tree filter after deletion of page record added

Also available in: Atom PDF