Bug #52374

Editing a sys_filemount is very slow

Added by Xavier Perseguers about 8 years ago. Updated 11 months ago.

Status:
New
Priority:
Should have
Category:
Performance
Target version:
-
Start date:
2013-09-30
Due date:
% Done:

0%

Estimated time:
TYPO3 Version:
10
PHP Version:
5.5
Tags:
performance, large-site
Complexity:
Is Regression:
No
Sprint Focus:

Description

When editing a filemount as popup window from the edit form of a be_group, the form takes about 9 sec to show up.

xhprof shows that this method is responsible for this amount of time:

TYPO3\CMS\Core\Resource\Service\UserFileMountService::renderTceformsSelectDropdown

Info

  • My list of sys_filemounts contains 2600 entries
  • I see 20 calls to TYPO3\CMS\Core\Resource\Folder::getSubfolders taking 4,728,752 ms
  • After 2nd recursion, still 298 calls to TYPO3\CMS\Core\Resource\Service\UserFileMountService::getSubfoldersForOptionList taking 4,505,079 ms

Actually there is much recursion but it's always TYPO3\CMS\Core\Resource\Folder::getSubfolders that eats so much time


Related issues

Related to TYPO3 Core - Feature #89762: Add pagination to forms listResolved

Actions
Related to TYPO3 Core - Epic #93547: Collection of problems with large sitesAccepted2021-02-19

Actions
Has duplicate TYPO3 Core - Bug #51402: Filemount record UI issuesAccepted2013-08-27

Actions
#1

Updated by Markus Klein almost 8 years ago

  • Parent task changed from #52304 to #52949
#2

Updated by Mathias Schreiber almost 7 years ago

  • Status changed from New to Needs Feedback
  • Assignee set to Mathias Schreiber

Hi Xavier,

I just have 600 folders, but for me it's fine on master.
Is this done?

#3

Updated by Xavier Perseguers almost 7 years ago

  • TYPO3 Version changed from 6.1 to 6.2
  • PHP Version changed from 5.3 to 5.5

Updated the PHP and TYPO3 version.

Using a real life server (that is, not with SSD but quite powerful anyway):

  • 3522 directories (8 at the 1st level)
  • about 27 sec to show up the edit form (edit pencil for a filemount record)

This is still not usable.

The same using SSD (development environment):

  • 2250 directories (8 at the 1st level)
  • about 3 sec to show up the edit form

This time it's totally OK of course but users are usually not using a development machine with local MySQL and SSD.

Didn't test with master, 6.2 is less than 1 year old and is a LTS with big projects so I'd like to see improvements for it if possible (we shouldn't forget that 6.2 is even not yet used by many bigger websites).

#4

Updated by Alexander Opitz over 6 years ago

  • Status changed from Needs Feedback to New
#5

Updated by Xavier Perseguers over 6 years ago

Did not debug again but this huge amount of time just appeared again this morning while configuring a Dropbox storage. Since every directory is searched for recursively, it took minutes (!) to fetch the structure of directories in my Dropbox account (total of about 50 directories?). So locally this may be slow but with remote storages this is even worse.

#6

Updated by Benni Mack about 4 years ago

  • Parent task deleted (#52949)
#7

Updated by Tymoteusz Motylewski over 1 year ago

  • Related to Bug #51402: Filemount record UI issues added
#8

Updated by Sybille Peters over 1 year ago

Creating a new filemount or editing a filemount creates a select box of all (!) directories in the storage (e.g. fileadmin).

We had this problem too. It can make managing filemounts very sluggish if you have lots of directories. Takes several minutes.

It would be nice if you could have the choice of filtering before TYPO3 starts scanning everything. Also, the usability of the long select list is not so great.


btw. we decided to split up the fileadmin into several storages on our site (mostly because of this problem). It takes the load off this problem. This also made sense for organizational reasons and has so far not had had any negative effect. It's made a bit difficult by the fact that you currently can't move entire directories through the GUI across storages (see #77988. #66514, #87662), but it is possible.

#9

Updated by Tymoteusz Motylewski over 1 year ago

I think the workaround would be to change the field type to simple input text input field.
The issue then is that there is no validation and user needs to put correct path.

Another workaround is to limit the number of levels for fetched folders.
In my case I've just commented the foreach in "getSubfoldersForOptionList()", as I just need one level of folders.

Another solution would be to utilize file browser with filed type group

'config' => [
                'type' => 'group',
                'internal_type' => 'folder',
                'maxitems' => 1,
            ]

The problem with this approach is that it saves data with slightly different format in db (path is prepended with storage id) e.g "2:/foldername/",
and there is no build in functionality right now to limit element browser to just one storage.

On the other hand maybe the whole UX could be then simplified:
- the "base" field being removed
- only one group field with element browser being there in which you select both storage and path in one go?

#10

Updated by Sybille Peters over 1 year ago

  1. About the input field: I think that would not be so good from usability view: User has to type in (or copy/paste) the complete path name
  2. Limit the recursion depth: I think this might work if it's possible to do this optionally (at the beginning you have to select the storage anyway and you get a reload.). But it's not so elegant. It seems technical and is not intuitive for everyone.
  3. with the group + internal_type=folder: This might be a less better solution than current for high recursion depth but few directories (because choosing them from the select list is 1 click). But for a lot of directories in the storage, it is a better solution than current.

About selecting both storage and path in one go: I think that would be nice for usability (because the current choose storage, reload is a bit clunky anyway).

All considered, I would probably prefer option 3 (group) and the combining of storage + folder selection.

#11

Updated by Sybille Peters about 1 year ago

#12

Updated by Sybille Peters about 1 year ago

  • Tags set to performance, large sites, many directories, many
#13

Updated by Sybille Peters about 1 year ago

  • Tags changed from performance, large sites, many directories, many to performance, large-site
#14

Updated by Richard Haeser 11 months ago

  • Related to deleted (Bug #51402: Filemount record UI issues)
#15

Updated by Richard Haeser 11 months ago

  • Has duplicate Bug #51402: Filemount record UI issues added
#16

Updated by Richard Haeser 11 months ago

  • Assignee changed from Mathias Schreiber to Richard Haeser
  • TYPO3 Version changed from 6.2 to 10
#17

Updated by Sybille Peters 10 months ago

  • Related to Epic #93547: Collection of problems with large sites added

Also available in: Atom PDF