Epic #93547

Collection of problems with large sites

Added by Sybille Peters 8 months ago. Updated 8 months ago.

Status:
Accepted
Priority:
Should have
Assignee:
-
Category:
Miscellaneous
Target version:
-
Start date:
2021-02-19
Due date:
% Done:

0%

Estimated time:
Sprint Focus:

Description

Can be many pages, many files, many directories, many categories, etc. or a combination.

The problems are mostly related to:

  • performance and resource (memory) problems.
  • exceptions, e.g. database query - “Prepared statement contains too many placeholders”
  • usability: e.g. long, unusable lists (see sys_filemount: long list of directories + performance issue, unable to filter for specific pages in pagetree filter etc.)

Can the question of large sites be addressed in general?

e.g.

  • consider large sites when creating concepts for new features
  • create a reference "large site"
  • test patches on “large sites”

Related issues

Related to TYPO3 Core - Bug #57953: Rendering time of HMENU is really bad, maybe bug?Needs Feedback2014-04-15

Actions
Related to TYPO3 Core - Bug #76581: Backend Performance FlawsNew2016-06-10

Actions
Related to TYPO3 Core - Bug #78258: List view CSV export goes out of memoryAccepted2016-10-12

Actions
Related to TYPO3 Core - Bug #92493: linkvalidator: scheduler task + list of broken links dies if too many pages ("Prepared statement contains too many placeholders")Under Review2020-10-06

Actions
Related to TYPO3 Core - Bug #36744: Workspaces are unusable on larger installationsAcceptedOliver Hader2012-08-12

Actions
Related to TYPO3 Core - Feature #89015: Lazy Loading for TCA / Backend View to improve performanceNew2019-08-26

Actions
Related to TYPO3 Core - Bug #81996: Read ony select field 'renderType' => 'selectSingle' renders all available itemsNew2017-07-28

Actions
Related to TYPO3 Core - Epic #89759: Performance improvements in Forms moduleNew2019-11-24

Actions
Related to TYPO3 Core - Bug #84558: TYPO3 sys_category problem with performance/slow opening category in BENew2018-03-29

Actions
Related to TYPO3 Core - Task #90143: Redirects: Poor performance of redirect matching for large redirects tableUnder Review2020-01-17

Actions
Related to TYPO3 Core - Bug #92033: "Page tree error: Got unexpected reponse" with filter - allowed memory size exhaustedNew2020-08-17

Actions
Related to TYPO3 Core - Bug #92036: New behaviour of page tree filter might more easily submit "monster queries" or too many queriesAccepted2020-08-17

Actions
Related to TYPO3 Core - Feature #78760: Make pagetree panel resizable/expandable on large screensClosed2016-11-21

Actions
Related to TYPO3 Core - Feature #92576: Page Tree filter: make it possible to explicitly filter by uidUnder Review2020-10-16

Actions
Related to TYPO3 Core - Feature #89762: Add pagination to forms listResolved

Actions
Related to TYPO3 Core - Bug #52374: Editing a sys_filemount is very slowNewRichard Haeser2013-09-30

Actions
Related to TYPO3 Core - Bug #88873: ext: form hangs (run into 500 error) when a lot of files are in the fileadminClosed2019-11-05

Actions
Related to TYPO3 Core - Bug #91372: Filelist module stops working if a lot of files in one directoryNew2020-05-12

Actions
Related to TYPO3 Core - Epic #88474: Page tree performance in 9.5Closed2018-11-16

Actions
Related to TYPO3 Core - Bug #88134: Lots a database queries to sys_refindex cause slow page savingNew2019-04-10

Actions
Related to TYPO3 Core - Bug #84051: SelectTree (Category Tree) - add horizontal scrollingNew2018-02-26

Actions
Related to TYPO3 Core - Bug #88672: SlugHelper->isUniqueInSite() slow for installations with many sites and similar URL structures Under Review2019-07-02

Actions
Related to TYPO3 Core - Task #93305: Long lists of parameters in QueryBuilder handled differently ("too many placeholders")New2021-01-18

Actions
Related to TYPO3 Core - Bug #93937: Live search really slow for non admin users in large TYPO3 websitesNew2021-04-16

Actions
Related to TYPO3 Core - Epic #95690: Performance issues when hosting a large amount of websites, and optimizations propositionsAccepted2021-10-18

Actions
#1

Updated by Sybille Peters 8 months ago

content grouped by category in https://notes.typo3.org/xpZwXmWrRVi5bRCJ392p6Q here.

#2

Updated by Sybille Peters 8 months ago

  • Related to Bug #57953: Rendering time of HMENU is really bad, maybe bug? added
#3

Updated by Sybille Peters 8 months ago

  • Related to Bug #76581: Backend Performance Flaws added
#4

Updated by Sybille Peters 8 months ago

  • Related to Bug #78258: List view CSV export goes out of memory added
#5

Updated by Sybille Peters 8 months ago

  • Related to Bug #92493: linkvalidator: scheduler task + list of broken links dies if too many pages ("Prepared statement contains too many placeholders") added
#6

Updated by Sybille Peters 8 months ago

  • Related to Bug #36744: Workspaces are unusable on larger installations added
#7

Updated by Sybille Peters 8 months ago

  • Related to Feature #89015: Lazy Loading for TCA / Backend View to improve performance added
#8

Updated by Sybille Peters 8 months ago

  • Related to Bug #81996: Read ony select field 'renderType' => 'selectSingle' renders all available items added
#9

Updated by Sybille Peters 8 months ago

  • Related to Epic #89759: Performance improvements in Forms module added
#10

Updated by Sybille Peters 8 months ago

  • Related to Bug #84558: TYPO3 sys_category problem with performance/slow opening category in BE added
#11

Updated by Sybille Peters 8 months ago

  • Related to Task #90143: Redirects: Poor performance of redirect matching for large redirects table added
#12

Updated by Sybille Peters 8 months ago

  • Related to Bug #92033: "Page tree error: Got unexpected reponse" with filter - allowed memory size exhausted added
#13

Updated by Sybille Peters 8 months ago

  • Related to Bug #92036: New behaviour of page tree filter might more easily submit "monster queries" or too many queries added
#14

Updated by Sybille Peters 8 months ago

  • Related to Feature #78760: Make pagetree panel resizable/expandable on large screens added
#15

Updated by Sybille Peters 8 months ago

  • Related to Feature #92576: Page Tree filter: make it possible to explicitly filter by uid added
#16

Updated by Sybille Peters 8 months ago

#17

Updated by Sybille Peters 8 months ago

  • Related to Bug #52374: Editing a sys_filemount is very slow added
#18

Updated by Sybille Peters 8 months ago

  • Related to Bug #88873: ext: form hangs (run into 500 error) when a lot of files are in the fileadmin added
#19

Updated by Sybille Peters 8 months ago

  • Related to Bug #91372: Filelist module stops working if a lot of files in one directory added
#20

Updated by Riccardo De Contardi 8 months ago

  • Related to Epic #88474: Page tree performance in 9.5 added
#21

Updated by Riccardo De Contardi 8 months ago

  • Related to Bug #88134: Lots a database queries to sys_refindex cause slow page saving added
#22

Updated by Riccardo De Contardi 8 months ago

  • Related to Bug #84051: SelectTree (Category Tree) - add horizontal scrolling added
#23

Updated by Georg Ringer 8 months ago

  • Status changed from New to Accepted
#24

Updated by Stefan P 8 months ago

  • Related to Bug #88672: SlugHelper->isUniqueInSite() slow for installations with many sites and similar URL structures added
#25

Updated by Stefan P 8 months ago

A very important change in the heads of core devs or in the "mentality" in the future must be that new features are not developed simply "parallel" to the core, but so that it respect the already existing architecture. For example did the redirects module that came with v9 totally ignore user permissions (if you are allowed to see the module you just see all redirects from all domains and not just the ones you are allowed to see - no simple "bug", just not implemented at all).

Or #88672 just assumed totally arbitrary (with hardcoded magic number) that there are a maximum of 100 pages with the same name.

Or #91995 just assumed that never an unconfigured domain (in sys_domain or Site YAML) can (accidentally!) point to a TYPO3 system, leading to arbitrary output.

This shows that many new features are implmented from a single POV: single/few domain setups with mostly admin BE users. This mentaility automatically also leads to the problems in this epic: performance problems.

#26

Updated by Sybille Peters 8 months ago

@Stefan P I don't think it is intentionally. Some new features / changes are already pretty complex in themselves and considering user permissions, languages, workspaces etc. makes it even more complex. Really thinking everything through for multiple users, multiple sites, large sites - not an easy task.

But I agree with you that these things are not always considered, which is the reason I started this issue.

I think finding a method for testing with large sites (also with multiple users, multiple domains etc.) would at least make potential problems detected early.

What I also find missing is a definition of the limits. I realize, this is not easy to do. In some cases it also depends on the memory allocated and on other factors. But if you have a reference site and say, ok TYPO3 11 was tested with this reference site with 100 000 pages, 2000 backend users, 300 backend groups, 5 sites, 2000 redirects etc. Don't know if this is realistic, it might require some looking into.

And maybe additionally post some numbers, where problems may be expected, e.g. are there currently any hard limits?

And also finding best practices for how to solve things in general in the code might also help.


From my practical experience with a "large site" not everything is a huge problem and it depends. But I did expect TYPO3 to work well with large sites and as an enterprise solution, I think it should. Most of the problems are just a nuisance or do not affect us but some have been a big problem and have made time - and cost-intensive workarounds necessary.

#27

Updated by Sybille Peters 8 months ago

  • Related to Task #93305: Long lists of parameters in QueryBuilder handled differently ("too many placeholders") added
#28

Updated by Torben Hansen 6 months ago

  • Related to Bug #93937: Live search really slow for non admin users in large TYPO3 websites added
#29

Updated by Sybille Peters 1 day ago

  • Related to Epic #95690: Performance issues when hosting a large amount of websites, and optimizations propositions added

Also available in: Atom PDF