Project

General

Profile

Actions

Epic #93547

open

Collection of problems with large sites

Added by Sybille Peters about 3 years ago. Updated over 2 years ago.

Status:
Accepted
Priority:
Should have
Assignee:
-
Category:
Miscellaneous
Target version:
-
Start date:
2021-02-19
Due date:
% Done:

0%

Estimated time:
Sprint Focus:

Description

I used the following tags in sub-issues - where appropriate.

tags : "large-site", "memory hog", "prepared statement", "placeholderlimit", "memory", "performance", "throttle", "usability" (or UI related things)

All issues related to large site are tagged with large-site.


Can be many pages, many files, many directories, many categories, etc. or a combination.

The problems are mostly related to:

  • performance and resource (memory) problems.
  • exceptions, e.g. database query - “Prepared statement contains too many placeholders”
  • usability: e.g. long, unusable lists (see sys_filemount: long list of directories + performance issue, unable to filter for specific pages in pagetree filter etc.)

Can the question of large sites be addressed in general?

e.g.

  • consider large sites when creating concepts for new features
  • create a reference "large site"
  • test patches on “large sites”
  • define hard limits / soft limits (if possible) - e.g. are there specific known limitations, can we say TYPO3 easily runs with x number of pages (redirects, sites, files, be user groups etc.) in a typical installation (e.g. with a reference site)

Related issues 31 (14 open17 closed)

Related to TYPO3 Core - Bug #57953: Rendering time of HMENU is really bad, maybe bug?Closed2014-04-15

Actions
Related to TYPO3 Core - Bug #76581: Backend Performance FlawsClosed2016-06-10

Actions
Related to TYPO3 Core - Bug #78258: List view CSV export goes out of memoryClosed2016-10-12

Actions
Related to TYPO3 Core - Bug #92493: linkvalidator: scheduler task + list of broken links dies if too many pages ("Prepared statement contains too many placeholders")Closed2020-10-06

Actions
Related to TYPO3 Core - Bug #36744: Workspaces are unusable on larger installationsClosedOliver Hader2012-08-12

Actions
Related to TYPO3 Core - Feature #89015: Lazy Loading for TCA / Backend View to improve performanceNew2019-08-26

Actions
Related to TYPO3 Core - Bug #81996: Read ony select field 'renderType' => 'selectSingle' renders all available itemsNew2017-07-28

Actions
Related to TYPO3 Core - Epic #89759: Performance improvements in Forms moduleNew2019-11-24

Actions
Related to TYPO3 Core - Bug #84558: TYPO3 sys_category problem with performance/slow opening category in BEClosed2018-03-29

Actions
Related to TYPO3 Core - Task #90143: Redirects: Poor performance of redirect matching for large redirects tableUnder Review2020-01-17

Actions
Related to TYPO3 Core - Bug #92033: "Page tree error: Got unexpected reponse" with filter - allowed memory size exhaustedClosed2020-08-17

Actions
Related to TYPO3 Core - Bug #92036: New behaviour of page tree filter might more easily submit "monster queries" or too many queriesClosed2020-08-17

Actions
Related to TYPO3 Core - Feature #78760: Make pagetree panel resizable/expandable on large screensClosed2016-11-21

Actions
Related to TYPO3 Core - Feature #92576: Page Tree filter: make it possible to explicitly filter by uid - to go to a specific page in the pagetreeClosed2020-10-16

Actions
Related to TYPO3 Core - Feature #89762: Add pagination to forms listNew

Actions
Related to TYPO3 Core - Bug #52374: Editing a sys_filemount is very slowClosedRichard Haeser2013-09-30

Actions
Related to TYPO3 Core - Bug #88873: ext: form hangs (run into 500 error) when a lot of files are in the fileadminClosed2019-11-05

Actions
Related to TYPO3 Core - Bug #91372: Filelist module stops working if a lot of files in one directoryNew2020-05-12

Actions
Related to TYPO3 Core - Epic #88474: Page tree performance in 9.5Closed2018-11-16

Actions
Related to TYPO3 Core - Bug #88134: Lots a database queries to sys_refindex cause slow page savingNew2019-04-10

Actions
Related to TYPO3 Core - Bug #84051: SelectTree (Category Tree) - add horizontal scrollingNew2018-02-26

Actions
Related to TYPO3 Core - Bug #88672: SlugHelper->isUniqueInSite() slow for installations with many sites and similar URL structures Under Review2019-07-02

Actions
Related to TYPO3 Core - Task #93305: Long lists of parameters in QueryBuilder handled differently ("too many placeholders")New2021-01-18

Actions
Related to TYPO3 Core - Bug #93937: Live search really slow for non admin users in large TYPO3 websitesNew2021-04-16

Actions
Related to TYPO3 Core - Epic #95690: Performance issues when hosting a large amount of websites, and optimizations propositionsAccepted2021-10-18

Actions
Related to TYPO3 Core - Bug #86859: Search with indexed_search plugin throws exception: Prepared statement contains too many placeholders Resolved2018-11-05

Actions
Related to TYPO3 Core - Bug #96042: Row update wizard may consume too much memory for tables with many records and contentClosed2021-11-20

Actions
Related to TYPO3 Core - Task #89287: Make linkvalidator crawling politeNew2019-09-26

Actions
Related to TYPO3 Core - Feature #97017: Allow LiveSearch to be disabled via configurationAccepted2022-02-24

Actions
Related to TYPO3 Core - Bug #99326: DataHandler - Process too many recordsResolved2022-12-09

Actions
Related to TYPO3 Core - Bug #90518: DB problems with rootline cache: DELETE cf_cache_rootline double-JOIN with cf_cache_rootline_tags take excessively longResolved2020-02-24

Actions
Actions #1

Updated by Sybille Peters about 3 years ago

content grouped by category in https://notes.typo3.org/xpZwXmWrRVi5bRCJ392p6Q here.

Actions #2

Updated by Sybille Peters about 3 years ago

  • Related to Bug #57953: Rendering time of HMENU is really bad, maybe bug? added
Actions #3

Updated by Sybille Peters about 3 years ago

  • Related to Bug #76581: Backend Performance Flaws added
Actions #4

Updated by Sybille Peters about 3 years ago

  • Related to Bug #78258: List view CSV export goes out of memory added
Actions #5

Updated by Sybille Peters about 3 years ago

  • Related to Bug #92493: linkvalidator: scheduler task + list of broken links dies if too many pages ("Prepared statement contains too many placeholders") added
Actions #6

Updated by Sybille Peters about 3 years ago

  • Related to Bug #36744: Workspaces are unusable on larger installations added
Actions #7

Updated by Sybille Peters about 3 years ago

  • Related to Feature #89015: Lazy Loading for TCA / Backend View to improve performance added
Actions #8

Updated by Sybille Peters about 3 years ago

  • Related to Bug #81996: Read ony select field 'renderType' => 'selectSingle' renders all available items added
Actions #9

Updated by Sybille Peters about 3 years ago

  • Related to Epic #89759: Performance improvements in Forms module added
Actions #10

Updated by Sybille Peters about 3 years ago

  • Related to Bug #84558: TYPO3 sys_category problem with performance/slow opening category in BE added
Actions #11

Updated by Sybille Peters about 3 years ago

  • Related to Task #90143: Redirects: Poor performance of redirect matching for large redirects table added
Actions #12

Updated by Sybille Peters about 3 years ago

  • Related to Bug #92033: "Page tree error: Got unexpected reponse" with filter - allowed memory size exhausted added
Actions #13

Updated by Sybille Peters about 3 years ago

  • Related to Bug #92036: New behaviour of page tree filter might more easily submit "monster queries" or too many queries added
Actions #14

Updated by Sybille Peters about 3 years ago

  • Related to Feature #78760: Make pagetree panel resizable/expandable on large screens added
Actions #15

Updated by Sybille Peters about 3 years ago

  • Related to Feature #92576: Page Tree filter: make it possible to explicitly filter by uid - to go to a specific page in the pagetree added
Actions #16

Updated by Sybille Peters about 3 years ago

Actions #17

Updated by Sybille Peters about 3 years ago

  • Related to Bug #52374: Editing a sys_filemount is very slow added
Actions #18

Updated by Sybille Peters about 3 years ago

  • Related to Bug #88873: ext: form hangs (run into 500 error) when a lot of files are in the fileadmin added
Actions #19

Updated by Sybille Peters about 3 years ago

  • Related to Bug #91372: Filelist module stops working if a lot of files in one directory added
Actions #20

Updated by Riccardo De Contardi about 3 years ago

  • Related to Epic #88474: Page tree performance in 9.5 added
Actions #21

Updated by Riccardo De Contardi about 3 years ago

  • Related to Bug #88134: Lots a database queries to sys_refindex cause slow page saving added
Actions #22

Updated by Riccardo De Contardi about 3 years ago

  • Related to Bug #84051: SelectTree (Category Tree) - add horizontal scrolling added
Actions #23

Updated by Georg Ringer about 3 years ago

  • Status changed from New to Accepted
Actions #24

Updated by Stefan P about 3 years ago

  • Related to Bug #88672: SlugHelper->isUniqueInSite() slow for installations with many sites and similar URL structures added
Actions #25

Updated by Stefan P about 3 years ago

A very important change in the heads of core devs or in the "mentality" in the future must be that new features are not developed simply "parallel" to the core, but so that it respect the already existing architecture. For example did the redirects module that came with v9 totally ignore user permissions (if you are allowed to see the module you just see all redirects from all domains and not just the ones you are allowed to see - no simple "bug", just not implemented at all).

Or #88672 just assumed totally arbitrary (with hardcoded magic number) that there are a maximum of 100 pages with the same name.

Or #91995 just assumed that never an unconfigured domain (in sys_domain or Site YAML) can (accidentally!) point to a TYPO3 system, leading to arbitrary output.

This shows that many new features are implmented from a single POV: single/few domain setups with mostly admin BE users. This mentaility automatically also leads to the problems in this epic: performance problems.

Actions #26

Updated by Sybille Peters about 3 years ago

@Stefan P I don't think it is intentionally. Some new features / changes are already pretty complex in themselves and considering user permissions, languages, workspaces etc. makes it even more complex. Really thinking everything through for multiple users, multiple sites, large sites - not an easy task.

But I agree with you that these things are not always considered, which is the reason I started this issue.

I think finding a method for testing with large sites (also with multiple users, multiple domains etc.) would at least make potential problems detected early.

What I also find missing is a definition of the limits. I realize, this is not easy to do. In some cases it also depends on the memory allocated and on other factors. But if you have a reference site and say, ok TYPO3 11 was tested with this reference site with 100 000 pages, 2000 backend users, 300 backend groups, 5 sites, 2000 redirects etc. Don't know if this is realistic, it might require some looking into.

And maybe additionally post some numbers, where problems may be expected, e.g. are there currently any hard limits?

And also finding best practices for how to solve things in general in the code might also help.


From my practical experience with a "large site" not everything is a huge problem and it depends. But I did expect TYPO3 to work well with large sites and as an enterprise solution, I think it should. Most of the problems are just a nuisance or do not affect us but some have been a big problem and have made time - and cost-intensive workarounds necessary.

Actions #27

Updated by Sybille Peters about 3 years ago

  • Related to Task #93305: Long lists of parameters in QueryBuilder handled differently ("too many placeholders") added
Actions #28

Updated by Torben Hansen almost 3 years ago

  • Related to Bug #93937: Live search really slow for non admin users in large TYPO3 websites added
Actions #29

Updated by Sybille Peters over 2 years ago

  • Related to Epic #95690: Performance issues when hosting a large amount of websites, and optimizations propositions added
Actions #30

Updated by Sybille Peters over 2 years ago

  • Description updated (diff)
Actions #31

Updated by Sybille Peters over 2 years ago

  • Related to Bug #86859: Search with indexed_search plugin throws exception: Prepared statement contains too many placeholders added
Actions #32

Updated by Sybille Peters over 2 years ago

  • Related to Bug #96042: Row update wizard may consume too much memory for tables with many records and content added
Actions #33

Updated by Sybille Peters over 2 years ago

  • Related to Task #89287: Make linkvalidator crawling polite added
Actions #34

Updated by Sybille Peters over 2 years ago

  • Description updated (diff)
Actions #35

Updated by Stefan P almost 2 years ago

  • Related to Feature #97017: Allow LiveSearch to be disabled via configuration added
Actions #36

Updated by Guillaume Germain over 1 year ago

  • Related to Bug #99326: DataHandler - Process too many records added
Actions #37

Updated by Sybille Peters 11 months ago

  • Related to Bug #90518: DB problems with rootline cache: DELETE cf_cache_rootline double-JOIN with cf_cache_rootline_tags take excessively long added
Actions

Also available in: Atom PDF