Bug #87641

Increase StaticRangeMapper max range or make it configurable

Added by Christian Stern 3 months ago. Updated about 1 month ago.

Status:
New
Priority:
Must have
Assignee:
-
Category:
Link Handling, Site Handling & Routing
Target version:
Start date:
2019-02-04
Due date:
% Done:

0%

TYPO3 Version:
9
PHP Version:
7.2
Tags:
Complexity:
easy
Is Regression:
Sprint Focus:

Description

I guess there are more projects than ours that deals with more than 100k news or records that needs to get paginated. When using f:widget.paginate and nice routes we do not want to use 100 elements per page because the amount of maximum pages is limited.

This part limits the amount of max pages inside pagination. Is there a special reason for the amount of 1000?

 protected function buildRange(): array
    {
        $range = array_map('strval', range($this->start, $this->end));
        if (count($range) > 1000) {
            throw new \LengthException(
                'Range is larger than 1000 items',
                1537696771
            );
        }

For exmaple routeconfig used for pagination of EXT:news

routeEnhancers:
  NewsPlugin:
    type: Extbase
    limitToPages: 
      - 1337
      - 13370
    extension: News
    plugin: Pi1
    routes:
      - { routePath: '/{page}', _controller: 'News::list', _arguments: {'page': '@widget_0/currentPage'} }
      - { routePath: '/{news_title}', _controller: 'News::detail', _arguments: {'news_title': 'news'} }
    defaultController: 'News::list'
    requirements:
      page: '\d+'
    defaults:
      page: '0'
    aspects:
      page:
        type: StaticRangeMapper
        start: '1'
        end: '1000'
      news_title:
        type: PersistedPatternMapper
        tableName: tx_news_domain_model_news
        routeFieldPattern: '^(?P<path_segment>.+)-(?P<uid>\d+)$'
        routeFieldResult: '{path_segment}-{uid}'

History

#1 Updated by Stefan Froemken about 2 months ago

They convert the range into an array and yes...there is a limitation of max array elements. I have tried it with a timestamp in range:

Start: 1000000000
End: 99999999999

Which results in:

Warning: range(): The supplied range exceeds the maximum array size: start=1000000000 end=99999999999 in /Users/froemken/htdocs/test.test/temp.php on line 2

I would prefer to remove this array range strval thing.

Stefan

#2 Updated by Guido Schmechel about 2 months ago

The reason is described in the functions doc:
"The amount of items is limited to 1000 in order to avoid brute-force scenarios and the risk of cache-flooding."

+1 for removing the complete range construct. If you have, e.g. given a range from 4000 to 6000 for postals you haven't got a chance to define this.

Maybe we can change the exception to a log entry (warning)?

#3 Updated by Oliver Hader about 1 month ago

Exactly, the range limitation has been added in order to avoid brute force attacks which lead to many cache entries - the StaticRangeMapper is just mitigating that to 1000, PageRouter has a similar maximum on StaticMappableAspectInterface implementations set to 10000 (for all affected mappers/aspects).

This has to been seen as alternative to know cHash which creates a signature over all arbitrary parameters in order to make the request unique and valid - thus, verification if the requested resource really exists (and triggers a new cache entry) is the primary goal here.

This could be achieved by using an additional mapper/aspect which could guarantee that, e.g.

  • database lookup wether a record with a given uid exists - SELECT COUNT(uid) FROM <table> WHERE uid=<parameter> (must exist) - having 123
  • database lookup wether some date representation exists - SELECT COUNT(uid) FROM <table> WHERE DATE_FORMAT(tstamp, '%Y-%m')=<parameter> (must exist) - having 2019-03

An alternative would be a mapper/aspect that is storing valid URLs in the database, like RealURL did - however that also had some side-effects on cHash and would not have worked having empty "caches" (URL not generated yet).

In general new implementations depend on their actual use case (e.g. "date and month filter", "color filter", "combination of categories", "alphabetical prefix - titles starting with 'A'", ...).

#4 Updated by Oliver Hader about 1 month ago

Stefan Froemken wrote:

They convert the range into an array and yes...there is a limitation of max array elements. I have tried it with a timestamp in range:

Start: 1000000000
End: 99999999999

Which results in:

Warning: range(): The supplied range exceeds the maximum array size: start=1000000000 end=99999999999 in /Users/froemken/htdocs/test.test/temp.php on line 2

I would prefer to remove this array range strval thing.

Why and what is your use case? How is the timestamp used in your scenario?

#5 Updated by Oliver Hader about 1 month ago

I guess there are more projects than ours that deals with more than 100k news or records that needs to get paginated. When using f:widget.paginate and nice routes we do not want to use 100 elements per page because the amount of maximum pages is limited.

For this scenario the "valid values" would be in the range of [0; count(news) % page-size] where the news amount is dynamic (adding new records) and page-size is configured in TypoScript (but could be duplicated to route configuration in case it would be the same for all news plugins on that installation).

Also available in: Atom PDF