Bug #87641
open
Increase StaticRangeMapper max range or make it configurable
0%
Description
I guess there are more projects than ours that deals with more than 100k news or records that needs to get paginated. When using f:widget.paginate and nice routes we do not want to use 100 elements per page because the amount of maximum pages is limited.
This part limits the amount of max pages inside pagination. Is there a special reason for the amount of 1000?
protected function buildRange(): array
{
$range = array_map('strval', range($this->start, $this->end));
if (count($range) > 1000) {
throw new \LengthException(
'Range is larger than 1000 items',
1537696771
);
}
For exmaple routeconfig used for pagination of EXT:news
routeEnhancers:
NewsPlugin:
type: Extbase
limitToPages:
- 1337
- 13370
extension: News
plugin: Pi1
routes:
- { routePath: '/{page}', _controller: 'News::list', _arguments: {'page': '@widget_0/currentPage'} }
- { routePath: '/{news_title}', _controller: 'News::detail', _arguments: {'news_title': 'news'} }
defaultController: 'News::list'
requirements:
page: '\d+'
defaults:
page: '0'
aspects:
page:
type: StaticRangeMapper
start: '1'
end: '1000'
news_title:
type: PersistedPatternMapper
tableName: tx_news_domain_model_news
routeFieldPattern: '^(?P<path_segment>.+)-(?P<uid>\d+)$'
routeFieldResult: '{path_segment}-{uid}'
Updated by Stefan Froemken about 5 years ago
They convert the range into an array and yes...there is a limitation of max array elements. I have tried it with a timestamp in range:
Start: 1000000000
End: 99999999999
Which results in:
Warning: range(): The supplied range exceeds the maximum array size: start=1000000000 end=99999999999 in /Users/froemken/htdocs/test.test/temp.php on line 2
I would prefer to remove this array range strval thing.
Stefan
Updated by Guido Schmechel about 5 years ago
The reason is described in the functions doc:
"The amount of items is limited to 1000 in order to avoid brute-force scenarios and the risk of cache-flooding."
+1 for removing the complete range construct. If you have, e.g. given a range from 4000 to 6000 for postals you haven't got a chance to define this.
Maybe we can change the exception to a log entry (warning)?
Updated by Oliver Hader about 5 years ago
Exactly, the range limitation has been added in order to avoid brute force attacks which lead to many cache entries - the StaticRangeMapper is just mitigating that to 1000, PageRouter has a similar maximum on StaticMappableAspectInterface implementations set to 10000 (for all affected mappers/aspects).
This has to been seen as alternative to know cHash which creates a signature over all arbitrary parameters in order to make the request unique and valid - thus, verification if the requested resource really exists (and triggers a new cache entry) is the primary goal here.
This could be achieved by using an additional mapper/aspect which could guarantee that, e.g.
- database lookup wether a record with a given uid exists -
SELECT COUNT(uid) FROM <table> WHERE uid=<parameter>(must exist) - having123 - database lookup wether some date representation exists -
SELECT COUNT(uid) FROM <table> WHERE DATE_FORMAT(tstamp, '%Y-%m')=<parameter>(must exist) - having2019-03
An alternative would be a mapper/aspect that is storing valid URLs in the database, like RealURL did - however that also had some side-effects on cHash and would not have worked having empty "caches" (URL not generated yet).
In general new implementations depend on their actual use case (e.g. "date and month filter", "color filter", "combination of categories", "alphabetical prefix - titles starting with 'A'", ...).
Updated by Oliver Hader about 5 years ago
Stefan Froemken wrote:
They convert the range into an array and yes...there is a limitation of max array elements. I have tried it with a timestamp in range:
Start: 1000000000
End: 99999999999Which results in:
Warning: range(): The supplied range exceeds the maximum array size: start=1000000000 end=99999999999 in /Users/froemken/htdocs/test.test/temp.php on line 2
I would prefer to remove this array range strval thing.
Why and what is your use case? How is the timestamp used in your scenario?
Updated by Oliver Hader about 5 years ago
I guess there are more projects than ours that deals with more than 100k news or records that needs to get paginated. When using f:widget.paginate and nice routes we do not want to use 100 elements per page because the amount of maximum pages is limited.
For this scenario the "valid values" would be in the range of [0; count(news) % page-size] where the news amount is dynamic (adding new records) and page-size is configured in TypoScript (but could be duplicated to route configuration in case it would be the same for all news plugins on that installation).
Updated by Benni Mack almost 5 years ago
- Target version changed from next-patchlevel to Candidate for patchlevel
Updated by Markus Pircher about 4 years ago
For this case i created my own StaticNumberRangeMapper for only numeric range and do not need an array
Updated by Bernhard Berger about 3 years ago
This hardcoded limit is very annoying and makes it impossible to generate URLs for more than 25 pages in my news archive module..
Updated by Jan Kornblum over 1 year ago
+ 1 for making the range constraint optional.
Updated by Stefanos Karasavvidis 7 months ago
Markus Pircher wrote in #note-7:
For this case i created my own StaticNumberRangeMapper for only numeric range and do not need an array
Just hit this issue. I have ~20000 posts, with 15 items per post. Results in over 1000 pages. Really annoying...
@Markus Pircher is your StaticNumberRangeMapper public?