Bug #87641
closed
Increase StaticRangeMapper max range or make it configurable
0%
Description
I guess there are more projects than ours that deals with more than 100k news or records that needs to get paginated. When using f:widget.paginate and nice routes we do not want to use 100 elements per page because the amount of maximum pages is limited.
This part limits the amount of max pages inside pagination. Is there a special reason for the amount of 1000?
protected function buildRange(): array
{
$range = array_map('strval', range($this->start, $this->end));
if (count($range) > 1000) {
throw new \LengthException(
'Range is larger than 1000 items',
1537696771
);
}
For exmaple routeconfig used for pagination of EXT:news
routeEnhancers:
NewsPlugin:
type: Extbase
limitToPages:
- 1337
- 13370
extension: News
plugin: Pi1
routes:
- { routePath: '/{page}', _controller: 'News::list', _arguments: {'page': '@widget_0/currentPage'} }
- { routePath: '/{news_title}', _controller: 'News::detail', _arguments: {'news_title': 'news'} }
defaultController: 'News::list'
requirements:
page: '\d+'
defaults:
page: '0'
aspects:
page:
type: StaticRangeMapper
start: '1'
end: '1000'
news_title:
type: PersistedPatternMapper
tableName: tx_news_domain_model_news
routeFieldPattern: '^(?P<path_segment>.+)-(?P<uid>\d+)$'
routeFieldResult: '{path_segment}-{uid}'
Updated by Stefan Froemken over 5 years ago
They convert the range into an array and yes...there is a limitation of max array elements. I have tried it with a timestamp in range:
Start: 1000000000
End: 99999999999
Which results in:
Warning: range(): The supplied range exceeds the maximum array size: start=1000000000 end=99999999999 in /Users/froemken/htdocs/test.test/temp.php on line 2
I would prefer to remove this array range strval thing.
Stefan
Updated by Guido Schmechel over 5 years ago
The reason is described in the functions doc:
"The amount of items is limited to 1000 in order to avoid brute-force scenarios and the risk of cache-flooding."
+1 for removing the complete range construct. If you have, e.g. given a range from 4000 to 6000 for postals you haven't got a chance to define this.
Maybe we can change the exception to a log entry (warning)?
Updated by Oliver Hader over 5 years ago
Exactly, the range limitation has been added in order to avoid brute force attacks which lead to many cache entries - the StaticRangeMapper is just mitigating that to 1000, PageRouter has a similar maximum on StaticMappableAspectInterface implementations set to 10000 (for all affected mappers/aspects).
This has to been seen as alternative to know cHash which creates a signature over all arbitrary parameters in order to make the request unique and valid - thus, verification if the requested resource really exists (and triggers a new cache entry) is the primary goal here.
This could be achieved by using an additional mapper/aspect which could guarantee that, e.g.
- database lookup wether a record with a given uid exists -
SELECT COUNT(uid) FROM <table> WHERE uid=<parameter>(must exist) - having123 - database lookup wether some date representation exists -
SELECT COUNT(uid) FROM <table> WHERE DATE_FORMAT(tstamp, '%Y-%m')=<parameter>(must exist) - having2019-03
An alternative would be a mapper/aspect that is storing valid URLs in the database, like RealURL did - however that also had some side-effects on cHash and would not have worked having empty "caches" (URL not generated yet).
In general new implementations depend on their actual use case (e.g. "date and month filter", "color filter", "combination of categories", "alphabetical prefix - titles starting with 'A'", ...).
Updated by Oliver Hader over 5 years ago
Stefan Froemken wrote:
They convert the range into an array and yes...there is a limitation of max array elements. I have tried it with a timestamp in range:
Start: 1000000000
End: 99999999999Which results in:
Warning: range(): The supplied range exceeds the maximum array size: start=1000000000 end=99999999999 in /Users/froemken/htdocs/test.test/temp.php on line 2
I would prefer to remove this array range strval thing.
Why and what is your use case? How is the timestamp used in your scenario?
Updated by Oliver Hader over 5 years ago
I guess there are more projects than ours that deals with more than 100k news or records that needs to get paginated. When using f:widget.paginate and nice routes we do not want to use 100 elements per page because the amount of maximum pages is limited.
For this scenario the "valid values" would be in the range of [0; count(news) % page-size] where the news amount is dynamic (adding new records) and page-size is configured in TypoScript (but could be duplicated to route configuration in case it would be the same for all news plugins on that installation).
Updated by Benni Mack over 5 years ago
- Target version changed from next-patchlevel to Candidate for patchlevel
Updated by Markus Pircher over 4 years ago
For this case i created my own StaticNumberRangeMapper for only numeric range and do not need an array
Updated by Bernhard Berger over 3 years ago
This hardcoded limit is very annoying and makes it impossible to generate URLs for more than 25 pages in my news archive module..
Updated by Jan Kornblum almost 2 years ago
+ 1 for making the range constraint optional.
Updated by Stefanos Karasavvidis about 1 year ago
Markus Pircher wrote in #note-7:
For this case i created my own StaticNumberRangeMapper for only numeric range and do not need an array
Just hit this issue. I have ~20000 posts, with 15 items per post. Results in over 1000 pages. Really annoying...
@Markus Pircher is your StaticNumberRangeMapper public?
Updated by Georg Ringer 5 months ago
just being curious:
20000 posts, with 15 items per post
how do you solve this from UX side? no user will click through 1000 pages
Updated by Stefanos Karasavvidis 5 months ago
Georg Ringer wrote in #note-11:
just being curious:
20000 posts, with 15 items per post
how do you solve this from UX side? no user will click through 1000 pages
indeed, noone will click through 1000 pages.
I do have a detailed search form.
Some people change the url and set a page number (I do this regularly).
And if I ever find the time, I plan to change the paging ui to have a "+10 pages" button instead of just the "next page".
I can not delete the posts. There has to be some kind of "browse through all posts" functionality. For now, I just changed to more posts per page, but I expect to hit the same problem sometimes in the next 2 years.
Updated by Georg Ringer 5 months ago
- Status changed from New to Rejected
hey! I am closing this issue because core really tries to provide a secure featureset. as mentioned already, no user will click through hundreds of pages and of course it is also still possible to either have no mapping for the pagination or use a custom mapper.
if you don't agree, feel free to contact me via slack!
Updated by Jan Kornblum 5 months ago
Please reopen. I‘ve sent you my arguments via Slack…