Bug #44518

Caching problem with Gifbuilder

Added by Martin W. about 7 years ago. Updated 10 months ago.

Status:
Closed
Priority:
Must have
Assignee:
-
Category:
Image Generation / GIFBUILDER
Start date:
2013-01-14
Due date:
% Done:

100%

TYPO3 Version:
8
PHP Version:
7.1
Tags:
Complexity:
Is Regression:
No
Sprint Focus:

Description

I had a cachingproblem with the Gifbuilder. Everytime I cleared the cache, a new image was generated although the TS-setup wasn't changed.

Here my the TS-Code:
file = GIFBUILDER
file{
quality = 100
XY = [10.w],[10.h]
format = png
10 = IMAGE
10{
file.import.data = register:ORIG_FILENAME
file.maxH = 760m
file.maxW = 940m
}
20 = IMAGE
20{
file = fileadmin/img/wasserT.png
offset = [10.w]/2 - 25, [10.h] - 45
}
}

(Without the second layer I hadn't the problem (without 20).)

In typo3/sysext/frontend/Classes/Imaging/GifBuilder.php
I figured out that the generated filename was different every run.

@ public function fileName($pre) {

...
// WARNING: In PHP5 I discovered that rendering with freetype of Japanese letters was totally corrupt.
// Not only the wrong glyphs are printed but also some memory stack overflow resulted in strange additional
// chars - and finally the reason for this investigation: The Bounding box data was changing all the time
// resulting in new images being generated all the time. With PHP4 it works fine.
return $this->tempPath . $pre . $meaningfulPrefix . \TYPO3\CMS\Core\Utility\GeneralUtility::shortMD5(serialize($this->setup)) . '.' . $this->extension();
}@

(the comment shows that the problem already appeared in a different way)

The problem is the md5hash of $this->setup. $this->setup is an array witch contains 2 objects ['BBOX']['originalFile'] and ['BBOX']['originalFile']. These objects are different after the cache was cleared.

My solution: copy $this->setup to $this->setupLight and remove these objects from setupLight. I've done this in:

@
public function start($conf, $data) {

...
// BBOX (the fileinfo) is to big and different each run: $this->fileName() generates a new file every run, because it uses a md5hash of this array
// ['BBOX']['originalFile'] and ['BBOX']['originalFile'] are Objects (to big and different each run)
// $this-setupLight is used now in $this->fileName()
//
$this->setupLight = $this->setup;
foreach ($sKeyArray as $theKey) {
$this->setupLight[$theKey . '.']['BBOX']['originalFile']='';
$this->setupLight[$theKey . '.']['BBOX']['processedFile']='';
}
//
// Get trivial data
$XY = \TYPO3\CMS\Core\Utility\GeneralUtility::intExplode(',', $this->setup['XY']);
$maxWidth = isset($this->setup['maxWidth.']) ? intval($this->cObj->stdWrap($this->setup['maxWidth'], $this->setup['maxWidth.'])) : intval($this->setup['maxWidth']);
$maxHeight = isset($this->setup['maxHeight.']) ? intval($this->cObj->stdWrap($this->setup['maxHeight'], $this->setup['maxHeight.'])) : intval($this->setup['maxHeight']);
$XY[0] = \TYPO3\CMS\Core\Utility\MathUtility::forceIntegerInRange($XY[0], 1, $maxWidth ? $maxWidth : 2000);
$XY[1] = \TYPO3\CMS\Core\Utility\MathUtility::forceIntegerInRange($XY[1], 1, $maxHeight ? $maxHeight : 2000);
$this->XY = $XY;
$this->w = $XY[0];
$this->h = $XY[1];
$this->OFFSET = \TYPO3\CMS\Core\Utility\GeneralUtility::intExplode(',', $this->setup['offset']);
// this sets the workArea
$this->setWorkArea($this->setup['workArea']);
// this sets the default to the current;
$this->defaultWorkArea = $this->workArea;
}
}
@

then I changed in function fileName():

return $this->tempPath . $pre . $meaningfulPrefix . \TYPO3\CMS\Core\Utility\GeneralUtility::shortMD5(serialize($this->setup)) . '.' . $this->extension();

to

return $this->tempPath . $pre . $meaningfulPrefix . \TYPO3\CMS\Core\Utility\GeneralUtility::shortMD5(serialize($this->setupLight)) . '.' . $this->extension();


Related issues

Related to TYPO3 Core - Bug #86947: Gifbuilder: combine images generates duplicate images of the same picture. Closed 2018-11-17

Associated revisions

Revision 7d5e5944 (diff)
Added by Markus Klösges 12 months ago

[BUGFIX] GifBuilder returns already generated images

When combining image files with gifbuilder, the hash to identify the
resulting file is now stable with respect to the fact whether the
files are cropped or scaled in the current request or already cropped
before. That leads to stable hashes whenever the same images are
processed with the same configuration, and allows reuse as intended.

Also ensure that fileInfo returned from ContentObjectRenderer contains
width and height information as int, when they are returned from
database as that may lead to different serialized representations of the
configuration

Resolves: #44518
Resolves: #86947
Resolves: #87224
Releases: 8.7, 9.5, master
Change-Id: I833585034cacaf5a0ad66ba3ff04ac3920421085
Reviewed-on: https://review.typo3.org/c/59644
Tested-by: TYPO3com <>
Tested-by: Guido Schmechel <>
Tested-by: Benni Mack <>
Reviewed-by: Oliver Klee <>
Reviewed-by: Guido Schmechel <>
Reviewed-by: Wouter Wolters <>
Reviewed-by: Stephan Großberndt <>
Reviewed-by: Benni Mack <>

Revision 8d584f74 (diff)
Added by Markus Klösges 12 months ago

[BUGFIX] GifBuilder returns already generated images

When combining image files with gifbuilder, the hash to identify the
resulting file is now stable with respect to the fact whether the
files are cropped or scaled in the current request or already cropped
before. That leads to stable hashes whenever the same images are
processed with the same configuration, and allows reuse as intended.

Also ensure that fileInfo returned from ContentObjectRenderer contains
width and height information as int, when they are returned from
database as that may lead to different serialized representations of the
configuration

Resolves: #44518
Resolves: #86947
Resolves: #87224
Releases: 8.7, 9.5, master
Change-Id: I833585034cacaf5a0ad66ba3ff04ac3920421085
Reviewed-on: https://review.typo3.org/c/59956
Tested-by: TYPO3com <>
Tested-by: Benni Mack <>
Reviewed-by: Oliver Hader <>
Reviewed-by: Benni Mack <>

Revision 5bb260cb (diff)
Added by Markus Klösges 12 months ago

[BUGFIX] GifBuilder returns already generated images

When combining image files with gifbuilder, the hash to identify the
resulting file is now stable with respect to the fact whether the
files are cropped or scaled in the current request or already cropped
before. That leads to stable hashes whenever the same images are
processed with the same configuration, and allows reuse as intended.

Also ensure that fileInfo returned from ContentObjectRenderer contains
width and height information as int, when they are returned from
database as that may lead to different serialized representations of the
configuration

Resolves: #44518
Resolves: #86947
Resolves: #87224
Releases: 8.7, 9.5, master
Change-Id: I833585034cacaf5a0ad66ba3ff04ac3920421085
Reviewed-on: https://review.typo3.org/c/59959
Reviewed-by: Oliver Klee <>
Reviewed-by: Benni Mack <>
Tested-by: TYPO3com <>
Tested-by: Benni Mack <>

History

#1 Updated by Mathias Schreiber about 5 years ago

  • Target version set to 7.2 (Frontend)
  • Is Regression set to No

#2 Updated by Benni Mack over 4 years ago

  • Target version changed from 7.2 (Frontend) to 7.4 (Backend)

#3 Updated by Susanne Moog over 4 years ago

  • Target version changed from 7.4 (Backend) to 7.5

#4 Updated by Benni Mack over 4 years ago

  • Target version changed from 7.5 to 7 LTS

#5 Updated by Mathias Schreiber over 4 years ago

  • Category changed from Caching to Image Cropping
  • Target version deleted (7 LTS)

#6 Updated by Susanne Moog over 2 years ago

  • Category changed from Image Cropping to Image Generation / GIFBUILDER

#7 Updated by Thomas Oliver Moll about 1 year ago

  • Priority changed from Should have to Must have
  • Target version set to Candidate for patchlevel
  • TYPO3 Version changed from 6.0 to 8
  • PHP Version changed from 5.3 to 7.1

Hi,
this is a very old Bug, but it was never really solved or it is back with TYPO 3 8.7.

We recently updated our huge Installations from 7.6 to 8.7 and we now run into issues with the Gifbulder.

We use Gifbuilder to render metadata into images. This data doesn't change often, but Gifbulder creates the same Image over and over with different hashes.
Like Martin, I traced it back to that [BBOX] part in the setup array that is sent through md5 to get the hash.
BBOX changes constantly and is HUGE! I wrote it via print_r into Text-Files and my disk filled up instantly because these text files had 32MB each and there were thousands of them.

The BBOX part is included by the Gifbuilder layers, that load Images. The patch above seemed not to be working for me, but what I did was very similar:

$setupLight=this.setup
unset($setupLight[10.][BBOX]['originalFile'])
unset($setupLight[10.][BBOX]['prosessedFile'])
unset($setupLight[10.][BBOX]['fileHash'])

and then I changed the code to use $setupLight instead of the original to create the Hash.
Our Images have the included Image always on Layer 10, so I did not have to iterate over the whole array.
Setting the entries to '' did not work for me, but unset worked fine.

I don't have the code at hand, so I had to write it here by memory, I hope there are not too many syntax errors after all, I'm not a php coder.

I'm not sure why the FileHash within the BBOX changes as well, I did not find any differences that (I thought) mattered. But this is hard to say when you compare two 32MB text files with scrambled order and many RECURSION in it.

Getting rid of the 2 subtrees and the filehash resulted in much smaller setup arrays (the text file size went down to 1800 Bytes and such like)

Please inform me, If I should open a new issue instead opening this 6 year old one.
I can provide a patch that works on the same iterating principle like Martins does, but I'm not sure if there is a smarter way not to use too much data for hashing, after all I firstly create a copy of that huge array, and delete the unnecessary bits afterwards

Cheers
TOM

#8 Updated by Thomas Oliver Moll about 1 year ago

  • Related to Bug #86947: Gifbuilder: combine images generates duplicate images of the same picture. added

#9 Updated by Gerrit Code Review about 1 year ago

  • Status changed from New to Under Review

Patch set 1 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/59644

#10 Updated by Gerrit Code Review about 1 year ago

Patch set 2 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/59644

#11 Updated by Gerrit Code Review about 1 year ago

Patch set 3 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/59644

#12 Updated by Gerrit Code Review about 1 year ago

Patch set 4 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/59644

#13 Updated by Gerrit Code Review about 1 year ago

Patch set 5 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/59644

#14 Updated by Gerrit Code Review about 1 year ago

Patch set 6 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/59644

#15 Updated by Gerrit Code Review about 1 year ago

Patch set 7 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/59644

#16 Updated by Gerrit Code Review about 1 year ago

Patch set 8 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/59644

#17 Updated by Gerrit Code Review about 1 year ago

Patch set 9 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/59644

#18 Updated by Gerrit Code Review about 1 year ago

Patch set 10 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/59644

#19 Updated by Gerrit Code Review about 1 year ago

Patch set 11 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/59644

#20 Updated by Gerrit Code Review about 1 year ago

Patch set 12 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/59644

#21 Updated by Gerrit Code Review 12 months ago

Patch set 13 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/59644

#22 Updated by Gerrit Code Review 12 months ago

Patch set 1 for branch 9.5 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/59956

#23 Updated by Markus Klösges 12 months ago

  • Status changed from Under Review to Resolved
  • % Done changed from 0 to 100

#24 Updated by Gerrit Code Review 12 months ago

  • Status changed from Resolved to Under Review

Patch set 1 for branch TYPO3_8-7 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/59959

#25 Updated by Markus Klösges 12 months ago

  • Status changed from Under Review to Resolved

#26 Updated by Gerrit Code Review 12 months ago

  • Status changed from Resolved to Under Review

Patch set 2 for branch TYPO3_8-7 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/59959

#27 Updated by Markus Klösges 12 months ago

  • Status changed from Under Review to Resolved

#28 Updated by Benni Mack 10 months ago

  • Status changed from Resolved to Closed

Also available in: Atom PDF