Bug #44518

Caching problem with Gifbuilder

Added by Martin W. over 9 years ago. Updated about 3 years ago.

Status:
Closed
Priority:
Must have
Assignee:
-
Category:
Image Generation / GIFBUILDER
Start date:
2013-01-14
Due date:
% Done:

100%

Estimated time:
TYPO3 Version:
8
PHP Version:
7.1
Tags:
Complexity:
Is Regression:
No
Sprint Focus:

Description

I had a cachingproblem with the Gifbuilder. Everytime I cleared the cache, a new image was generated although the TS-setup wasn't changed.

Here my the TS-Code:
file = GIFBUILDER
file{
quality = 100
XY = [10.w],[10.h]
format = png
10 = IMAGE
10{
file.import.data = register:ORIG_FILENAME
file.maxH = 760m
file.maxW = 940m
}
20 = IMAGE
20{
file = fileadmin/img/wasserT.png
offset = [10.w]/2 - 25, [10.h] - 45
}
}

(Without the second layer I hadn't the problem (without 20).)

In typo3/sysext/frontend/Classes/Imaging/GifBuilder.php
I figured out that the generated filename was different every run.

@ public function fileName($pre) {

...
// WARNING: In PHP5 I discovered that rendering with freetype of Japanese letters was totally corrupt.
// Not only the wrong glyphs are printed but also some memory stack overflow resulted in strange additional
// chars - and finally the reason for this investigation: The Bounding box data was changing all the time
// resulting in new images being generated all the time. With PHP4 it works fine.
return $this->tempPath . $pre . $meaningfulPrefix . \TYPO3\CMS\Core\Utility\GeneralUtility::shortMD5(serialize($this->setup)) . '.' . $this->extension();
}@

(the comment shows that the problem already appeared in a different way)

The problem is the md5hash of $this->setup. $this->setup is an array witch contains 2 objects ['BBOX']['originalFile'] and ['BBOX']['originalFile']. These objects are different after the cache was cleared.

My solution: copy $this->setup to $this->setupLight and remove these objects from setupLight. I've done this in:

@
public function start($conf, $data) {

...
// BBOX (the fileinfo) is to big and different each run: $this->fileName() generates a new file every run, because it uses a md5hash of this array
// ['BBOX']['originalFile'] and ['BBOX']['originalFile'] are Objects (to big and different each run)
// $this-setupLight is used now in $this->fileName()
//
$this->setupLight = $this->setup;
foreach ($sKeyArray as $theKey) {
$this->setupLight[$theKey . '.']['BBOX']['originalFile']='';
$this->setupLight[$theKey . '.']['BBOX']['processedFile']='';
}
//
// Get trivial data
$XY = \TYPO3\CMS\Core\Utility\GeneralUtility::intExplode(',', $this->setup['XY']);
$maxWidth = isset($this->setup['maxWidth.']) ? intval($this->cObj->stdWrap($this->setup['maxWidth'], $this->setup['maxWidth.'])) : intval($this->setup['maxWidth']);
$maxHeight = isset($this->setup['maxHeight.']) ? intval($this->cObj->stdWrap($this->setup['maxHeight'], $this->setup['maxHeight.'])) : intval($this->setup['maxHeight']);
$XY[0] = \TYPO3\CMS\Core\Utility\MathUtility::forceIntegerInRange($XY[0], 1, $maxWidth ? $maxWidth : 2000);
$XY[1] = \TYPO3\CMS\Core\Utility\MathUtility::forceIntegerInRange($XY[1], 1, $maxHeight ? $maxHeight : 2000);
$this->XY = $XY;
$this->w = $XY[0];
$this->h = $XY[1];
$this->OFFSET = \TYPO3\CMS\Core\Utility\GeneralUtility::intExplode(',', $this->setup['offset']);
// this sets the workArea
$this->setWorkArea($this->setup['workArea']);
// this sets the default to the current;
$this->defaultWorkArea = $this->workArea;
}
}
@

then I changed in function fileName():

return $this->tempPath . $pre . $meaningfulPrefix . \TYPO3\CMS\Core\Utility\GeneralUtility::shortMD5(serialize($this->setup)) . '.' . $this->extension();

to

return $this->tempPath . $pre . $meaningfulPrefix . \TYPO3\CMS\Core\Utility\GeneralUtility::shortMD5(serialize($this->setupLight)) . '.' . $this->extension();


Related issues

Related to TYPO3 Core - Bug #86947: Gifbuilder: combine images generates duplicate images of the same picture.Closed2018-11-17

Actions
#1

Updated by Mathias Schreiber over 7 years ago

  • Target version set to 7.2 (Frontend)
  • Is Regression set to No
#2

Updated by Benni Mack almost 7 years ago

  • Target version changed from 7.2 (Frontend) to 7.4 (Backend)
#3

Updated by Susanne Moog almost 7 years ago

  • Target version changed from 7.4 (Backend) to 7.5
#4

Updated by Benni Mack over 6 years ago

  • Target version changed from 7.5 to 7 LTS
#5

Updated by Mathias Schreiber over 6 years ago

  • Category changed from Caching to Image Cropping
  • Target version deleted (7 LTS)
#6

Updated by Susanne Moog over 4 years ago

  • Category changed from Image Cropping to Image Generation / GIFBUILDER
#7

Updated by Thomas Oliver Moll over 3 years ago

  • Priority changed from Should have to Must have
  • Target version set to Candidate for patchlevel
  • TYPO3 Version changed from 6.0 to 8
  • PHP Version changed from 5.3 to 7.1

Hi,
this is a very old Bug, but it was never really solved or it is back with TYPO 3 8.7.

We recently updated our huge Installations from 7.6 to 8.7 and we now run into issues with the Gifbulder.

We use Gifbuilder to render metadata into images. This data doesn't change often, but Gifbulder creates the same Image over and over with different hashes.
Like Martin, I traced it back to that [BBOX] part in the setup array that is sent through md5 to get the hash.
BBOX changes constantly and is HUGE! I wrote it via print_r into Text-Files and my disk filled up instantly because these text files had 32MB each and there were thousands of them.

The BBOX part is included by the Gifbuilder layers, that load Images. The patch above seemed not to be working for me, but what I did was very similar:

$setupLight=this.setup
unset($setupLight[10.][BBOX]['originalFile'])
unset($setupLight[10.][BBOX]['prosessedFile'])
unset($setupLight[10.][BBOX]['fileHash'])

and then I changed the code to use $setupLight instead of the original to create the Hash.
Our Images have the included Image always on Layer 10, so I did not have to iterate over the whole array.
Setting the entries to '' did not work for me, but unset worked fine.

I don't have the code at hand, so I had to write it here by memory, I hope there are not too many syntax errors after all, I'm not a php coder.

I'm not sure why the FileHash within the BBOX changes as well, I did not find any differences that (I thought) mattered. But this is hard to say when you compare two 32MB text files with scrambled order and many RECURSION in it.

Getting rid of the 2 subtrees and the filehash resulted in much smaller setup arrays (the text file size went down to 1800 Bytes and such like)

Please inform me, If I should open a new issue instead opening this 6 year old one.
I can provide a patch that works on the same iterating principle like Martins does, but I'm not sure if there is a smarter way not to use too much data for hashing, after all I firstly create a copy of that huge array, and delete the unnecessary bits afterwards

Cheers
TOM

#8

Updated by Thomas Oliver Moll over 3 years ago

  • Related to Bug #86947: Gifbuilder: combine images generates duplicate images of the same picture. added
#9

Updated by Gerrit Code Review over 3 years ago

  • Status changed from New to Under Review

Patch set 1 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/59644

#10

Updated by Gerrit Code Review over 3 years ago

Patch set 2 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/59644

#11

Updated by Gerrit Code Review over 3 years ago

Patch set 3 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/59644

#12

Updated by Gerrit Code Review over 3 years ago

Patch set 4 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/59644

#13

Updated by Gerrit Code Review over 3 years ago

Patch set 5 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/59644

#14

Updated by Gerrit Code Review over 3 years ago

Patch set 6 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/59644

#15

Updated by Gerrit Code Review over 3 years ago

Patch set 7 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/59644

#16

Updated by Gerrit Code Review over 3 years ago

Patch set 8 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/59644

#17

Updated by Gerrit Code Review over 3 years ago

Patch set 9 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/59644

#18

Updated by Gerrit Code Review over 3 years ago

Patch set 10 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/59644

#19

Updated by Gerrit Code Review over 3 years ago

Patch set 11 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/59644

#20

Updated by Gerrit Code Review over 3 years ago

Patch set 12 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/59644

#21

Updated by Gerrit Code Review about 3 years ago

Patch set 13 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/59644

#22

Updated by Gerrit Code Review about 3 years ago

Patch set 1 for branch 9.5 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/59956

#23

Updated by Markus Klösges about 3 years ago

  • Status changed from Under Review to Resolved
  • % Done changed from 0 to 100
#24

Updated by Gerrit Code Review about 3 years ago

  • Status changed from Resolved to Under Review

Patch set 1 for branch TYPO3_8-7 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/59959

#25

Updated by Markus Klösges about 3 years ago

  • Status changed from Under Review to Resolved
#26

Updated by Gerrit Code Review about 3 years ago

  • Status changed from Resolved to Under Review

Patch set 2 for branch TYPO3_8-7 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/59959

#27

Updated by Markus Klösges about 3 years ago

  • Status changed from Under Review to Resolved
#28

Updated by Benni Mack about 3 years ago

  • Status changed from Resolved to Closed

Also available in: Atom PDF