Bug #87224

GifBuilder creates filename hashes over and over

Added by Thomas Oliver Moll 6 months ago. Updated about 2 months ago.

Status:
Closed
Priority:
Must have
Assignee:
-
Category:
Image Generation / GIFBUILDER
Target version:
Start date:
2018-12-19
Due date:
% Done:

100%

TYPO3 Version:
8
PHP Version:
7.1
Tags:
Complexity:
Is Regression:
Sprint Focus:

Description

Hi,

we recently updated all our installations from 7.6 to 8.7. and we experienced a problem:

/typo3temp/assets/images overflowed with images.

Our filesystem can handle only ~22000 files per folder, and our Installations are huge. But that was not the main problem. Some files were in there many hundered times, in one case over 7000 times.
Each time with a different hash.

I'm not a PHP developer and not too familiar with the innards of the TYPO3 core, but my analysis showed, that the hash is generated in GifBuilder.php by putting $setup through md5. Not a problem per se.
But in our case $setup was up to 32MB big (when written als Text to disk) - and even worse differed from call to call.
The culprit seemed to be a sub-array of $setup ['BBOX'] containing huge amounts of data, recursions and a changing order of its elements.

The code is nearly unchanged since TYPO3 6.2 (oldest copy I had) and I even found a open issue going back some 6 years:
https://forge.typo3.org/issues/44518#change-391424

I did not receive any answer when I updated that old issue, so I do it here. I hope double-posting this is not too bad.

I now have a working work-around in place, but I don't think it's the best solution:

in function fileName in typo3/sysext/frontend/Classes/Imaging/GifBuilder.php I changed the line 708

        return 'typo3temp/' . $pre . $filePrefix . '_' . GeneralUtility::shortMD5(serialize($this->setup)) . '.' . $this->extension();

to

        $setupLight=$this->setup;        
        foreach ($setupLight as $theKey =>$theValue) {          
            unset($setupLight[$theKey.'.']['BBOX']['originalFile']);
            unset($setupLight[$theKey.'.']['BBOX']['processedFile']);
            unset($setupLight[$theKey.'.']['BBOX']['fileCacheHash']);          
        }
        return 'typo3temp/' . $pre . $filePrefix . '_' . GeneralUtility::shortMD5(serialize($setupLight)) . '.' . $this->extension();

This is roughly the same approach of the mentioned issue 44518.

Since the problem seemingly to arrived for us with 8.7 (or at least worsened to be noticeable) and since the code in GifBuilder.php hasn't really changed in a long time, I think the problem lies more in having too much data in that BBOX array.
To my understanding it is for the "BoundingBox" of an image layer, but it contains so much more 'stuff'.

I can not rule out that this array gets this big because of something we do somewhere else, but I have no idea, what this could be.

I have to admit, I struggled with getting GifBuilder to work with FAL when we switched to TYPO3 7.6 so my quirky typoscript does some 'things' to the image ID to produce a version of it with text layered on top:

lib.imageplus=FILES
lib.imageplus{
  # The Fluid Template gives an array of "fields", using currentValueKey we set the current object,
  # which contains the id of the file we want
  files.current=1

  # The String we get starts with "file:" in front of the id, so we cut it away using substing
  files.stdWrap.substring=5

  # But sometimes this code is called in a way, that gives us the id without "file:" 
  # So we check if the given String already is a number. There seems to be no isNumber() function though
  # so we use this abomination that gives us the same result 
  # This was written to be compatible with data stored in databases written with TYPO3 6.2, 7.6 and 8.7
  # and was not re-written for a long time so we had to cope with old obsolete data formats
  # The version for 8.7 now uses a ViewHelper to get the ID from strings containing t3://
  # So this code may be not needed anymore.
  files.stdWrap.substring.if.isLessThan=100000000
  files.stdWrap.substring.if.value={field:image}
  files.stdWrap.substring.if.value.insertData=1

  renderObj = COA
  renderObj{
    # After entering the scope of GIFBUILDER we seem to be unable to access the file 
    # object so we store everything we need in the register
    10 = LOAD_REGISTER
    10 {
      fileid.data = file:current:uid
      creator.data = file:current:creator
      status.data = file:current:status
      visible.data = file:current:visible

      storage.data = file:current:storage

      identifier.data = file:current:identifier
      identifier_hash.data = file:current:identifier_hash
      width.data= file:current:width
      height.data= file:current:height
      filecaption.data = file:current:caption
      desc.data = file:current:description
      title.data = file:current:title
      alternative.data = file:current:alternative
      link{
    typolink.parameter.data = file:current:link
    typolink.returnLast = url

      }
    }
    20=IMG_RESOURCE
    20{
      file = GIFBUILDER
      file {
    # The size of the to-be-created images is equal to the image we load in layer 10:
    XY=[10.w],[10.h]
    XY.override.if.isFalse.data = register:visible
    XY.override=[15.w],[15.h]
    format = jpg
    quality=90
    10 = IMAGE
    10 {
      if.isTrue.data = register:visible
      file {
        #We get the ID from the register and use it as the source of the image:
        import.data = register:fileid
        # we get the wanted width from the ViewHelper in the field array
        maxW={field:width}
        maxW.insertData=1
      }
    }

...


Related issues

Related to TYPO3 Core - Bug #86947: Gifbuilder: combine images generates duplicate images of the same picture. Closed 2018-11-17

Associated revisions

Revision 7d5e5944 (diff)
Added by Markus Klösges 4 months ago

[BUGFIX] GifBuilder returns already generated images

When combining image files with gifbuilder, the hash to identify the
resulting file is now stable with respect to the fact whether the
files are cropped or scaled in the current request or already cropped
before. That leads to stable hashes whenever the same images are
processed with the same configuration, and allows reuse as intended.

Also ensure that fileInfo returned from ContentObjectRenderer contains
width and height information as int, when they are returned from
database as that may lead to different serialized representations of the
configuration

Resolves: #44518
Resolves: #86947
Resolves: #87224
Releases: 8.7, 9.5, master
Change-Id: I833585034cacaf5a0ad66ba3ff04ac3920421085
Reviewed-on: https://review.typo3.org/c/59644
Tested-by: TYPO3com <>
Tested-by: Guido Schmechel <>
Tested-by: Benni Mack <>
Reviewed-by: Oliver Klee <>
Reviewed-by: Guido Schmechel <>
Reviewed-by: Wouter Wolters <>
Reviewed-by: Stephan Großberndt <>
Reviewed-by: Benni Mack <>

Revision 8d584f74 (diff)
Added by Markus Klösges 4 months ago

[BUGFIX] GifBuilder returns already generated images

When combining image files with gifbuilder, the hash to identify the
resulting file is now stable with respect to the fact whether the
files are cropped or scaled in the current request or already cropped
before. That leads to stable hashes whenever the same images are
processed with the same configuration, and allows reuse as intended.

Also ensure that fileInfo returned from ContentObjectRenderer contains
width and height information as int, when they are returned from
database as that may lead to different serialized representations of the
configuration

Resolves: #44518
Resolves: #86947
Resolves: #87224
Releases: 8.7, 9.5, master
Change-Id: I833585034cacaf5a0ad66ba3ff04ac3920421085
Reviewed-on: https://review.typo3.org/c/59956
Tested-by: TYPO3com <>
Tested-by: Benni Mack <>
Reviewed-by: Oliver Hader <>
Reviewed-by: Benni Mack <>

Revision 5bb260cb (diff)
Added by Markus Klösges 4 months ago

[BUGFIX] GifBuilder returns already generated images

When combining image files with gifbuilder, the hash to identify the
resulting file is now stable with respect to the fact whether the
files are cropped or scaled in the current request or already cropped
before. That leads to stable hashes whenever the same images are
processed with the same configuration, and allows reuse as intended.

Also ensure that fileInfo returned from ContentObjectRenderer contains
width and height information as int, when they are returned from
database as that may lead to different serialized representations of the
configuration

Resolves: #44518
Resolves: #86947
Resolves: #87224
Releases: 8.7, 9.5, master
Change-Id: I833585034cacaf5a0ad66ba3ff04ac3920421085
Reviewed-on: https://review.typo3.org/c/59959
Reviewed-by: Oliver Klee <>
Reviewed-by: Benni Mack <>
Tested-by: TYPO3com <>
Tested-by: Benni Mack <>

History

#1 Updated by Guido Schmechel 6 months ago

  • Related to Bug #86947: Gifbuilder: combine images generates duplicate images of the same picture. added

#2 Updated by Gerrit Code Review 5 months ago

  • Status changed from New to Under Review

Patch set 1 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/59644

#3 Updated by Gerrit Code Review 5 months ago

Patch set 2 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/59644

#4 Updated by Gerrit Code Review 5 months ago

Patch set 3 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/59644

#5 Updated by Gerrit Code Review 5 months ago

Patch set 4 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/59644

#6 Updated by Gerrit Code Review 5 months ago

Patch set 5 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/59644

#7 Updated by Gerrit Code Review 5 months ago

Patch set 6 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/59644

#8 Updated by Gerrit Code Review 5 months ago

Patch set 7 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/59644

#9 Updated by Gerrit Code Review 5 months ago

Patch set 8 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/59644

#10 Updated by Gerrit Code Review 5 months ago

Patch set 9 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/59644

#11 Updated by Gerrit Code Review 4 months ago

Patch set 10 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/59644

#12 Updated by Gerrit Code Review 4 months ago

Patch set 11 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/59644

#13 Updated by Gerrit Code Review 4 months ago

Patch set 12 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/59644

#14 Updated by Gerrit Code Review 4 months ago

Patch set 13 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/59644

#15 Updated by Gerrit Code Review 4 months ago

Patch set 1 for branch 9.5 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/59956

#16 Updated by Markus Klösges 4 months ago

  • Status changed from Under Review to Resolved
  • % Done changed from 0 to 100

#17 Updated by Gerrit Code Review 4 months ago

  • Status changed from Resolved to Under Review

Patch set 1 for branch TYPO3_8-7 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/59959

#18 Updated by Markus Klösges 4 months ago

  • Status changed from Under Review to Resolved

#19 Updated by Gerrit Code Review 4 months ago

  • Status changed from Resolved to Under Review

Patch set 2 for branch TYPO3_8-7 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/59959

#20 Updated by Markus Klösges 4 months ago

  • Status changed from Under Review to Resolved

#21 Updated by Benni Mack about 2 months ago

  • Status changed from Resolved to Closed

Also available in: Atom PDF