Bug #66614

FAL processed files are deleted

Added by Markus Klein over 4 years ago. Updated about 2 years ago.

Status:
Closed
Priority:
Must have
Assignee:
Category:
File Abstraction Layer (FAL)
Target version:
Start date:
2015-04-27
Due date:
% Done:

100%

TYPO3 Version:
6.2
PHP Version:
Tags:
Complexity:
hard
Is Regression:
No
Sprint Focus:
Stabilization Sprint

Description

How to reproduce

  • set `config.noScaleUp = 0` in TS
  • add a magic image to RTE
  • preview the page in FE

Result: Image broken

Root cause

Setting config.noScaleUp causes this line to be executed in typo3/sysext/frontend/Classes/Page/PageGenerator.php:88

$GLOBALS['TYPO3_CONF_VARS']['GFX']['im_noScaleUp'] = isset($GLOBALS['TSFE']->config['config']['noScaleUp'])
   ? '' . $GLOBALS['TSFE']->config['config']['noScaleUp']
   : $GLOBALS['TYPO3_CONF_VARS']['GFX']['im_noScaleUp'];

This makes ['GFX']['im_noScaleUp'] a string, while in BE (when the preview image is generated in RTE) this is only loaded from install tool, where this is a boolean value.

This tiny difference has the effect that the calculated checksums used to determine whether a processed file already exists mismatch and the already generated processed file is deleted from disk.


Related issues

Related to TYPO3 Core - Task #67071: Add cleanup tool for processed files to Install Tool Closed 2015-05-21
Related to TYPO3 Core - Bug #67385: SQL parser does not support string as join condition Closed 2015-06-10

Associated revisions

Revision ab13da3d (diff)
Added by Markus Klein over 4 years ago

[BUGFIX] Avoid unnecessary generation of processed files

Removing [GFX] off the checksum data used to identify a processed file
ensures that data type inconsistencies within [GFX] do not cause a
regeneration of the processed file.

A new upgrade wizard for the Install Tool is added to migrate existing
processed files without the need to regenerate those.

Resolves: #66614
Releases: master, 6.2
Change-Id: Id90ce923b26ee726d15cf3d0cfcafc4ec4d7405d
Reviewed-on: http://review.typo3.org/39644
Reviewed-by: Andreas Fernandez <>
Reviewed-by: Xavier Perseguers <>
Tested-by: Xavier Perseguers <>
Reviewed-by: Stefan Neufeind <>
Reviewed-by: Alexander Opitz <>
Tested-by: Alexander Opitz <>

Revision 9067813e (diff)
Added by Xavier Perseguers over 4 years ago

[BUGFIX] SQL parser does not support string as join condition

The SQL parser is capable of parsing

JOIN T1 ON T1.some_field = T2.some_other_field

and

JOIN T1 ON T1.some_field = 4

but not

JOIN T1 ON T1.some_field = 'some string'

Change-Id: I7806699ea44685ab717cc0e8a75f7b53a1b2b1cc
Resolves: #67385
Related: #66614
Releases: master, 6.2
Reviewed-on: http://review.typo3.org/40144
Reviewed-by: Andreas Fernandez <>
Tested-by: Andreas Fernandez <>
Reviewed-by: Alexander Opitz <>
Reviewed-by: Xavier Perseguers <>
Tested-by: Xavier Perseguers <>

Revision 6d44ed99 (diff)
Added by Xavier Perseguers over 4 years ago

[BUGFIX] SQL parser does not support string as join condition

The SQL parser is capable of parsing

JOIN T1 ON T1.some_field = T2.some_other_field

and

JOIN T1 ON T1.some_field = 4

but not

JOIN T1 ON T1.some_field = 'some string'

Change-Id: I7806699ea44685ab717cc0e8a75f7b53a1b2b1cc
Resolves: #67385
Related: #66614
Releases: master, 6.2
Reviewed-on: http://review.typo3.org/40145
Reviewed-by: Xavier Perseguers <>
Tested-by: Xavier Perseguers <>

Revision 66ab2d68 (diff)
Added by Alexander Opitz over 4 years ago

[BUGFIX] Avoid unnecessary generation of processed files

Removing [GFX] off the checksum data used to identify a processed file
ensures that data type inconsistencies within [GFX] do not cause a
regeneration of the processed file.

A new upgrade wizard for the Install Tool is added to migrate existing
processed files without the need to regenerate those.

Resolves: #66614
Releases: master, 6.2
Change-Id: Id90ce923b26ee726d15cf3d0cfcafc4ec4d7405d
Reviewed-on: http://review.typo3.org/40143
Reviewed-by: Xavier Perseguers <>
Tested-by: Xavier Perseguers <>

Revision acdca0a6 (diff)
Added by Morton Jonuschat over 4 years ago

[BUGFIX] Make ProcessedFileChecksumUpdate compatible with DBAL

Use explicit CAST statements in the SQL queries performed by the
updater to allow other DBMS to compare a VARCHAR with an INT field.

The SQLparser has been extended to support CAST statements as part
of a JOIN clause.

Resolves: #67708
Related: #66614
Releases: master
Change-Id: I4a13eae92290b9ce8149aa5ef93df1305c02715a
Reviewed-on: http://review.typo3.org/40576
Reviewed-by: Markus Klein <>
Reviewed-by: Andreas Fernandez <>
Tested-by: Andreas Fernandez <>
Reviewed-by: Xavier Perseguers <>
Reviewed-by: Christian Kuhn <>
Tested-by: Christian Kuhn <>

History

#1 Updated by Gerrit Code Review over 4 years ago

  • Status changed from Accepted to Under Review

Patch set 1 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/38972

#2 Updated by Markus Klein over 4 years ago

  • Description updated (diff)

#3 Updated by Markus Klein over 4 years ago

  • Status changed from Under Review to Accepted
  • Priority changed from Must have to Should have
  • Target version deleted (next-patchlevel)
  • Complexity changed from no-brainer to hard

Abandoned patch.

The issue is way more dramatic.

We have the situation that user experience will differ for BE and FE, if GFX|im_noScaleUp does not match config.noScaleUp.
BE and FE would behave differently.

#4 Updated by Markus Klein over 4 years ago

  • Priority changed from Should have to Must have
  • Target version set to next-patchlevel

#56147 deals with the issue that processed files are not regenerated. This is related here, since with this bug you can trigger the other bug easily.
Actually, if #56147 would be working, I would have never noticed this bug here.

Regarding this issue: We decided to remove the [GFX] part from the checksum and add a cleanup-util to the Install Tool.

#5 Updated by Markus Klein over 4 years ago

  • Target version changed from next-patchlevel to 6.2.13
  • Sprint Focus set to Stabilization Sprint

I figured out the whole process now.

  1. After inserting the magic file into RTE
  2. a ProcessedFile is generated for the thumb
  3. When calling FE, it tries to render the file but (due to the data type issue) a new thumb (with new filename) is generated (so a new ProcessedFile)
  4. Having the click-enlarge static template loaded of RTE, causes it to render the name of the original processed file (from ->cObj->parameters, which contains the parsed RTE content), which has been deleted by the last step, when the new processed file was generated.
  5. In FE it looks like the thumb-regeneration has failed, which is not true. Just the filename of the original thumb is used, which does not exist anymore.

Ultima ratio:
Fixing the configuration change issue (removing GFX from the checksum) fixes the issue. The remaining FAL handling is totally fine in this case.

Implementation steps are:
  1. Remove [GFX] from the checksum generated in \TYPO3\CMS\Core\Resource\Processing\AbstractGraphicalTask::getChecksumData()
  2. Add a cleanup utility to Install-Tool, which allows to remove all processed files (#67071)

#6 Updated by Gerrit Code Review over 4 years ago

  • Status changed from Accepted to Under Review

Patch set 1 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/39644

#7 Updated by Gerrit Code Review over 4 years ago

Patch set 2 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/39644

#8 Updated by Gerrit Code Review over 4 years ago

Patch set 3 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/39644

#9 Updated by Gerrit Code Review over 4 years ago

Patch set 4 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/39644

#10 Updated by Gerrit Code Review over 4 years ago

Patch set 5 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/39644

#11 Updated by Gerrit Code Review over 4 years ago

Patch set 6 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/39644

#12 Updated by Gerrit Code Review over 4 years ago

Patch set 7 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/39644

#13 Updated by Gerrit Code Review over 4 years ago

Patch set 8 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/39644

#14 Updated by Gerrit Code Review over 4 years ago

Patch set 9 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/39644

#15 Updated by Gerrit Code Review over 4 years ago

Patch set 10 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/39644

#16 Updated by Gerrit Code Review over 4 years ago

Patch set 11 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/39644

#17 Updated by Gerrit Code Review over 4 years ago

Patch set 12 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/39644

#18 Updated by Mathias Schreiber over 4 years ago

  • Target version changed from 6.2.13 to 7.3 (Packages)

#19 Updated by Gerrit Code Review over 4 years ago

Patch set 13 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/39644

#20 Updated by Gerrit Code Review over 4 years ago

Patch set 1 for branch TYPO3_6-2 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/40143

#21 Updated by Markus Klein over 4 years ago

  • Status changed from Under Review to Resolved
  • % Done changed from 0 to 100

#22 Updated by Gerrit Code Review over 4 years ago

  • Status changed from Resolved to Under Review

Patch set 2 for branch TYPO3_6-2 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/40143

#23 Updated by Alexander Opitz over 4 years ago

  • Status changed from Under Review to Resolved

#24 Updated by Riccardo De Contardi about 2 years ago

  • Status changed from Resolved to Closed

Also available in: Atom PDF