Bug #63519
closed
sys_file_processedfile rows contain zero dimensions
100%
Description
There is an error with the scaling process if the same picture is placed multiple times on the same page.
Some rows from table sys_file_processedfile contain zero values for width and height although the serialized configuration contains settings for width and height.
The LocalDriver moves the temporary (scaled) file after scaling but the GraphicalFunctions (scaling process)
assumes (on the basis of a database query) at the next time that the temporary file exists and try to get the image dimensions on a not existing file.
Nothing is returned and a zero value reaches the Database.
... \TYPO3\CMS\Core\Resource\Processing\LocalImageProcessor::processTask() 60: $helper = $this->getHelperByTaskName($task->getName()); 62: $result = $helper->process($task);
\TYPO3\CMS\Core\Resource\Processing\LocalCropScaleMaskHelper::process()
75: $result = $gifBuilder->imageMagickConvert(
$originalFileName,
$configuration['fileExtension'],
$configuration['width'],
$configuration['height'],
$configuration['additionalParameters'],
$configuration['frame'],
$options
);
\TYPO3\CMS\Core\Imaging\GraphicalFunctions::imageMagickConvert() 2258: $output = $this->absPrefix . $this->tempPath . 'pics/' . $this->filenamePrefix . $theOutputName . '.' . $newExt;
$output contains the temporary filename like /var/www/basisbox.local/htdocs/typo3temp/pics/5e0e4724b9.jpg
\TYPO3\CMS\Core\Imaging\GraphicalFunctions::file_exists_typo3temp_file()
This function look into the database for a entry that contain the temporary filepath ($output).
If the entry exist no scaling will be performed because there is already an temporary file.
2261: if ($this->dontCheckForExistingTempFile || !$this->file_exists_typo3temp_file($output, $imagefile)) {
2262: $this->imageMagickExec($imagefile, $output, $command, $frame);
2263: }
2264: if (file_exists($output)) {
After the file_exists_typo3temp_file() check, we check the existence of the temporary file again and return its dimensions.
If the file does not exist, nothing will be returned.
Back to the processTask
\TYPO3\CMS\Core\Resource\Processing\LocalImageProcessor::processTask() 73: $task->getTargetFile()->updateWithLocalFile($result['filePath']);
After the scaling the scaled file will be added to the storage
$result['filePath'] contain the temporary filepath eg. /var/www/basisbox.local/htdocs/typo3temp/pics/5e0e4724b9.jpg
\TYPO3\CMS\Core\Resource\ProcessedFile::updateWithLocalFile() 181: $addedFile = $this->storage->updateProcessedFile($filePath, $this);
\TYPO3\CMS\Core\Resource\ResourceStorage::updateProcessedFile() 1118: $fileIdentifier = $this->driver->addFile($localFilePath, $this->getProcessingFolder()->getIdentifier(), $processedFile->getName());
\TYPO3\CMS\Core\Resource\Driver\LocalDriver::addFile() 662: $result = rename($localFilePath, $targetPath);
At this point the temporary file /var/www/basisbox.local/htdocs/typo3temp/pics/5e0e4724b9.jpg will be moved to another location.
The next scaling process will fail because
\TYPO3\CMS\Core\Imaging\GraphicalFunctions::file_exists_typo3temp_file()
find a database record that contains this filename.
The process don't render the file again because
\TYPO3\CMS\Core\Imaging\GraphicalFunctions::imageMagickConvert()
2264: if (file_exists($output)) {
don't find the file and return nothing and the sys_file_processedfile row will be filled with height = 0 and width = 0
Updated by Ralf Zimmermann over 9 years ago
I forgot to mention this happens if GFX[enable_typo3temp_db_tracking] = 1
Updated by Ingo Schmitt about 9 years ago
We should check, if this is still present after applying 64643. Because GFX[enable_typo3temp_db_tracking] is removed.
Updated by Björn Jacob about 9 years ago
#64643 removes GFX[enable_typo3temp_db_tracking] only for 7.x. So the bug is still in 6.2 present.
Updated by Gerrit Code Review almost 9 years ago
- Status changed from New to Under Review
Patch set 1 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/39372
Updated by Gerrit Code Review almost 9 years ago
Patch set 1 for branch TYPO3_6-2 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/39373
Updated by Gerrit Code Review almost 9 years ago
Patch set 2 for branch TYPO3_6-2 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/39373
Updated by Gerrit Code Review almost 9 years ago
Patch set 3 for branch TYPO3_6-2 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/39373
Updated by Morton Jonuschat almost 9 years ago
- Status changed from Under Review to Resolved
- % Done changed from 0 to 100
Applied in changeset c5b7a0d92d66ee670d53f976511e44540682195e.
Updated by Ralph Brugger over 8 years ago
For me it does not to be resolved.
Currently our sys_file_processedfile has 138.158 records
SELECT count(uid) FROM `sys_file_processedfile` WHERE `identifier` = '';
=> 1.557
SELECT count(uid) FROM `sys_file_processedfile` WHERE `width` = 0 OR height=0;
=> 1.557
SELECT count(uid) FROM `sys_file_processedfile` WHERE `name` IS NULL;
=> 1.557
TYPO3 6.2.15
The result ist, if the FE tries to output such a image, the image can't be found and the original image will ne used instead, because the sys_file_processedfile entry is corrupt.
The only way to correct those images, is to delete the affected sys_file_processedfile record.
Maybe it would be a good idea to never write any record to sys_file_processedfile which has:
identifier NULL
name NULL
width =0 0
height == 0
Updated by Frans Saris over 8 years ago
Hi Ralf,
the 0/NULL values are intended for when the original images should be used.
Can you reproduce the situation when you get a corrupted sys_file_processedfile entry? Maybe we there is still an issue to tackle where creating the processed file fails and a wrong entry is created.
gr. Frans
Updated by Gabe Blair over 8 years ago
Hi All,
We have run into the same issues as Ralph on a couple of sites, too, most recently with TYPO3 6.2.17. In both cases, we had upgraded from pre-6.2.14 to 6.2.14+. Because 6.2.14 introduced a new hash calculation, all our sys_file_processedfile rows became effectively invalid, and all temporary images needed to be regenerated. Some failed, resulting in blank/0/NULL rows in sys_file_processedfile.
We don't know why the processing failed for a handful of images. However, we are not able to reproduce the problem on cloned environments, or in local DEV environments, using the same exact content set and codebase. In both cases, only our production environments exhibited the behavior (or we wouldn't have released the upgrade changes to production). One thing I do know is that both of the sites on which we've seen this issue are heavy users of image processing, with 20+ and 60+ processed images (of which the originals are quite large) included in a typical page load, respectively. Both of the sites have potentially steady levels of traffic, with multiple active users loading pages at any given time. Many of the images that failed to load in both cases were based on originals of large dimensions/filesize. This hints at the possibility that there is an issue with memory usage or similar, perhaps at the OS level, or within ImageMagick itself.
Clearing the NULL rows from sys_file_processedfile resulted in one of the sites successfully generating the files. On the other, image-heavier site (a prominent museum), not a single file was generated, and all the rows in the sys_file_processedfile table were of the blank/0/NULL variety. On that site, removing the blank rows did not help, and we ended up rolling back the TYPO3 version. We haven't returned to tackle the problem yet, since it has been a low priority for our client. Our first foray will involve switching to GraphicsMagick instead of ImageMagick, and seeing if that makes a difference.
Because the issue has not been reproduced on local or cloned environments, it's quite difficult to determine the root cause of the wrongly NULL entries being created. A less direct solution for us would be to have TYPO3 try more than once to generate an image before giving up and never trying again. Something like $TYPO3_CONF_VARS['SYS']['GFX']['maxImageProcessingTryCount'] might be handy. The current count would be tracked on the sys_file_processedfile table, and once it was incremented to the maximum number, the system would simply use the original. Setting the new option to a value of "1" (would be the default) would make the system behave exactly as it currently does. Does anyone have a sense for: if we develop something like this, how likely do you think it is to get accepted into the core? Because of the review workflow, we would be spending some rather valuable time/effort to push it through.
Thanks and regards,
Gabe
Updated by Anja Leichsenring over 8 years ago
- Sprint Focus deleted (
On Location Sprint)
Updated by