Feature #81989
open
ImageMagick - Prevent Concurrent Processing of Image Files
0%
Description
Current Setup:
- Apache 2.4.23
- PHP 7.0.20 Nonthread safe
- TYPO3 7.6.20
- ImageMagick 6.7.8-9
Things to note:
- A website uploads pictures in extremely high resolution, meaning that the average filesize of each file is about 30-40MB.
- There are a number of custom content elements, each with its own DataProcessor. For instance, one creates 8 different processed versions of an image for various viewports.
- Switching to GraphicsMagick did not solve the problem.
Problem:
ImageMagick fails to process all images for all viewports and thus results in creating corrupted files when multiple requests to the server are made that causes the same image files to be processed multiple times simultaneously.
Previously attempted strategies, etc.:
- Create a database locking mechanism that prevents a second conversion if a flag is set signaling that processing is already in progress.
- Create smaller versions of the file to then convert instead of the original
- Analyzing the errors thrown by ImageMagick. No recognizable pattern was noticable. The conversion failed at multiple points with different pictures with different resolutions.
Next strategy
Under the assumption that ImageMagick is working properly AND/OR cannot be reconfigured to prevent this problem:
- Implement preprocessing by utilizing appropriate hooks in DataHandler. For this to be reliable, we would like to limit requests to those made by the TYPO3 backend, thus, essentially turning off "on the fly" conversion.
Preliminary conclusions
- The overall amount of traffic seems to be irrelevant. This has been tested in development and production environments. It is sufficient that two processes are started (at least more or less) simultaneously to process an image that causes ImageMagick to fail, at least partially.
- The filesize seems to be a large contributing factor. If it is smaller than 5MB, this problem rarely (if at all) occurs.
Nice to have
- We would like to be able to use the original file with no intermediary reduction of the image quality.
TL;DR
Concurrent processing of large images using ImageMagick fails.
Files
Updated by Markus Klein over 6 years ago
Did you try locking via the API instead of DB?
In general using flags in DB is not an atomic procedure (test and set) and hence is of course prone to race conditions.
Updated by Joseph Linden over 6 years ago
I agree that database locking could be too slow. Yes, file locking has been considered, but we were slightly wary of that approach since access exceptions/warnings could be thrown which could result in problems for the request that "lost", since access to the file itself would be denied. Are these concerns justified?
By API, I assume you mean the class \TYPO3\CMS\Core\Locking\Locker?
Updated by Markus Klein over 6 years ago
No, that's the old deprecated API. Checkout the LockFactory.
It has capabilities, so you can use non-blocking locks for instance and stuff like that.
Updated by Frans Saris over 6 years ago
Nice to have We would like to be able to use the original file with no intermediary reduction of the image quality.
You want to deliver the original file to the front-end? This is already possible. Just make sure there are no max dimensions set.
Updated by Joseph Linden over 6 years ago
@Markus: OK, I will try implementing that.
@Frans: No, I meant to process the original file, not a smaller version of the original file, for example, one created by lowering the DPI.
Updated by Joseph Linden over 6 years ago
OK, we extended the LocalImageProcessor and forced images to be processed after saving. Both use the locking mechanism. This seems to work, but we still need to test in a production environment to test for locking failures. I have also posted the issue here:
http://imagemagick.org/discourse-server/viewtopic.php?f=3&t=32490
Please do not yet close this ticket.
Updated by Webadmin no-lastname-given over 5 years ago
- File csm_hanau_marktplatz_2108_dpa_deb769d2ae.jpg csm_hanau_marktplatz_2108_dpa_deb769d2ae.jpg added
We discovered a similar problem. On Ubuntu 18.04., PHP 7.2, with GraphicsMagick even with "normal" image sizes around 1 to 2 MB.
Requestion a Typo3 page renderes all the images and will store them under fileadmin/_processed_. So far this works fine. But as soon as there are more requests to different pages at the same time which contains the same images, we got corrupted JPGs. Mostly the images are grey. One example is attached. Sometimes the image isn't processed in any way and delivered in the original size to the client.
We think that typo3 starts the processing for the image multiple times if it is not already processed. Then both jobs will store their result in the same file?
For testing we now switched from GM to IM.
How did you solve the problem in the meantime?
Updated by Attila Glück over 5 years ago
I have the same problem.. very large jpg 7MB 7000*3000px
The image editor crop the images randomly.
GM and IM same results.
php 7.2
typo3 8.7.19
Updated by Webadmin no-lastname-given over 5 years ago
In the meantime we could collect some more details:
- Seems to happen especially for pictures which are includes on several pages. So a parallel processing will start as soon as two pages are requested at the same time.
- We have two different effects: The image is rendered in a corrupt way or it is outputted in it's original size.
- If the picture is outputted in it's original size we are seeing a new entry in the sys_file_processedfile table which contains an empty identifier and NULL in name, width and heigt.
Especially the last point is very bad for the user experience. Often we deliver 7MB JPGs for an 50x50 teaser image.
Typo3 should implement some locking within the image processing which should prevent the system from processing images with the same processing instructions at the same time.
We also tested IM, GM, PHP 7.0, PHP 7.2: Same results. Other PHP based image processing tools are working fine. So it's an typo3 related problem.
Updated by Webadmin no-lastname-given over 3 years ago
- PHP Version changed from 7.0 to 7.4
Retested with Typo3 LTS 10 and PHP 7.4. Still the same problems. Even with smaller pictures around 2000x2000 px.
It only happens to pictures which are used on two or more pages.
So far, we only tested GM. There the original file size was delivered. No pictures with grey bars in it. Now switching to IM...
Updated by Markus Klein over 3 years ago
- Related to Bug #92188: Deferred image processing in the backend is only partly implemented added
Updated by Markus Klein over 3 years ago
Can you test, if the patch https://review.typo3.org/c/Packages/TYPO3.CMS/+/65237 for #92188 solves this for you?
Updated by Oliver Weiss almost 3 years ago
Same problem here. TYPO3 8.7
Our pics are not too large, but we have a lot of users and it seems, that while the image for the first user is in process, the second user creates a new database entry with NULL values in identifier an name. TYPO3 seems to think, the image needs no processing - this is what these kind of database rows stand for.
The database entry of the first user comes with identical timestamp, but short after the second user.
During next rendering process, the first database row (the wrong one) is taken. In our case, the images are not scaled, but the original file is delivered.
I checked the code, and it seems there is no locking mechanism implemented at all. This should probably been done.
Updated by Webadmin no-lastname-given 11 months ago
Within our installation the problem seems to be gone after Updating to Ubuntu 22.04 with PHP 8.1 (Typo3 LTS 11, GM for image processing).
Running LTS 11 with PHP 7.4 showed the same problems as before.