Under heavy load, two request can simultaneously write to same cache file - should use locking
On sites with rather high load, we have witnessed a rare problem, where two request is apparently writing to the same cache file at the same time. This happens if the cache is cleared, and two requests both try to populate the cache_core cache, which is i SimpleFileBackend cache.
In some cases the file might be a file that is only needed by an extensions, and pages using that will fail, in some cases the file might be one that is also used by the backend, which means the backend is effectively broken, and its not possible to clear cache (other that manually deleteing the typo3temp/Cache/Code directory).
The problem is much worse on a slow filesystem, and I believe that it is due to SimpleFileBackend nt using a filelock before writing the file.
The fix should be very easy. Simply either issue a flock directly in SimpelFileBackend, og use TYPO3 locking mechanism.
#3 Updated by Markus Klein almost 6 years ago
- Status changed from Accepted to Needs Feedback
Which cashed are affected for you Jan-Erik?
If it is the classloader caches, then the fix is pending already in #54906.
If other caches are affected please tell us which, because each of them needs careful manual investigation.
We can't employ a general and safe locking mechanism to all caches.
#4 Updated by Alexander Opitz over 5 years ago
- Status changed from Needs Feedback to Closed
- Target version deleted (
No feedback within the last 90 days => closing this ticket.
If you think that this is the wrong decision or experience this issue again, then please write to the mailing list typo3.teams.bugs with issue number and an explanation or open a new ticket and add a relation to this ticket number.
#5 Updated by Gabe Blair about 5 years ago
We are experiencing the same issue on a relatively high-traffic site using TYPO3 6.1. This is harsh when it happens, because it brings down the entire site with a PHP error, when a PHP cache file is partially or doubly written. This issue has proven quite difficult to reproduce for development/testing purposes, but it definitely is real.
The SimpleFileBackend uses file_put_contents() to create the temp file that eventually becomes (https://github.com/TYPO3/TYPO3.CMS/blob/master/typo3/sysext/core/Classes/Cache/Backend/SimpleFileBackend.php#L238). However, file_put_contents() is not locking, by default (http://us.php.net/file_put_contents). Can anyone think of a downside to using the "LOCK_EX" flag in file_put_contents() to allow PHP to handle the locking?
Obviously there is some activity at the moment around cache locking in general. But how about we at least use native PHP support for locking as a stopgap measure for SimpleFileBackend until a more holistic locking mechanism can be developed overall for caches (http://wiki.typo3.org/Blueprints/LockingForCaches)?
#6 Updated by Markus Klein about 5 years ago
- Status changed from Closed to Needs Feedback
as author of the mentioned blueprint I have to tell you, that I stopped working in this direction for the time being.
The main issue is that there's no guaranteed native way of locking available in PHP that works across all platforms.
Generally I'm not aware of the locking mechanisms employed in version 6.1, but I can tell you that 6.2 does a way better job already.
I can only encourage you to upgrade to 6.2 as soon as possible.
For version 7 my only intention is to replace the current locking API with a more flexible one, which allows registering of other locking technologies, so the use can choose which locking technology fits best on his environment.
Besides that we still face the problem in various caches that we need to know the state of a lock. This information is not available with all locking mechanisms. The ones where this is available are not safe to use in all environments, which could be a cause for your problems, but that is just a guess.