TYPO3 Core

Patch set 2 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/39899

Patch set 1 for branch TYPO3_6-2 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/39928

Patch set 2 for branch TYPO3_6-2 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/39928

There was a report on gerrit from "Gilles Fouet" about this patch solving problems in their installation.

I can add information about the setup on which we encountered caching problems:

It is the website of a large customer whose work is intertwined with government responsibilites. The have guidelines specified by the government which dictates them to have a redundant, fail-over safe setup for the webhosting.

I am not informed about the internals of the mysql database - we just have the access credentials - but I assume there is a mysql-cluster behind it. Both of our TYPO3 web instances access the same database using the same credentials.

The access to the two parallel webservers we run is being split up by a load balancer I also have not much information about - but AFAIK this is a piece of hardware.

So for our part of the solution we have to take care about the TYPO3 instances and partly of the apache webservers. While the apache webservers are still maintained from people directly employed by our customer.

Both webservers share their complete webroot via NFS. I already heard some advices to not use NFS. But what else would be possible? I read some discussions about TYPO3 HA but most of them also take NFS into account. I was adviced (in slack) to exclude "typo3temp/" from NFS. But I fear that recent changes in TYPO3 (FAL) would cause problems then. For example scaled images will get logged in the database - but get eventually stored in typo3temp/. I know they are usually stored in "fileadmin/_processed_" but this is only true for resources being found in the "/fileadmin" default storage. There is also no special recommendation NOT to use NFS for TYPO3.

So when we would overlay "/typo3temp" with a per-server (local) filesystem I fear there will be issues with image generation.

I would also not want to just mount "/fileadmin" via NFS as this does not really ease the process of updating TYPO3 or installing an extension.

The patch here is taking care of regenerating class-loader configurations which have not been generated yet. Maybe because a server process died, etc. I have not cleared the exact circumstances. I was accused that my solution is taking care of the symptoms. But compare it to the following: Someone is working in his garage on his cycle and cuts himself badly in the finger. Now I as a doctor will not start to discuss with him about his cycle but simply but start to desinfect the wound, put some butterfly-strips over the wound and let it heal.

This is exactly what my patch is doing: When the cache is not complete (i.e.: there is a wound) I regenerate any missing cache file (let it heal).

As this only occurs in our setup I would assume the whole issue has to do with non-working locking. I guess we do not need to talk about mandatory/advisory locking (https://en.wikipedia.org/wiki/Lock_%28computer_science%29#Types) as in our case the goal is to avoid a race condition - not to have exclusive access to a file and both processes surely cooperate.

The cause for the whole locking issue might be because "LOCKING_METHOD_SIMPLE" is used for "Locker" in "EXT:core/Classes/Core/ClassLoader.php". According to the following article "fopen(..., 'x')" is not portable across NFS:

http://lwn.net/Articles/251004/

This article on the other hand describes that "fopen()" and O_EXCL should work from kernel 2.6.5 on:
http://nfs.sourceforge.net/#faq_d10

On the other hand "flock()" is specified to be properly working over NFS when the mount option "lock" is set and both NFS parties (server & client) use recent enough kernel versions:

Since Linux 2.6.12, NFS clients support flock() locks by emulating them as byte-range locks on the entire file.

The man page for nfs "man 5 nfs" also contains a section about the "lock" option when mounting an NFS filesystem:

lock / nolock

Selects whether to use the NLM sideband protocol to lock files on the server. If neither option is specified (or if lock is specified), NLM locking is used for this mount point. When using the nolock option, applications can lock files, but such locks provide exclusion only against other applications running on the same client. Remote applications are not affected by these locks. 

So from my experience locking over NFS works when using NFS version 4 on both server and client, both machines have a recent kernel and the option "lock" is specified by all clients which mount the share.

In the end there are quite a lof of dicussions about whether flock() is supported over NFS or not. I read "flock()" is not supported over SMB, and semaphores and other IPC locking mechanisms would not come into discussion as they are restricted to the same physical/virtual machine.

First proposal: Make the locking mechanism which is used by "ClassLoader" configurable.

Another proposal: Add a "LOCKING_METHOD_DATABASE" option to the LOCKER.

The whole discussion starts to become senseless in 7.x as there the composer class loader is used. I wonder what issues this approach will bring with it :)

http://bytes.com/topic/php/answers/11381-flock-nfs

A few words from my side.

Make the locking mechanism which is used by "ClassLoader" configurable.

Not possible, since this is needed so early that no configuration is available. That was the reason to hardcode it to Simple. (since flock() might fail in one or the other way, also for a single instance.)

Add a "LOCKING_METHOD_DATABASE" option to the LOCKER

In 6.2 we will not touch the locking anymore, as it is deprecated in 7 and replaced by more flexible API. You can implement a DB locking there. I didn't have the time/need to do that yet.

there the composer class loader is used. I wonder what issues this approach will bring with it :)

None so far, as there is no cache anymore and therefore also no locking is involved. ;-)
The available classes are discovered only once (on ext install for instance) and are stored statically, so no dynamic class discovery, no stale cache files.
The generated files (typo3temp/autoload/) should be prefectly sharable via NFS.

Thanks for digging into the NFS locking btw.

Patch set 3 for branch TYPO3_6-2 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/39928

Bernhard Kraft wrote:

There is also no special recommendation NOT to use NFS for TYPO3.

There is no special recommendation how to properly do load balancing and high availability with TYPO3, true.
However, it should be clear that, loading PHP source files from a NFS drive highly impacts performance in a bad way, which generally should be avoided.
Since typo3temp/Cache is also a PHP file cache, moving this to a temporary (or even ram disk) is recommended. Doing so is possible with a simple symlink (given all servers have the same directory structure)

This is exactly what my patch is doing: When the cache is not complete (i.e.: there is a wound) I regenerate any missing cache file (let it heal).

There are several different ways how to deal with this situation.
The first apparent question which comes to my mind: If you have such issues with empty caches in your setup: Why do you clear them like AT ALL

Anyway: Your suggestion is a hack and incomplete. But it can help in some situations for 6.2 Installations AND it does not impact other installation too badly.

The whole discussion starts to become senseless in 7.x as there the composer class loader is used. I wonder what issues this approach will bring with it :)

This is fixed in master mostly because we do not treat class loading information as a cache any more, so it will only be re-build if needed or requested. This can be done in a controlled environment.

Patch set 4 for branch TYPO3_6-2 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/39928

Helmut Hummel wrote:

Since typo3temp/Cache is also a PHP file cache, moving this to a temporary (or even ram disk) is recommended. Doing so is possible with a simple symlink (given all servers have the same directory structure)

The problem here is the install tool and it's clear cache functionality (and also the "red" clear cache button).

The "typo3temp/Cache" directory is cleared/flushed by calling "GeneralUtility::flushDirectory()" which in turn uses a rename/mkdir scheme to empty the directory as quickly as possible:

https://git.typo3.org/Packages/TYPO3.CMS.git/blob/HEAD:/typo3/sysext/core/Classes/Utility/GeneralUtility.php#l2827

So using a symlink would not be possible as every cache-clear would recreate the path as directory.

This whole issue lets me rethink the "convention over configuration" issue again - TYPO3 was so beloved because every aspect of it could get configured and thus it could get adopted to a very broad range of applications.

This whole caching issue turns out to be hardly interconnected with the IT infrastructure the instance is running on and therefore would require configuration options again. If one could set how to flush a directory, how to create locks, how to cache code files, etc. the options would rise again exponentially. Of course the default settings should be sane ones for most common cases.

We on the other hand could set up a HA TYPO3 instance(s) being perfectly tailored to the requirements of any our clients.

Bernhard Kraft wrote:

The problem here is the install tool and it's clear cache functionality (and also the "red" clear cache button).

You use these buttons in PRODUCTION on a system under heavy load? Well …

So using a symlink would not be possible as every cache-clear would recreate the path as directory.

$cacheConfigurations = &$GLOBALS['TYPO3_CONF_VARS']['SYS']['caching']['cacheConfigurations'];
$cacheConfigurations['cache_phpcode']['options']['cacheDirectory'] = 'file:///absolute/path/to/main/cache/dir/';
$cacheConfigurations['cache_core']['options']['cacheDirectory'] = 'file:///absolute/path/to/main/cache/dir/';
$cacheConfigurations['cache_classes']['options']['cacheDirectory'] = 'file:///absolute/path/to/main/cache/dir/';

done! No symlink needed. No worries about non temporary assets in other typo3temp/ folder

Bernhard Kraft wrote:

This whole caching issue turns out to be hardly interconnected with the IT infrastructure the instance is running on and therefore would require configuration options again. If one could set how to flush a directory, how to create locks, how to cache code files, etc. the options would rise again exponentially. Of course the default settings should be sane ones for most common cases.

Did you look into caching? You can configure almost anything. And if that still does not fit your needs, you can add your own caching frontend or backend. Flexibility galore.
Yes, locking is a different story. It is tackled as well in master.

We on the other hand could set up a HA TYPO3 instance(s) being perfectly tailored to the requirements of any our clients.

Documentation on that, including best practice from experience is highly welcome ^^

Do you mind setting up a Wiki page? I can look over that and add some things I know about that topic.