TYPO3 Core

1. I agree that we should think about improving the db cleanup. I also run into situations where functional tests suite leaves many db after execution.
2. The sha1 suffix for the db names is needed, as usually we're running the functional tests in parallel, see travis.yml

find . -wholename '*typo3/sysext/*/Tests/Functional/*Test.php' | parallel --gnu 'echo; echo "Running functional test suite {}"; ./bin/phpunit --colors -c typo3/sysext/core/Build/FunctionalTests.xml {}'

maybe we can pass some flag as a param/env to switch on/off this prefix?

We solve these concerns by not mixing test and production DBs.

The tests are run in an isolated test environment: separated MySQL process on a dedicated TCP port/ socket, which sole purpose is to serve tests.

for example:

mysqld_safe --port=3307 --socket="/var/run/mysqld/mysqld-testing.sock" --user mysql --pid-file="/var/run/mysqld/mysqld-testing.pid" --datadir="/mnt/testramdisk/mysql" --skip-grant-tables

Patch set 2 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/46218

hey.

Ok, some background explanation:

First in "the old days" the unit tests had database requirements and always worked on the database configured in LocalConfiguration. This was fixed, unit tests don't need DB at all anymore. At this point functional tests didn't exist at all. So, this situation can not be compared to "old days".

The functionals need a DB. the name, username and password SHOULD be set via environment variables and fetching them from LocalConfiguration is only a fallback for lazy people. In general functionals should not be executed in an environment that has a "real" system in parallel.

Why multiple DB's are needed: Every test case (=test file) can define an own set of loaded extensions which define a set of tables + fields per test case. The system is created in a way that single tests of a test case must use the same db layout. Therefore one test case has one set of table+field definitions, a second test case probably has different db layout since a different set of extensions is loaded. If you then want to support to run multiple test cases in parallel (like the travis travis-ci does, have a look at .travis.yml in core root), you need different DB's for each test case. Parallelizing the tests has a huge positive performance impact.

Basically, the system creates the unique db per test file, populates the schema, then inserts whatever rows should be inserted, then runs the first test, then truncates this DB, then inserts rows of the next test, runs it, truncates, .... The difference between recreating a DB for each and every test and just truncating it between single tests is done since table creation is a pretty slow thing in mysql/maria and truncating is very fast. Not re-creating a DB for every single test thus speeds up the whole test suite enormously, even if mysql is running in a ramdisk. We're talking about factor 10 or something at the moment. This is significant since it is a difference between 10 minutes to an hour or something.

The problem with this setup is: phpunit is constructed in a way that single tests can not share information very well. So, the system does not know if a former test just created the DB already. It is not easy for a single test to find out whether it should create a fresh DB or if it should just truncate. Problem is here that in between multiple runs the schema may change and then you want your DB to be re-created, while in between single tests, the same DB should be re-used. Thus, instead we just store the date when a DB was created in a file per test case. A test reads this file and if the DB is older than some minutes, it will drop DB and create new, if it is "young" it will truncate and reuse. Goal: If you're in the process of hacking up a test and changing it, you're likely to have a fresh DB created each time, while multiple tests after each other truncate instead. BTW: You can force to re-create a DB in between runs, if the functional_* directories from typo3temp/ are deleted, this is where the single functional test instances are set up.

Another problem with phpunit is that a single test does not know if it is "the last one" of a test case: This is the main reason why there is no cleanup of DB's after finishing a test run. The only way to do this sort of clean ups is "tearDownAfterClass" (if i remember correctly), but unfortunately this method is a) static which gives us huge problems with information sharing to the helper class, it b) it does not help with the "process encapsulating" - every single tests runs in a single php process forked by phpunit. This forking is done to encapsulate single tests on PHP level from each other, if that is not enabled the system would for instance throw errors on defined constants. So, giving information from one single test run to another one is nearly impossible because of this process isolation. That's the reason why the database creation date is written to a file ...

I hope this explanation makes it a bit more clear why a) there is a single DB for each test case and b) why DB's are not deleted at the end. If you find a better solution to resolve all the above without speed loss, I'd be very interested.

All in all we currently need all these DB's. For the "suffix" thing: If you really insist on having the "_ft" configurable, the patch should imho be changed that a) no additional LocalConfiguration parameter is introduced, but maybe - if really needed - an additional environment variable is introduced to set this prefix.

Hi Christian,

Thanks for the elaboration, I mostly agree with you.

The important part in my suggestion is not that I want to override the normal behavior, "just" want to extend the possibility to suffix databases and not use the _ft $this->identifier.

I know this comes with a performance price when running the testing, but for us it a matter of use these settings to limit the amount of DB's that has to be created to run a functional test for a given extension. We are not running the TYPO3 Core functional testing, we trust the active contributors to make sure that these are maintained and kept working.

Main take away from my suggestion, it's an option not a requirement to use the suffix.

About getting the settings from CLI arguments and not from LocalConfiguration.php is totally valid to me, and I can change that.