Minutes from the 28th meeting of the release 4.5 team

These minutes are also posted to the typo3.projects.v4 mailing list. If you want to comment on specific topics, feel free to reply on the list. Thanks for your participation!

January 24th, 2011

We had a meeting today, one of the last meetings before our final release of 4.5.0.

Present were: Olly Hader (core team leader), Ben van't Ende (community manager), Steffen Kamper (4.5 technical leader) and Ernesto Baschny (4.5 release manager).

Pending issues

We discussed the pending issues briefly which are crucial for the final release:

t3editor won't save

Patch from Tobi in core list, 17078, needs further reviews. [Post-Note: Resolved by now]

Upgrade Wizard "Workspaces"

Workspaces Upgrade Wizard cannot be "skipped" (using the new "Next>", "Next>" approach, which means everybody will get Workspaces installed upon upgrade. Patch from Francois solves this by making this step optional. Issue 17260 pending. Needs more reviews.

CSH in Extbase

Issue reported by Georg (17265): CSH not working for Flexform in Extbase extensions. Pending already in the core list, needs more review.

New EM with DBAL

New Extension manager's "Remote Repository" tab uses subselects which are not DBAL compatible. Steffen will try to find a solution (which is doing subselects only if DBAL is not activated, else do the calculation in PHP afterwards).

Other issues

Olly create a link pointing to all issues reported since RC1: http://tinyurl.com/4ps9sms

He will do some triage to get a more comprehensive list. Also Ernesto will work on organizing the open issues.

Performance issues

Steffen talked with Rupi Germann which did some performance checks on our latest code and compared it with previous releases. We have added some features improving the overall performance of the backend (removing the FRAMESET) but on the other hand many features came in which require more processing performance.

The main issue tested is the amount of MySQL queries, which increased dramatically in 4.5. Mainly three issues were analyzed:

CSRF

New "Form protection" adds several queries when tokens are generated (and later deleted). E.g. the list view with the extended view open creates several hundreds of queries. For each icon, one additional SQL query is added.

See later on for the minutes of our post-meeting with Helmut, where we discussed the issue.

Pagetree (initial load)

As soon as the pagetree is loaded for the first time (backend.php reload or hitting the "Refresh" button) the whole tree of pages is fetched through individual queries. Up to 500 pages.

Steffen later talked to Stefan Galinski and this seems to be a pre-loading of pages (up to 500) to speed up the user interface on future clicks on the tree. Steffen will come up with a solution to have this pre-loading configurable because its effectiveness depends on various factors (it was defaulting to 500 pages).

Ernesto mentioned "nested sets" which would help speeding up loading certain tree parts. Steffen knows this, but there was yet no time to integrate such things. At least we now have a very stable API and a good abstraction layer to be able to very easily add such features without the fear of breaking some API => Optimization can start!

T3 Compressor

Analysing with kCacheGrind seems to point to t3lib_compressor as a main performance brake (10%). Apparently doing serialize / unserialize operations. Ernesto mentioned that the t3_compressor doesn't do any serialization (it "just" gets md5 hashes from files and compares...) so this would require further analysis to get solved.

ExtDirect / Ajax calls

Since we now have lots of more Ajax requests in our backend, we have to keep in mind that every one of them goes through the whole init.php process to get the correct environment to work with (BE User, etc). Even a simply CSH request does that.

This would be in the scope of the init.php cleanup project, to get that cleaned up and optimized (in a similar way that we already have for the Frontend).

Performance-Sprint after the release

We will concentrate on major issues we still have left until the final release which has a settled date already. We will then try to organize a code sprint to tackle the most annoying performance issues after the release one by one and with proper time for testing.

In the past we have avoided performance improving patches in released versions (due to the risk of breaking something), but Ernesto considers the first weeks after the release a good time to further improve performance of the backend, as we will see it performing "in the field". Of course every patch needs to be carefully reviewed and tested.

Cross Site Request Forgery (CSRF)

Later today we had a Skype meeting about CSRF in particular (Steffen Kamper, Ernesto Baschny and Helmut Hummel from the Security Team).

We talked about the open issues which are basically:

  • Performance (list module, too many queries)
  • Race conditions (ExtDirect, multiple IFRAMEs)
  • Other non-protected places

And also potential solutions to them.

Performance

Currently "to be save" the code does a "persis token" after every new generated token, because at first sight there is no central point in the code where the persistence of such things could be done.

After some brainstorming the idea came to generate all tokens during one execution in memory and persist in $this->doc->endPage() method, which is basically the exit point for all modules using template.php. On ExtDirect / Ajax calls, still persist on every call.

Race condition

Race condition happen when two scripts (e.g. nav + content) inits BE users almost on the same time but with different timings. Since tokens are stored in be_users's session (which is a huge serialized array), the problem pops up if one fetches the session data, the second one also, then the first one writes it back later (with new tokens) and the second one also (overwriting the first tokens).

Potential solutions are:
  1. locking shortly before token persistence: get session data, lock, change it, save, unlock
  2. store tokens in PHP session: PHP does the locking (session_start(), ..) and also no further SQL is required for tokens.

For the simplicity of it and also the potential performance improvement, we chose the PHP session path. For this to work reliably, Helmut will re-use the tx_install_session class, which already allows more secure PHP sessions in the Install Tool.

More protection

There is already a pending patch to add CSRF protection to the Web>Template module (17189, needs testing and reviews).

Helmut wants to add CRSF also to file-upload (tce_file.php), because he thinks its a crucial place. The patch is almost ready for core.

Working on it

Helmut is currently working on these tasks and we will see the results later on tonight, and hopefully be able to add those missing pieces to the puzzle before RC2.

Release Candidate 2

We decided that we will be releasing a Release Candidate (RC2) tomorrow morning between 10 AM - 11 AM (GMT+1). This gives the opportunity to everybody to contribute in the "night hours" to solve the remaining issues.

Ernesto has informed all participant teams, so that they merge their projects in time.

It has been mentioned that the fixed date of the final release should have been postponed before. Ernesto thinks we all have to get used to the rhythm of keeping the promised dates. We're proud to be able to release at the date which we set up on the first day where we started this journey, which has never happened on a release before. And even with the lack of time for more intensive testing and some remaining issues, we have overall a very stable and feature-rich version which we will all be proud of.

This also means that our active development team needs to be aware that we require more testing environments throughout the whole development process and also a more early reviews and helping out teams by noticing eventual performance bottlenecks and security issues at an earlier stage.

Documenting new features

Ernesto mentioned that he wants to see more Buzz articles on new features so that we could link to them in the release notes. Suggestion is to use a common prefix on these articles: "TYPO3 4.5 LTS: <topic>". If the information requires lots of code snippets or screenshots, this could be added to the Wiki and linked in the Buzz article. So the buzz articles should be the "perma-URL" to these new feature descriptions, which then link to the relevant technical documentations.

He already talked to Xavier about an article about "Prepared statements", and the article is already available.

Ernesto also contacted Francois on his opinion on this approach.

Further steps

  • We will release RC2 tomorrow morning.
  • We will release 4.5.0 on Wednesday throughout the day.
  • More tiny meetings will happen in between as we think is necessary.
  • We will make a pause after the release to renew the energies (Steffen Kamper mentioned the need of "90h of nonstop sleep" => he deserves it!)