Feature #19503

Implement autoloading for TYPO3

Added by Dmitry Dulepov over 12 years ago. Updated almost 11 years ago.

Status:
Closed
Priority:
Must have
Assignee:
Category:
-
Target version:
-
Start date:
2008-10-24
Due date:
% Done:

0%

Estimated time:
PHP Version:
5.2
Tags:
Complexity:
Sprint Focus:

Description

The attached patch implements autoloading for TYPO3 classes.

The patch consists from two files. The first file adds autoloading, the second file changes TYPO3 Frontend to use autoloading.

Autoloading is based on the concept of class registry and uses SPL to add TYPO3 autoloader to the chain of system autoloaders.

Class registry consists from two groups, each group contains two files.

The first group is located at t3lib/stddb/autoload_*.ser and includes information about all classes from t3lib/ and typo3/ directories. This group is used when TYPO3 is just installed.

The second group consists from files with the same name in typo3conf/ directory. These files are unique for the installation and list classes for core + installed extensions. Extension Manager updates these files when extension is installed or uninstalled. This group reduced memory usage at runtime.

Registry exists for Frontend and Backend. Frontend version does not include Backend classes from typo3/ directory (but includes typo3/ext/ and typo3/sysext/).

When user just installs TYPO3, he can login to the system. The big yellow box will prompt him to update autoloader database and provide a link to Tools>DB module. Additionally command line script exists to update autoloader data if something goes wrong in Backend.

The second file removes require_once calls from TYPO3 Frontend and thus can be used as demo/test for autoloader feature.

(issue imported from #M9633)


Files

9633-part1-add-autoloader.diff (103 KB) 9633-part1-add-autoloader.diff Administrator Admin, 2008-10-24 10:07
9633-part2-use-autoloader-in-fe.diff (7.31 KB) 9633-part2-use-autoloader-in-fe.diff Administrator Admin, 2008-10-24 10:07
9633-part3-enable-autoload-for-BE.diff (2.2 KB) 9633-part3-enable-autoload-for-BE.diff Administrator Admin, 2008-10-25 11:46
9633-part1-v2-no_singleton.diff (103 KB) 9633-part1-v2-no_singleton.diff Administrator Admin, 2008-10-26 09:18
0009633_v5_essentials.patch (43.5 KB) 0009633_v5_essentials.patch Administrator Admin, 2009-02-20 21:41
0009633_v5_lang.patch (17.4 KB) 0009633_v5_lang.patch Administrator Admin, 2009-02-20 21:41
0009633_v5_cleanup.patch (107 KB) 0009633_v5_cleanup.patch Administrator Admin, 2009-02-21 12:07
0009633_v6_cleanup_t3lib.patch (11.9 KB) 0009633_v6_cleanup_t3lib.patch Administrator Admin, 2009-03-01 13:32
0009633_v6_essentials.patch (47.1 KB) 0009633_v6_essentials.patch Administrator Admin, 2009-03-01 13:33
typo3v4-autoload.patch (43.9 KB) typo3v4-autoload.patch Administrator Admin, 2009-04-26 15:59
extdeveval.patch (11.7 KB) extdeveval.patch Administrator Admin, 2009-04-26 15:59
0009633_v8_autoloader.patch (16.6 KB) 0009633_v8_autoloader.patch Administrator Admin, 2009-05-01 17:37
0009633_v8_enable.patch (8.81 KB) 0009633_v8_enable.patch Administrator Admin, 2009-05-01 17:37
0009633_v9_autoloader.patch (15.7 KB) 0009633_v9_autoloader.patch Administrator Admin, 2009-05-12 18:41

Related issues

Related to TYPO3 Core - Bug #19580: Some extensions do not work anymore in the frontendClosedOliver Hader2008-11-06

Actions
Related to TYPO3 Core - Bug #20483: Misbehaviours due to integration of autoloaderClosedOliver Hader2009-05-20

Actions
Is duplicate of TYPO3 Core - Feature #19192: Add autoloading to reduce require(_once) callsClosedIngo Renner2008-08-04

Actions
Is duplicate of TYPO3 Core - Feature #18548: Adding autoloadClosedOliver Hader2008-04-03

Actions
Has duplicate TYPO3 Core - Feature #16385: automatic file loading checkClosedChris topher2006-07-20

Actions
#1

Updated by Oliver Hader over 12 years ago

Hi Dmitry, nice one! :)

  • "AUTOLOAD_REGISTRY_FILENAME_BE" could be "FILENAME_BE" since it's already in t3lib_autoload
  • line 235: "(?:class|interface))[ \t]*" should be "(?:class|interface))[ \t]+" since at least one space is required
  • line 235: class/interface names have to start with an character out of a-z

I'm going to test this the next few days. Do you think we could already integrate this into TYPO3 4.3-alpha1 if it seems to be almost stable?

olly

#2

Updated by Steffen Kamper over 12 years ago

Hi Dmitry,

after apllying both patches i get error in BE when calling a page with classic page-module:
Fatal error: Class 't3lib_transl8tools' not found in C:\xampp\htdocs\t3\home\t3lib\class.t3lib_div.php on line 4592

vg Steffen

#3

Updated by Dmitry Dulepov over 12 years ago

Steffen, this is because autoloading is not enabled in BE yet. I removed all require_once in cms, probably page module got affected accidentally.

You can add the following code in typo3/init.php:

// *********************
// Autoloader
// *********************
require_once(PATH_t3lib . 'class.t3lib_autoload.php');
t3lib_autoload::getInstance()->installAutoloader();

after the inclusion of config_default.php (after line 165) and page module will work.

#4

Updated by Dmitry Dulepov over 12 years ago

Olly, the patches do not enable autoloading in BE yet. But it is easy to do. I think I will post the third patch here soon. Next we can put it all into action :)

#5

Updated by Ingo Renner over 12 years ago

first thanks for working on this
second why didn't you inform Masi and me about it as we signed up for the task initially: #19192 (I thought we talked about communication during T3TD)

now some comments:

  • the autoloader (AL) should use the new caching framework so that there's no need for the *.ser files
  • use a cache with file backend to store everything in typo3temp/
  • the AL should not need to be triggered manually, instead I'd say it should build the registry on its first run
  • a registry the way you did it is not necessary for most classes and takes up a lot of memory (at least a lot more then needed) during execution
  • the way you scan for classes might be optimized and made more reliable when using PHP's tokenizer instead of the regexes (http://us2.php.net/manual/en/intro.tokenizer.php)
  • t3lib/sttdb could happen to not be writable
  • if a class implements t3lib_Singleton, there shouldn't be a getInstance method, do it either way

thanks anyways, but there's still room for improvement and simplyfication

#6

Updated by Dmitry Dulepov over 12 years ago

Part 3 of the patch enables autoloading for Backend. I removed require_once for classes, which are not always used. But I left require_once for classes that are always in use. This way Backend will be faster.

Steffen, this patch will solve the problem with page module that you have :)

#7

Updated by Dmitry Dulepov over 12 years ago

Just one more note: I run SVN version of TYPO3 with this patch for several days now. I do not see any problem so far.

Some numbers:
  • memory_limit is set to 128M.
  • 26 extensions installed (most of them have lots of classes – MVC stuff)
  • There 671 classes in the registry for my installation (BE mode, which has more classes than FE). Without typo3conf/ext it is 495 classes.
  • 671 classes in registry take 56K (56949 bytes) in serialized format. This should be approximately the same in PHP memory.
  • The speed of loading serialized format is extremely fast. Other formats will be slower (tested with plain PHP file and database). I made tests on my local PC, which does not have any load. Storing registry in other places will have bad impact on performance on a real server. I think only memcached can beat it but data still have to be stored serialized there and memcached can erase data leaving us without any autoload data (crash!!!)
#8

Updated by Steffen Kamper over 12 years ago

Hi Dmitry,

after applying 3rd patch all is working fine ;-)

One question:
You store the classes in t3lib/stddb/autoload_xx.ser
This might conflict with servers using a central installation and symlinks.

Imho it would be better to use typo3conf as it is different for each installation.

Indeed the autoloading is very fast and works proper, good work! This is only a subjective impression as i didn't reviewed code yet, but very promising! Thanks for your work.

#9

Updated by Dmitry Dulepov over 12 years ago

It will not conflict :) There are two registries: one is system, which is in the t3lib/stddb/, another is site-wide, it is in the typo3conf/.

System registry is used only if site-wide is not found. It contains classes from t3lib/ and typo3/ and generally used at the first log in to the system.

User will be prompted to create site registry when he logs in. Also site registry will be created automatically each time when an extension is (un)installed. It takes approximately 5 seconds on 2MHz CPU to rebuild registry.

So it is covered :)

#10

Updated by Dmitry Dulepov over 12 years ago

Following the discussion about singletons, I removed t3lib_Singleton from the t3lib_autoload (see part 1 version 2).

This is my final work on this patch. The solution is provided, I am switching to other tasks now.

#11

Updated by Steffen Kamper over 12 years ago

Hi Dmitry,

thx for the explanation. The problem on my site was that the files wasn't created after login, no prompt. After i changed status of an extension in EM the files were created, so i will look why they wasn't created for the first time.

Everything works fine without any error.

#12

Updated by Dan Osipov over 12 years ago

Works for me too. I haven't tested performance, though.

#13

Updated by Oliver Hader over 12 years ago

Hi! Any news on this great feature?
What is currently missing to put it into the next alpha of TYPO3 4.3?

#14

Updated by Ingo Renner over 12 years ago

I sent a status request to Masi last week but didn't get a reply yet either...

#15

Updated by Dmitry Dulepov over 12 years ago

"To be or not to be", that is a question...

#16

Updated by Oliver Hader over 12 years ago

I'm just about to overwork the last version I received from beginning of January 2009.
The tasks are splitted in:
  • integrating the autoloader and file parser ("essentials" patch)
    + changed to fetch file contents with php_strip_whitespace() so save some memory
    + "copied" lang.php to class.language.php
    + fixed bug in builder class - only "ux_" classes were taken into account
  • defining typo3/sysext/lang/lang.php as deprecated ("lang" patch)
  • cleanup by removing require(_once) statements from core ("cleanup" patch)

Notice: The cleanup patch has some strange behaviour on repatching... it produces some hunk failures, however it's about whitespaces or SVN controlls like "Id: $Id$"...

#17

Updated by Oliver Hader over 12 years ago

I've updated the essentials and cleanup patch:
  • the autoloading follow now a naming scheme
    1) t3lib_TCEmain -> /t3lib/class.t3lib_tcamain.php
    2) tx_ttnews_controller_frontController -> .../tt_news/controller/class.tx_ttnews_controller_frontcontroller.php (could also be in the sub-dir classes, as often used)
    3) t3lib_browseLinksHook -> /t3lib/interfaces/interface.t3lib_browselinkshook.php (since no regular class was found in previous check)
  • The core come with a prebuild legacy registry file
  • Extensions have the possibility to create their own registry with extdeveval but should follow the naming scheme (thus, they won't need the registry file at all)
#18

Updated by Sebastian Kurfuerst about 12 years ago

Problem:
Currently, lots of files get included at each request, without being needed at all (Example: Caching framework). To get rid of this problem, we're introducing the new autoloader here.

Solution:
The autoloader has to deal with quite a lot of different situations:
- core files (tslib, t3lib, but also core files not having these naming conventions)
- classical extensions (tx_myextension_piX, ...)
- new extensions with FLOW3 naming conventions, f.e. all Extbase based extensions
The FLOW3 naming convention is the easiest to implement, because it has very detailed rules where a file implementing a certain class must reside in the filesystem. Unfortunately, no such strict rules exist inside TYPO3 core and classical extensions (example: t3lib_Singleton resides in t3lib/interfaces/class.t3lib_singleton.php).

Thus, the autoloader is divided into two parts:
1) If the class which should be loaded starts with Tx_ (the backported naming convention from FLOW3), we can use a very simple autoloader because of the rigid conventions (implemented in t3lib_autoloader::getClassPathByDirectoryStructure).

2) For classical extensions and the TYPO3 core, we use the "registry" concept which is basically an associative array mapping from class names to fully qualified file names.
Every "classical" extension wanting to use the autoloader needs to ship a file called ext_autoload.php in its root directory (can be autogenerated with extdeveval). The core ships such a file as well, located in t3lib/core_autoload.php.
When the autoloader is started, the core files are automatically added to the registry by loading the t3lib/core_autoload.php file. ext_autoload.php files are loaded on demand when needed.

Resolving a class:
If the autoloader should load a class, it checks if the class starts with Tx_. If yes, it'll use the resolver for the new naming conventions. If no, it looks into the registry and checks if the class name is already in there (This happens in two cases: 1. it is implemented in a core file, 2. it is in an extension which has already been loaded).
If the class is not yet in the registry, then the class name is splitted into its parts and the extension key is resolved. Now, the entries of the ext_autoload.php file of the extension are added to the registry. We maintain a separate cache to speed up this process ($extensionHasAutoloadConfiguration).

Building the ext_autoload.php / core_autoload.php:
We implemented a new extdeveval module to do this. It makes sense to pregenerate it, in order to speed up the autoloader and to make the autoloader as a whole work a lot more predictable. Basically, ext_autoload.php contains metainformation about an extension which will only ever change if the extension is restructured.
The reason why we (currently) use multiple files for the autoloader registry is that typically, we only need very few extensions to render a page. Additional caching can be implemented lateron if we see a need for it.

The autoloader is static to ensure maximum speed.

Notes:
This patch is part of a patch series to implement the autoloader. The patches have been worked out together with Olly, based on work by Dmitry, Ingo, Ingmar, Masi and Jochen.

#19

Updated by Oliver Hader about 12 years ago

Splitted up the autoloader patch to
  • integrate autoloading functionality
  • enable basic autoloading registry

The extdeveval patch belongs to a different RFC for the accordant extension.

#20

Updated by Oliver Hader about 12 years ago

Committed to SVN Trunk:
  • functionality (rev. 5448)
  • enabling the autoloader (rev. 5449)

Also available in: Atom PDF