Bug #18048: Generating extension import list takes very long - TYPO3 Core - TYPO3 Forge

Actions

Copy link

Bug #18048

closed

Generating extension import list takes very long

Added by Steffen Heinrich over 16 years ago. Updated over 12 years ago.

Status:

Rejected

Priority:

Should have

Assignee:

Category:

Extension Manager

Target version:

Start date:

2008-01-24

Due date:

% Done:

Estimated time:

TYPO3 Version:

4.1

PHP Version:

5.2

Tags:

Complexity:

Is Regression:

Sprint Focus:

Description

Apache 1.3, MySQL 5.0.27, PHP 5.2.3, Windows XP

Importing from TER and searching the imported list is taking about 6 minutes.

At first I had to increase max_execution_time to 600 in the php.ini to get beyond the white page. (The .tgz archive was always properly downloaded very quickly, but the inserts into table cache_extensions took about 5 minutes.)

Next I found that every lookup into the cached information also consumed around 6 minutes and extensions were sometimes not found, although they were in the table.

Some profiling done on class.em_index.php reveiled, that 96% of that time was spent on the effort to identify installed extensions that are not found in the cached repository. Specifically, for each such extension in line 861 this call gets executed:
$this->xmlhandler->searchExtensionsXML($extKey, '', '', true);

And inside this function which seems to have been implemented to retrieve / filter the repository once and for all, several requests to the db are done.
I'm sure this can be implemented more cleverly. (And I also put in question that this information "is-in-repository: yes/no" is relevant to many people.)

Cheers, Steffen

(issue imported from #M7264)

Files

7264.diff (881 Bytes) 7264.diff

Administrator Admin, 2009-08-04 10:30

Actions

Copy link

Updated by Steffen Heinrich over 16 years ago

BTW, it looks as though this bug was introduced by fixing bug 4822
("Import Extensions" shows all imported extensions also as "only found on this server") in January 2007, and that this may have caused some or many of EM's time/memory issues reported.

http://bugs.typo3.org/view.php?id=4822

Actions

Copy link

Updated by Franz Holzinger over 15 years ago

I confirm this bug with TYPO3 4.3svn on my notebook with AMDTurionX2:
Unfortunaltely I can hardly import any extension from TER:

Fatal error: Maximum execution time of 30 seconds exceeded in /var/www/html/trunk/typo3/mod/tools/em/class.em_xmlhandler.php on line -2115769472
Call Stack

Time Memory Function Location
1 0.0067 117320 {main}( ) ../index.php:0
2 2.0865 45237240 SC_mod_tools_em_index->main( ) ../index.php:54
3 2.0889 45240544 SC_mod_tools_em_index->extensionList_import( ) ../class.em_index.php:611
4 3.3571 49420528 SC_mod_tools_em_index->fetchMetaData( ) ../class.em_index.php:865
5 5.5276 50962792 SC_mod_tools_em_xmlhandler->parseExtensionsXML( ) ../class.em_index.php:1601
6 34.7423 52314656 xml_parse ( ) ../class.em_xmlhandler.php:377
7 34.7516 52331552 SC_mod_tools_em_xmlhandler->characterData( ) ../class.em_xmlhandler.php:0

Actions

Copy link

Updated by Michiel Roos almost 15 years ago

Attached file 7264.diff.

This obvious optimization reduces the number of SQL queries done by the function searchExtensionsXML by 50%.

Actions

Copy link

Updated by Michael Stucki almost 15 years ago

The idea is nice but it will not work.

Have a look at the code, it contains a limit statement in the 2nd query. However, this means that mysql_num_rows() will return $limit instead of the real number of results.

For a workaround, have a look here: http://dev.mysql.com/doc/refman/5.1/en/information-functions.html#function_found-rows

However, this again means that this makes the code less portable. Suggestion: First, introduce a global flag that indicates what DBMS is used, then apply the new query containing SQL_CALC_FOUND_ROWS, and as a fallback, stick to the old way using two queries...

After all, it's not that easy as it seems...

Actions

Copy link