Project

General

Profile

Actions

Bug #58948

closed

EM in 6.2.2 not loading list of extension correctly

Added by Paul A. Auwehla over 10 years ago. Updated about 9 years ago.

Status:
Closed
Priority:
Should have
Category:
Extension Manager
Target version:
-
Start date:
2014-05-20
Due date:
% Done:

0%

Estimated time:
TYPO3 Version:
6.2
PHP Version:
5.3
Tags:
Complexity:
medium
Is Regression:
No
Sprint Focus:
Remote Sprint

Description

There are lots of reports to EM around here.
But EM still is not working stable as expected.

The problem seems that the process of getting the
extension list from TER is not stable and has no error or state checking.
Situation on a very good working installation of 6.2.2 is the following:

The extension-list is fetched from TER as 2 files in typo3temp:
1.extensions.xml.gz
1.mirrors.xml.gz
Is there any check whether this files are complete ok or not?
By hash or size or extensions included or anything else?

These 2 xml-lists are then read into 2 db-tables:
tx_extensionmanager_domain_model_repository (one row)and
tx_extensionmanager_domain_model_extension (around 20500 rows)

(1) problem seems that the fields
tx_extensionmanager_domain_model_repository-last_update and
tx_extensionmanager_domain_model_repository-extension_count
are not updated correctly.
Sometimes EM finds the list is never updated,
sometimes it is updated 2 years ago
sometimes it is yesterday, with still the same or newer extensions.xml from typo3temp.

The db-field -extension_count is not set correctly.
Sometimes it is empty, sometimes there are the 6.xxx extensions found in the extension.xml.
It does not matter which size and row-count the exntensions-table is.

(2) problem seems that the update of the db-tables from the xml
file is without any error check.
The xml is read into db-table over and over again from the very beginning.
You can see the EM fill the table from 0 to 20500 rows if you are lucky.
But most often the update process stops before reaching the full count of rows.
16000 or 11500 rows are read into table or something like this.
Looks like the process is not checking whether the data is received
from the DBserver correctly.

If the db-update process fails to get all data into db-table it tries again sending from the first row.
So if there is a tiny timing or update problem with the DBserver this is like playing for good luck.

Then the two fields from (1) problem are not set as expected ...

PS.
Removed all files in typo3temp/* and all caches over and over again,
and deleted/truncated the two db-tables and recreating them with Install-tool
does help only randomly and by good luck - sometimes.

Actions #1

Updated by Markus Klein over 10 years ago

  • Status changed from New to Needs Feedback
  • Target version deleted (next-patchlevel)

Reads like your max_execution_time is reached and the import process to the DB is getting stopped. At least this is what we figured out on an installation where we had the same symptoms.

Actions #2

Updated by Paul A. Auwehla over 10 years ago

Exactly, this time-out is the problem.
Chances are the time-out comes from the db-server importing 20.000 records.
On hosted webspace you cannt set time out to 240 or even more.

Solution could be to send just the records still to update to the db-server
instead of sending the whole extensions list to the db-server again and again
from the very first record.

We do have the time of the last update, so it should be no problem to
see on update-php-scripts start whether to send the whole list of extensions
or just the remaining x records not yet in db-table.

Or to make sure by php that there is no
timeout by restarting db-update more than once when needed.
Or just sending xy records at a time and waiting for db-servers answers.

Actions #3

Updated by Paul A. Auwehla over 10 years ago

Not all hosted webspaces do allow script time-outs of 240 SECONDS or even more.
Saving more than 20.000 records in 240 seconds is a pretty good value for hosted db-servers!?

Actions #4

Updated by Markus Klein over 10 years ago

We got that working on shared hosting with execution times below 100 seconds. So the DB backend seems to be really slow on your side.

Still, I agree there's much room for improvement, but I've not time to dig into this code. Maybe you can?

Actions #5

Updated by Markus Klein over 10 years ago

  • Category set to Extension Manager
  • Status changed from Needs Feedback to Accepted
  • Complexity set to medium
Actions #6

Updated by Nicole Cordes over 9 years ago

  • Target version set to 7.3 (Packages)
  • Sprint Focus set to On Location Sprint
Actions #7

Updated by Benni Mack over 9 years ago

  • Target version changed from 7.3 (Packages) to 7.4 (Backend)
Actions #8

Updated by Pierrick Caillon over 9 years ago

  • Status changed from Accepted to In Progress
  • Assignee set to Pierrick Caillon
Actions #9

Updated by Christian Kuhn over 9 years ago

Hey,

thanks for this report. The issue at this point is that the current import process of the extension list is already optimized pretty well and can't be changed without refactoring the whole thing on a structural level and with a lot of effort.

The max_execution_time is checked in the system within the install tool, the default system requirement of TYPO3 is to have 240 seconds execution time. This is done to mitigate the risk of such cases were a broken import will leave the extension manager with broken or incomplete data. Also, this limit is sometimes important for other time consuming tasks by the system.

At the moment, the sytems warns at a couple of places if this limit is ignored: The install tool shown this in the environment test, it is shown in the default screen that is rendered for admins after login. Furthermore, when installing the system on an environment that does not fit TYPO3 needs, a button "i know what i'm doing" has to be pressed.

At the moment, I tend to reject this issue with "won't fix" since there is no way to get rid of this longish import process currently, and any change here (eg. an incremental import based on ajax) puts the structure of the extension manager at risk and is very time consuming. We will not do that at the moment.

Actions #10

Updated by Christian Kuhn over 9 years ago

  • Status changed from In Progress to Needs Feedback

Setting this issue to "need feedback". This issue will closed soon, as long as the issue author does not provide a bullet proof patch, since no core related person will probably work on this soon.

Actions #11

Updated by Susanne Moog over 9 years ago

  • Sprint Focus changed from On Location Sprint to Remote Sprint
Actions #12

Updated by Susanne Moog over 9 years ago

  • Target version changed from 7.4 (Backend) to 7.5
Actions #13

Updated by Benni Mack about 9 years ago

  • Target version deleted (7.5)
Actions #14

Updated by Wouter Wolters about 9 years ago

  • Status changed from Needs Feedback to Closed

In version 7 we optimized this even more. Not much more to improve on the TYPO3 CMS side. Closing this ticket now.

Actions

Also available in: Atom PDF