Project

General

Profile

Actions

Bug #15663

closed

EM2 need more than 32 MB

Added by Christian Leicht about 18 years ago. Updated about 17 years ago.

Status:
Closed
Priority:
Should have
Category:
Extension Manager
Target version:
-
Start date:
2006-02-17
Due date:
% Done:

0%

Estimated time:
TYPO3 Version:
4.0
PHP Version:
4
Tags:
Complexity:
Is Regression:
Sprint Focus:

Description

The EM2 needs more than 32 MB memory_limit .
I think its to much. The most WebSpaces don´t have more than 32MB of Memory for PHP.
I checked 64 MB and it works.

(issue imported from #M2615)


Files

em_1638.diff (112 KB) em_1638.diff Administrator Admin, 2006-07-15 14:18
class.em_tslib_pibase.php (19.9 KB) class.em_tslib_pibase.php Administrator Admin, 2006-07-15 14:19
em_1638_unix_save.diff (61.3 KB) em_1638_unix_save.diff Administrator Admin, 2006-07-27 14:07
EM_MemFix_2006-08-15.diff (21 KB) EM_MemFix_2006-08-15.diff Administrator Admin, 2006-08-15 13:58
EM_MemFix_20061008(4.0.2).diff (21.7 KB) EM_MemFix_20061008(4.0.2).diff Administrator Admin, 2006-10-08 12:33
EM_MemFix_2006-11-28_SVNrev1816.diff (22.7 KB) EM_MemFix_2006-11-28_SVNrev1816.diff Administrator Admin, 2006-11-28 13:57
EM_MemFix_2006-12-10_SVNrev1860.diff (23.4 KB) EM_MemFix_2006-12-10_SVNrev1860.diff Administrator Admin, 2006-12-10 10:39
EM_MemFix_2006-12-11_SVNrev1860.diff (23.4 KB) EM_MemFix_2006-12-11_SVNrev1860.diff Administrator Admin, 2006-12-11 23:10

Related issues 7 (0 open7 closed)

Related to TYPO3 Core - Feature #15640: search word also in title and descriptionsClosedChris topher2006-02-16

Actions
Related to TYPO3 Core - Bug #16912: Warning on uploading extension to TER (unserialize)ClosedOliver Hader2007-01-28

Actions
Has duplicate TYPO3 Core - Bug #15746: Allocating far toooo much memoryClosedSebastian Kurfuerst2006-02-28

Actions
Has duplicate TYPO3 Core - Bug #15817: Retrieve/Update generates extensions.xml.gz file, but shows no extensions at allClosedKarsten Dambekalns2006-03-13

Actions
Has duplicate TYPO3 Core - Bug #16070: Fatal error with function gzfile in class.em_index.php on line 1424ClosedMichael Stucki2006-04-20

Actions
Has duplicate TYPO3 Core - Bug #16028: No matching extensions found.ClosedKarsten Dambekalns2006-04-11

Actions
Has duplicate TYPO3 Core - Bug #16770: Fatal error: Allowed memory sizeClosedMichael Stucki2006-12-07

Actions
Actions #1

Updated by Stefano Cecere about 18 years ago

i had problems qith 24Mb, but worked perfectly with 32 (Debian, PHP 4.1, mysql 4.1)

Actions #2

Updated by Christian Leicht about 18 years ago

I use Suse 9.3 with PHP 4.3.10 and MySQL 4.1.

The TER2 is working with > 43 MB

Actions #3

Updated by old_martinficzel about 18 years ago

i can confim the problem, on my host the extension manager cant retrieve the list of extension from the miror because of to much memory usage.

the look up function with keywords also does not give a result but i think that's because the retieve before did not suceed.

Actions #4

Updated by Marcel Ellerbrok about 18 years ago

Jepp, same problems on my side.

Please make it run with less then 16MB (thats the default memory size)!

Actions #5

Updated by Staffan Ericsson about 18 years ago

An idea might be to import it in parts - update be-ext first, fe-ext there after and so on.

Requering a lot om memory is a problem that is not going to get smaller - the TER will grow and the update function must be able to handle large quantites of data.

But the new ter seems really promising.

Actions #6

Updated by Staffan Ericsson about 18 years ago

It needs more than 32MB now and the memory requierment is growing as new extension is added to the TER.

Actions #7

Updated by Ralf Abele about 18 years ago

I've got the same problem...

Would it be possible as a workaround to just import the extensions.xml.gz into the ter2 in order to get it working ? I've downloaded the file directly from typo3.org/fileadmin/ter/.

I've got the extensions.xml.gz into a local installation by setting max_execution_time to 240 and the memory_limit to 64M. I've copied this file and the extensions.bin and reviewstates.bin to my online version and then this extension manager also had the repository file.

Yet the Problem was not solved because a look up ended in the same fatal error regarding the exhausted memory :-( My online memory limit is 20M at the moment

Actions #8

Updated by RaS about 18 years ago

... after manual deleting of the files
'typo3temp/extensions.xml.gz' and
'typo3temp/extensions.bin'
i was able to load the list of extension from the miror by setting
max_execution_time = 60 and
memory_limit = 8M
in my php.ini

may one of the two deleted files cause the problem???

SYS: WIN2K | APACHE2 | PHP5 | MYSQL4 | TYPO3 4.0rc1

Actions #9

Updated by Franz Holzinger about 18 years ago

I have a big site and use the import/export extension to export the whole site with all tables. Then I get the attached image which asks me what to do with the index.php file.

The error_log file says:

[24-Mar-2006 20:19:11] PHP Fatal error: Allowed memory size of 33554432 bytes exhausted (tried to allocate 3888093 bytes) in /var/www/vhosts/fholzinger.com/httpdocs/TYPO3core/typo3/sysext/impexp/class.tx_impexp.php on line 1122

So no export from the homepage is possible any more.

Actions #10

Updated by Ralf Abele about 18 years ago

Still the same problem in 4.0rc2

Actions #11

Updated by Dimitri Tarassenko about 18 years ago

Out of curiousity, I tried executing

implode(gzfile('extensions.xml.gz'));

- the same technique that is used in the EM to unzip metadata files - in shell (i.e. outside of TYPO3 environment) and even there it would only work with memory limit set at 17M. There are certainly better ways of handling a 5M xml file than splitting it into an array one line per element and then glueing it together into another variable.

Actions #12

Updated by Ralf Abele about 18 years ago

Still the same problem in 4.0 final.

Actions #13

Updated by old_chihoang almost 18 years ago

I have wrote a new EM which works in 24 MB. It could be interesting for you.

the extension-key is ch_lightem.

enjoy.

Actions #14

Updated by old_chihoang almost 18 years ago

If my extension "ch_lightem" is of interest for the core team I can provide a diff-file.

Actions #15

Updated by Karsten Dambekalns almost 18 years ago

@chihoang: Yes, a diff would be cool.

Actions #16

Updated by old_chihoang almost 18 years ago

@Karsten Hachmeister: can I have cvs write to em?

Actions #18

Updated by Karsten Dambekalns almost 18 years ago

@chihoang: Slow down, this isn't just me who has to decide this. The EM is a central component, and I haven't even seen your patch yet - why do you need write access to CVS to attach a patch to this bugtracker item?

Actions #19

Updated by Andreas Wolf almost 18 years ago

I had the same problem some days ago. When I updated the ext. list, it kept saying that the memory is not sufficient until I rose the limit to 64 MB. The same problem occured at work a few days earlier, but when I tried it the second or third time, the allocated 40 MB were sufficient.

But I also noticed another problem: Somehow it seems that if the memory-error occured, the extension list will not be updated when you try again because the EM thinks the list is already up to date. But new or updated extensions won't be displayed.

Actions #20

Updated by old_chihoang almost 18 years ago

@Karsten Hachmeister: I don't need CVS-Write access, although I find it sexy to have it. By the way did you look at my work? It is quite complex. And it's not just one patch, because I have included class tslib_pibase. How can I provide that in a patch?

Actions #21

Updated by Franz Holzinger almost 18 years ago

I have now tried to upload a new version of tt_products 2.5.0 with the latest TYPO3svn and got this error message:

[14-Jul-2006 14:32:08] PHP Fatal error: Allowed memory size of 67108864 bytes exhausted (tried to allocate 16438785 bytes) in /var/www/html/typo3-svn/TYPO3core/trunk/typo3/mod/tools/em/class.nusoap.php on line 268

Then it went fine after changing in /etc/php.ini
former:
memory_limit = 64M ; Maximum amount of memory a script may consume (32MB)

new:
memory_limit = 128M ; Maximum amount of memory a script may consume (32MB)

Actions #22

Updated by Ernesto Baschny almost 18 years ago

@chihoang,

I would find it cool if you could try to incorporate your changes in the current SVN version of TYPO3core. Then you just make a diff and post it here, so we can see what you have changed. This would be the easiest way to start a discussion about it. Else noone (except you) will be able to see what you changed and why.

Actions #23

Updated by old_chihoang almost 18 years ago

@Karsten Hachmeister: I have uploaded 2 files. em_1638.diff is the diff against the svn, 1638 branch. class.em_tslib_pibase.php is an additional class you need to run my diff. it has to placed in the same directory (i.e. em). sorry for any inconvenience.

Actions #24

Updated by Karsten Dambekalns over 17 years ago

Moved the issue to TYPO3 Core and changed category to Extension Manager; updated summary and description accordingly.

~9336: Thanks for the patches

Actions #25

Updated by Ernesto Baschny over 17 years ago

@chihoang

I have taken a quick look at your "patches", and before I go further down, I would need more input:

1) the em_1638.diff is not very useful, because you have changed the newline ending (from unix to MS-format) and all lines have changes. Try to remake the diff using "-b", which will ignore blank changes (including new-lines), so that the patch would be cleaner and we will be able to see what you have really changed.

2) why are you using pi_base? this is a backend module and "pi_base" is meant to be used in frontend plugins. Couldn't you just apply your memory-boost change to the current CORE code, so that we can this fixed, before we go into cosmetical details? That would be cool.

Thanks for your help, Chi!

Actions #26

Updated by old_chihoang over 17 years ago

@ernesto:

1) Fixed
2) At that time I didn't know about class.t3lib_recordlist.php. And I didn't know about the MySQL IN statement, either. For your favour, I will do a clean patch without pi_base in 3 days.

Actions #27

Updated by Ernesto Baschny over 17 years ago

@chihoang

nice, thanks a lot! This already looks much better. If you are reworking that, you could also indent the code with TABs instead of 4 spaces, this way it would integrate better in the CORE code. Another thing I noticed is that the patch is not based on current SVN, if you can, please do (some things have changed since the 4.0.0 release in that file).

As far as I see (I have just skimmed over the code) you are not only improving speed by storing the extension database in SQL, but you also add some features:

- pagination in large result sets, currently using pi_base
- filter by "state" (alpha/beta...) and by category
- maybe other things I overlooked in the code

For a core patch, it would be nicer to split that up into smaller changes, so that we don't have to review such a monster.

The most important thing would be to solve the original bug report problem. Could you create a diff from current SVN version only with the retrieval and storing of the xml data in SQL? Keep all other stuff (display, filters, etc) as they are now in SVN. This would provide us a solution for this bug report without adding more features (yet!). We could then release this in 4.0.2 (no new features) and have that in trunk for later 4.1.x, where we can than come back to the new features.

Thanks again for the help and fast feedback, Chi. Sorry if sometimes you don't get feedback sooner, but I think this is pretty important and all core team is glad it is being looked at.

Actions #28

Updated by Michael Stucki over 17 years ago

Thanks for your patches, Chi!
I tried applying it, but unfortunately it doesn't do either against HEAD nor r1638.

Can you make a new patch please? Please create an additional patch using the -w flag (ignore whitespace changes).

Thanks

Actions #29

Updated by Michael Stucki over 17 years ago

I just notice that I didn't refresh this page for some time ;-) Ignore my last post, everything has been written by Ernesto and Karsten already.

Actions #30

Updated by old_chihoang over 17 years ago

@ernesto: FYI, I'm working with scite, tabs are converted to whitespace, so I hope this -w flag will do the job.

Could you create a diff from current SVN version only with the retrieval and >storing of the xml data in SQL? Keep all other stuff (display, filters, etc) as >they are now in SVN.

display uses SQL too, I will do a diff with display but without filters.

I noticed is that the patch is not based on current SVN

what is the current SVN?

Actions #31

Updated by Ernesto Baschny over 17 years ago

@chi,

FYI, I'm working with scite, tabs are converted to whitespace, so I hope this -w flag will do the job.

The -w will make a good diff, but your code will still have wrong indentation. Maybe you can configure your editor to insert tabs instead of 4 spaces. If not, we also can do that in your patch later, that shouldn't hold you off.

display uses SQL too, I will do a diff with display but without filters.

Right, but it would be better to leave the display engine as it is now without pagination. So don't use pi_base or other routine just yet, use the regular output that we already have.

I noticed is that the patch is not based on current SVN

what is the current SVN?

It's our versioning repository, the successor of CVS. Check out http://sourceforge.net/svn/?group_id=20391 (the web-access to it seems to be broken right now).

Actions #32

Updated by old_chihoang over 17 years ago

@ernesto: all the diffs are based already on the svn. branch 1638 like I named the files. by the way how can I checkout only the EM. Currently I'm fetching 300 MB, thats is the whole svn...

Actions #33

Updated by Michael Stucki over 17 years ago

You don't need to fetch everything! branches/TYPO3_4-0/ is enough.
See this page for easy webaccess: http://svn.t3.digitaldistrict.de/cgi-bin/trac.cgi/browser/typo3/TYPO3core/branches/TYPO3_4-0

Actions #34

Updated by Robert Lemke over 17 years ago

Hi folks,

I have implemented 50% of a new feature in the TER2 server scripts which now create a file called "extensions.bin" parallel to "extensions.xml.gz". It contains (or should contain in the future) exactly the same serialized array which is needed by the extension manager.

You can download a sample extensions.bin file from my server: http://ter.dev.robertlemke.de/fileadmin/ter/extensions.bin

This file isn't complete yet and will most likely create a foreach error or similar in your extension manager but I will finish that tomorrow. Of course we can still follow other paths for solutions, but at least we got something ... it's quick solution we can provide.

Can someone test if that really helps in low-memory enviroments?

Thanks,
robert

Actions #35

Updated by old_chihoang over 17 years ago

@Robert Lemke: Don't test it, it won't help. The only solution is to load it in parts. And to load a serialized array in parts, is imo academic.

Actions #36

Updated by old_chihoang over 17 years ago

@ernesto: at the soureforge page it is told:

(Warning: ... in most cases, you will want to add '/trunk' to the HTTPS URL above to check out only trunk (main development line)).

When I check the EM of the trunk, it has version 1647, When I check the EM from the branch TYPO3_4-0 it has version 1645.

The webaccess trunk and TYPO3_4-0 has no version at all.

My Question: Which branch should I fetch to make my diff? Trunk, TYPO3_4.0, webaccess. ATM webaccess is my favourite.

Actions #37

Updated by Ernesto Baschny over 17 years ago

@chi

I don't know what you mean with "webaccess".

You should make your patch against TYPO3_4.0 (which fixes the EM-memory problem bug). This will later be released as 4.0.2.

We will merge that change later into "trunk", which is the basis of 4.1.x and later, where we then can add the new features, if we want.

Actions #38

Updated by old_chihoang over 17 years ago

@ernesto:

this is webaccess:

You don't need to fetch everything! branches/TYPO3_4-0/ is enough.
See this page for easy webaccess: >http://svn.t3.digitaldistrict.de/cgi-bin/trac.cgi/browser/typo3/TYPO3core/branches/TYPO3_4-0

so it is a bug or a feature?

Actions #39

Updated by Ernesto Baschny over 17 years ago

@chi,

this is only an interface to the branches. You can use a SVN client or this webaccess client, it will access the same repositories. In this case, branches/TYPO3_4-0/ is the "branch" called "TYPO3_4_0". This is the same as you see using a SVN client. So use that!

Actions #40

Updated by old_chihoang over 17 years ago

@ernesto: I'm aware that this is a webinterface. But the version numbers are missing. So, it is a bug or a feature?

Actions #41

Updated by Peter Niederlag over 17 years ago

[OT] @chi: The uperrmost directories in TYPO§core are

branches/*
tags/*
trunk/

trunk/ is for the most recent stuff, tags/* is for singel versions (like 4.0.1rc1 etc.)

As we only ask for a diff against the 4.0 branch you should use 'TYPO3core/branches/TYPO3_4_0/' This will always reflect the most recent code of the 4.x series.

Have a look at http://bscw.fit.fraunhofer.de/bscw_help-4.3/german/sec-09070000.html (German!) how you can easily access this via Windows-Explorer (suggested) or Internet-Explorer: The url you need to supply there is:
https://svn.sourceforge.net/svnroot/typo3/TYPO3core/branches/TYPO3_4_0/

If you want additional comfort have a look at http://tortoisesvn.tigris.org

Feel free to contact me via PM if you need some additional assitance.

Greets,
Peter

Actions #42

Updated by old_chihoang over 17 years ago

@ernesto: This doesn't help me. I'm still missing the version number in the webaccess.

Actions #43

Updated by Ernesto Baschny over 17 years ago

@chi,

I am not sure what else you need. If you need help with using Subversion, I think you should try to read the docs. Either use this URL:

http://svn.t3.digitaldistrict.de/cgi-bin/trac.cgi/browser/typo3/TYPO3core/branches/TYPO3_4-0

If you want to use the web to access the files.

I personally find it easier just to check out the whole branch using the svn client (read the docu or Peter's post on how to do that). From the Unix command prompt:

% svn checkout https://svn.sourceforge.net/svnroot/typo3/TYPO3core/branches/TYPO3_4-0

Or use your favorite SVN client.

I don't think the bug tracker is a good place to get support on how to use SVN. I'm sure you will be able to figure it out, as you seem to be a smart guy. :)
Actions #44

Updated by Bernhard Kraft over 17 years ago

Hello all,

I created a patch against the EM of the 4.0 version of TYPO3 during the Developer Days meeting.

EM_MemFix_2006-08-13_train.diff

I talked to Robert, Karsten and Kasper if they are fine with having an additional table in the core which stores the list of extension and they were all fine with it telling that they already suggested such a thing.

I named the table "cache_extensions" as it is a table which contains just the cached extension list from TER and can allways get regenerated using the "Update/Retrieve" button.

When I had finished my patch I took a look at Chi Hoang's solution (believe me - I read your code afterwards) and found that he mostly did what i did. But instead of using unreadable strpos constructs I rather used an regexp to split the extensions.xml file to single <extenssion> ... </extension> pieces.

I also named the table not like an extension table - which would not be fine if we want to integrate the extension into the core.

@Chi Hoang: If you want to have your code integrated into the core you must have it look like exactly like all other parts of the core - I hope this is understandable for you.
You should read the coding guidelines:
http://typo3.org/documentation/document-library/core-documentation/doc_core_cgl/current/
Cause you allowed SQL injections in your code. Altough the EM is only available to Admins we do not want such constructs as:
$query = ...' AND variable='.$postvar.'
in our code. You always have to escape posted values properly.

You also should stick to TABs as indenting element and only use \n as linebreaks. This is all documented in the coding guidelines.

The reason for this is simple: We do not want our code to look if it would be patchwork from hundreds of people ... it shall look like one neat piece of code :)

It does not make really sense to create a patch from an extension - you should rather create a patch for the core and then make an extension out of it.

This is what I did. The extension key is "kb_test_em" and this testing extension for those who are to lazy to apply a patch will become available in TER during the next hours.

Chi Hoang: I was inspired by your idea of having a pagebrowser which I added to my code AFTER reading yours !
But you should not have used tslib_pibase in the BE. tslib classes are purely for use in the FE. You can use them in your own extensions as you like and nobody will keep you off using BE classes in FE and vice versa (I do that myself) - but it has not to get into the core.

Actions #45

Updated by old_chihoang over 17 years ago

- why only 1 table? why not 3 tables with true m:n relationship?
- strpos might be faster then preg_match
- do less INSERTS by combining them
- store icons url in the database (prefetch)
- unset unused variables
- why not use pibase in be when it enables such a performance? is it to high cost?
- could you please upload 2 diffs, for each patch 1 diff? it is unreadable!

Actions #46

Updated by Bernhard Kraft over 17 years ago

- why only 1 table? why not 3 tables with true m:n relationship?

  • People like Kasper are very restrictive about new tables - I could convince him that at least one new table is necessary for storing the extensions. Why do you need three of them ? You can't store the extension "basis" information in some table and all available versions in some other - cause each version can have different basis information (for example the extension "adaltas_doktype" has different categories between different versions, etc.).

- strpos might be faster then preg_match

  • But strpos constructs are more "fragile". If for example once the return "false" you would handle it the same as "0" ... As far as I read your code. Of course you can come around this but you even have a fixed string lenght value (55 or so) at the beginning to skip the

- do less INSERTS by combining them

  • You should not use "$GLOBALS['TYPO3_DB']->sql_query" as this is not fully DBAL compatible and takes more time to get parsed (an effect when DBAL is installed) than the ->exec_INESRTquery method. You should rather stick to those exec_ methods. The speed is not that different and as far as I can read from your code you also use exec_INSERTquery for the extensions themselve.

- store icons url in the database (prefetch)

  • It does not really help to simply store the icons URL - it would only help if you really prefetch the icons from the server (is this is what you meant by prefetch) - and store them in some location on the client. Currently extension icons do not get shown in the EM - cause people posted a bug that it takes sometimes minutes to load all those icons and while they get loaded the size of the list changes all the time so you can't easily click an icon on the bottom.

- unset unused variables

  • I did that - which ones are you missing ?

- why not use pibase in be when it enables such a performance? is it to high cost?

  • As I said the tslib_ classes are strictly for the FE. Possibly in 4.5 or 5.0 this differenciation between BE and FE will get dropped but currently this is the situation and we would not like to change this. the _pibase class is a base class for FE plugins and should never get used in the BE. If you need to retrieve records from BE modules have a look at the class t3lib_BEfunc - but this one does not allow you to create "raw" SQL queries but can just get used for fetching specific records (where the tables must comply with the T3 rules like having "uid" and "pid" fields, etc.). For fetching "raw" SQL records in the BE you should use ->exec_SELECTquery or a similar one.

- could you please upload 2 diffs, for each patch 1 diff? it is unreadable!

  • This is only ONE patch which contains changes in TWO files. In the core list we keep it that way that patches which belong together and wouldn't work on their own are combined in two files. Which part do you find undreadable ? There is a mark:

diff ru typo3_src-4.0/typo3/mod/tools/em/class.em_xmlhandler.php typo3_src-4.0.em/typo3/mod/tools/em/class.em_xmlhandler.php
--
typo3_src-4.0/typo3/mod/tools/em/class.em_xmlhandler.php 2006-04-07 02:18:29.000000000 0200
++ typo3_src-4.0.em/typo3/mod/tools/em/class.em_xmlhandler.php 2006-08-13 23:02:42.722728480 +0200

somewhere in the middle of the patch which specifies that from here on the new files begin - if you are not familar with
such usage you should quickly get used to it :) cause it is common in T3 worlds ....

Of course I will include your name into the Changelog when applying the patch cause you supplied lot of information and additional ideas like the page-browser or the icon caching. About the icon caching: I will have a look when and how icons can still get displayed - and if a user has the possibility to show the icons I will add the possibility to prefetch them from the server - I would not like to add this feature by default as possibly many people do not want 1800 icons to get loaded onto their server ... I guess you understand the reasons.

Actions #47

Updated by Bernhard Kraft over 17 years ago

I've also added a new version:

EM_MemFix_2006-08-15.diff

the last one missed the CREATE table statements in t3lib/stddb/tables.sql

Actions #48

Updated by old_chihoang over 17 years ago

  • People like Kasper are very restrictive about new tables - I could convince him that at least one new table is necessary for storing the extensions. Why do you need three of them ? You can't store the extension "basis" information in some table and all available versions in some other - cause each version can have different basis information (for example the extension "adaltas_doktype" has different categories between different versions, etc.).

- I will remember that.

  • But strpos constructs are more "fragile". If for example once the return "false" you would handle it the same as "0" ... As far as I read your code. Of course you can come around this but you even have a fixed string lenght value (55 or so) at the beginning to skip the

- strpos is safe. see php manual. And in my algorithm it is not necessary to check for open tags. Keep it simple!

  • You should not use "$GLOBALS['TYPO3_DB']->sql_query" as this is not fully DBAL compatible and takes more time to get parsed (an effect when DBAL is installed) than the ->exec_INESRTquery method. You should rather stick to those exec_ methods. The speed is not that different and as far as I can read from your code you also use exec_INSERTquery for the extensions themselve.

- I will remember that.

  • It does not really help to simply store the icons URL - it would only help if you really prefetch the icons from the server (is this is what you meant by prefetch) - and store them in some location on the client. Currently extension icons do not get shown in the EM - cause people posted a bug that it takes sometimes minutes to load all those icons and while they get loaded the size of the list changes all the time so you can't easily click an icon on the bottom.

- Yes, it helps. You don't need to prefetch the icons, it should be enough to precalculate the url and store it in the db -> it saves some, maybe little time.

  • I did that - which ones are you missing ?

- I can't see any unset in your code. Did you understand me?

  • As I said the tslib_ classes are strictly for the FE. Possibly in 4.5 or 5.0 this differenciation between BE and FE will get dropped but currently this is the situation and we would not like to change this. the _pibase class is a base class for FE plugins and should never get used in the BE. If you need to retrieve records from BE modules have a look at the class t3lib_BEfunc - but this one does not allow you to create "raw" SQL queries but can just get used for fetching specific records (where the tables must comply with the T3 rules like having "uid" and "pid" fields, etc.). For fetching "raw" SQL records in the BE you should use ->exec_SELECTquery or a similar one.

- I will remember that. Although I think there should be exepctions. And this is one. I myself won't miss the comfort of my patch anymore.

  • This is only ONE patch which contains changes in TWO files. In the core list we keep it that way that patches which belong together and wouldn't work on their own are combined in two files. Which part do you find undreadable ? There is a mark:

diff ru typo3_src-4.0/typo3/mod/tools/em/class.em_xmlhandler.php typo3_src-4.0.em/typo3/mod/tools/em/class.em_xmlhandler.php
--
typo3_src-4.0/typo3/mod/tools/em/class.em_xmlhandler.php 2006-04-07 02:18:29.000000000 0200
++ typo3_src-4.0.em/typo3/mod/tools/em/class.em_xmlhandler.php 2006-08-13 23:02:42.722728480 +0200

somewhere in the middle of the patch which specifies that from here on the new files begin - if you are not familar with
such usage you should quickly get used to it :) cause it is common in T3 worlds ....

- I will remember that.

Actions #49

Updated by Bernhard Kraft over 17 years ago

unset's are here:

-----------------------
@ -815,9 +824,11 @
}

$lines[]=$this->extensionListRow($extKey,$ext,array('&lt;td class=&quot;bgColor&quot;&gt;'.$loadUnloadLink.'&lt;/td&gt;'),$theRowClass,$inst_list,1,'index.php?CMD[importExtInfo]='.rawurlencode($extKey));
+ unset($list[$extKey]);
}
}
}
+ unset($list);
----------------------------------

and here:

----------------------------
@ -3057,8 +3092,10 @
);
}
$this->setCat($cat, $list[$extKey]['versions'][$version], $extKey);
+ if ($unsetProc) {
+ unset($this->xmlhandler->extensionsXML[$extKey]);
+ }
---------------------------

the only locations where I could found the list to get copied ...

Actions #50

Updated by Dimitrij Denissenko over 17 years ago

Uploaded EM_MemFix_20061008\(4.0.2\).diff.

It is an adaptation of the original EM_MemFix_2006-08-15.diff to the current stable TYPO3 version (4.0.2).

Usage:
patch -p1 < EM_MemFix_20061008(4.0.2).diff

Actions #51

Updated by Michael Stucki over 17 years ago

Link to kb_test_em extension: http://typo3.org/extensions/repository/view/kb_test_em/0.0.1/

Please test and provide feedback.

Actions #52

Updated by Staffan Ericsson over 17 years ago

kb_test_em generates no errors for me but the update date is not changed at all.

TER2 update tool is not responding either. It reports update successfull but the date is old and no updated extensions are visible.

Actions #53

Updated by Franz Koch over 17 years ago

@test extension
well, finally the EM works on shared hosts like 1&1 but it only lists the reviewed extensions even if the override checkbox is set.
The bug resides in a naming error in line 55 of class.ux_em_index.php.

wrong:
$this->xmlhandler->useUnsupported = $this->MOD_SETTINGS['display_unsupported'];

correct:$this->xmlhandler->useUnsupported = $this->MOD_SETTINGS['display_unchecked'];

tested with 4.0.2

Actions #54

Updated by Bernhard Kraft over 17 years ago

Hi !

I uploaded a new diff made against revision 1816 from SVN ...

it contains fixes for the last two problems reported by Staffan Ericson and Franz Xaver Koch - it is based on the updated version by Dmitry.

Actions #55

Updated by Benni Mack over 17 years ago

So will this be fixed in TYPO3 4.0.3 / 4.1 ?

Actions #56

Updated by Michael Stucki over 17 years ago

Not in 4.0.x, but in 4.1

Actions #57

Updated by Karsten Dambekalns over 17 years ago

The attached diff against rev 1860 will go into SVN if no objections are raised on the core list within 24 hours.

The diff is based on the rev 1816 diff done by Bernhard and changes the following things:

  • Added escapeStrForLike() call to extension search em_xmlhandler::searchExtensionsXML() so one can search for literal _% in ext keys
  • Fixed arguments in em_xmlhandler::loadExtensionsXML() by adding another empty param before the "true"
  • Added an ini_set('pcre.backtrack_limit', 500000) call to avoid running into the PCRE backtrack limit for extension with a large number of versions and/or long comments (one extension-block longer than the default 100000 byte, e.g. with "ext_tbl", 102 versions in TER)
  • Changed the way the EOF is detected (no string length comparison anymore)
  • Removed nonsense 1==1 in if-clause in fetchExtension(), circumventing error checking (probably left there by accident)
  • The XML parser is now always set to utf-8 for the extensions XML file, added preg modifier "u" to enable utf-8
  • various minor cleanups
Actions #58

Updated by Staffan Ericsson over 17 years ago

Which base is diff 1860 built from? Could I apply it on a 4.0.n installation or do I need to apply several of the diffs?

Stucki wrote that it will not be applied to 4.0.x branch - why not? Isn't it really a bug fix that should be applied to the stable version?

Actions #59

Updated by Karsten Dambekalns over 17 years ago

Regarding ~11359: since the EM is about the same for 4.0.x and 4.x.x the patch should apply cleanly to 4.0.x, too. You only need the latest (my) patch.

This won't go into 4.0.x, as it changes the database, which is a no-no with our release policy.

Actions #60

Updated by Staffan Ericsson over 17 years ago

Tested patch 1860 on T3-4.1beta1a without success.

Unpacked a fresh source
Applied patch
Ran install tool to update DB
Tried EM without sucess.

Extensions.xml.gz is fetched to typo3temp

No error messages but only 54 records in table cache_extensions. No warnings , no timeout errors or out of memory errors so far - just can't get all extension into the database.

Actions #61

Updated by Karsten Dambekalns over 17 years ago

~11379: what PHP version are you using?

Actions #62

Updated by Staffan Ericsson over 17 years ago

PHP 4.3.10-18 (cli) (built: Nov 3 2006 21:56:29)
Copyright (c) 1997-2004 The PHP Group
Zend Engine v1.3.0, Copyright (c) 1998-2004 Zend Technologies

Actions #63

Updated by Karsten Dambekalns over 17 years ago

Staffan, that is strange. I ran into a problem that "felt" the same and was caused by hitting the pcre.backtrack_limit - but that is only available since PHP 5.2.0, so it should not be causing your problems. Could you send me the extensions.xml.gz and a dump of the cache_extensions table ()? Maybe that gives me a hint...

Actions #64

Updated by Karsten Dambekalns over 17 years ago

The patch introduces preg_last_error() usage, but i just discovered that this has been added in PHP 5.2, so the code is broken for all older PHP versions. I fixed that and uploaded a new patch (EM_MemFix_2006-12-11_SVNrev1860.diff).

Actions #65

Updated by Staffan Ericsson over 17 years ago

Patch EM_MemFix_2006-12-11_SVNrev1860.diff did the job - works good now.

The list of "extension found only on this server" seems a little bit strange. Got trunkload of extensions there as false postives. (For example tt_news)

A warning might be needed. The patch breaks ext. ter_update_check

.staffan

Actions #66

Updated by Karsten Dambekalns over 17 years ago

I know it breaks the update check extension, I'll see into fixing this. That should be integrated into the EM anyway...

Actions #67

Updated by Karsten Dambekalns over 17 years ago

The fix is in SVN now. Phew. Thanks to all contributors!

Actions

Also available in: Atom PDF