Bug #26993

Pagetree: Copying large branches duplicates branch

Added by Steffen Gebert over 8 years ago. Updated about 1 year ago.

Status:
Closed
Priority:
Should have
Category:
Backend User Interface
Target version:
-
Start date:
2011-05-23
Due date:
% Done:

100%

TYPO3 Version:
4.5
PHP Version:
Tags:
Complexity:
Is Regression:
Sprint Focus:

Description

Copying large branches (500+ pages, depending on the server) has a big problem: The AJAX request is answered too late (after Ext.Direct timeout), so Ext.Direct will resend the request. This will copy the branch once again. So the copied branch appears twice after copying.

The whole process fails with a connection timeout error message and the user has to reload the page tree.


Related issues

Related to TYPO3 Core - Bug #44055: ExtJS connection timeout when mass publishing workspace changes Closed 2012-12-17
Related to TYPO3 Core - Bug #49813: Copying large branches creates multiple copies New 2013-07-09
Duplicated by TYPO3 Core - Bug #29123: Duplicate copy created when copying larger page trees Closed 2011-08-19
Duplicated by TYPO3 Core - Bug #32632: Doppelter Seitenbaum nach Kopie Rejected 2011-12-16

Associated revisions

Revision c35860c6 (diff)
Added by Steffen Gebert over 7 years ago

[BUGFIX] Pagetree: Copying large branches duplicates branch

Copying large branches (500+ pages, depending on the server) has a big
problem: The AJAX request is answered too late (after Ext.Direct
timeout), so Ext.Direct will resend the request. This will copy the
branch once again. So the copied branch appears twice after copying.

The whole process fails with a connection timeout error message and the
user has to reload the page tree.

Change-Id: If0307586ec3196c3c3dedde92cfb6aee07bc9064
Resolves: #26993
Releases: 6.0, 4.7, 4.6, 4.5
Reviewed-on: http://review.typo3.org/8981
Reviewed-by: Stefan Neufeind
Reviewed-by: Stefan Galinski
Reviewed-by: Philipp Gampe
Tested-by: Stefan Galinski
Reviewed-by: Susanne Moog
Tested-by: Susanne Moog

Revision c4d93451 (diff)
Added by Steffen Gebert over 7 years ago

[BUGFIX] Pagetree: Copying large branches duplicates branch

Copying large branches (500+ pages, depending on the server) has a big
problem: The AJAX request is answered too late (after Ext.Direct
timeout), so Ext.Direct will resend the request. This will copy the
branch once again. So the copied branch appears twice after copying.

The whole process fails with a connection timeout error message and the
user has to reload the page tree.

Change-Id: Id45c2cb213b6b8bd85360641a9275a38467ab96a
Resolves: #26993
Releases: 6.0, 4.7, 4.6, 4.5
Reviewed-on: http://review.typo3.org/11334
Reviewed-by: Susanne Moog
Tested-by: Susanne Moog
Reviewed-by: Christian Kuhn
Tested-by: Christian Kuhn

Revision 1def46ae (diff)
Added by Steffen Gebert over 7 years ago

[BUGFIX] Pagetree: Copying large branches duplicates branch

Copying large branches (500+ pages, depending on the server) has a big
problem: The AJAX request is answered too late (after Ext.Direct
timeout), so Ext.Direct will resend the request. This will copy the
branch once again. So the copied branch appears twice after copying.

The whole process fails with a connection timeout error message and the
user has to reload the page tree.

Change-Id: I2f0aa73357674751e7745654383f420802f0eb17
Resolves: #26993
Releases: 6.0, 4.7, 4.6, 4.5
Reviewed-on: http://review.typo3.org/11335
Reviewed-by: Susanne Moog
Tested-by: Susanne Moog
Reviewed-by: Christian Kuhn
Tested-by: Christian Kuhn

Revision d4da7991 (diff)
Added by Steffen Gebert over 7 years ago

[BUGFIX] Pagetree: Copying large branches duplicates branch

Copying large branches (500+ pages, depending on the server) has a big
problem: The AJAX request is answered too late (after Ext.Direct
timeout), so Ext.Direct will resend the request. This will copy the
branch once again. So the copied branch appears twice after copying.

The whole process fails with a connection timeout error message and the
user has to reload the page tree.

Change-Id: I133566bf12c7511f7e8ef9cb9e6ff8683f07249c
Resolves: #26993
Releases: 6.0, 4.7, 4.6, 4.5
Reviewed-on: http://review.typo3.org/11336
Reviewed-by: Susanne Moog
Tested-by: Susanne Moog
Reviewed-by: Christian Kuhn
Tested-by: Christian Kuhn

Revision b27cc992 (diff)
Added by Steffen Gebert over 7 years ago

[BUGFIX] Pagetree: Copying large branches duplicates branch

Copying large branches (500+ pages, depending on the server) has a big
problem: The AJAX request is answered too late (after Ext.Direct
timeout), so Ext.Direct will resend the request. This will copy the
branch once again. So the copied branch appears twice after copying.

The whole process fails with a connection timeout error message and the
user has to reload the page tree.

Change-Id: If0307586ec3196c3c3dedde92cfb6aee07bc9064
Resolves: #26993
Releases: 6.0, 4.7, 4.6, 4.5
Reviewed-on: http://review.typo3.org/8981
Reviewed-by: Stefan Neufeind
Reviewed-by: Stefan Galinski
Reviewed-by: Philipp Gampe
Tested-by: Stefan Galinski
Reviewed-by: Susanne Moog
Tested-by: Susanne Moog

Revision b4d5351c (diff)
Added by Helmut Hummel almost 7 years ago

[BUGFIX] ExtJS connection timeout when mass publishing

When doing a mass publish in large installations
it can happen that the ExtJS Ajax request runs into
the 30 Seconds timeout limit which results in weird
error messages.

Increasing the ExJS timeout, like it has been done for
the pagetree copy operations solved the problem.

Fixes: #44055
Related: #26993
Releases: 6.1, 6.0, 4.7, 4.6, 4.5

Change-Id: I05d81635604a9ea983ab6ee272afde64cfa69526
Reviewed-on: https://review.typo3.org/17192
Reviewed-by: Mattias Nilsson
Tested-by: Mattias Nilsson
Reviewed-by: Anja Leichsenring
Tested-by: Anja Leichsenring
Reviewed-by: Helmut Hummel
Tested-by: Helmut Hummel

Revision e999cebe (diff)
Added by Helmut Hummel almost 7 years ago

[BUGFIX] ExtJS connection timeout when mass publishing

When doing a mass publish in large installations
it can happen that the ExtJS Ajax request runs into
the 30 Seconds timeout limit which results in weird
error messages.

Increasing the ExJS timeout, like it has been done for
the pagetree copy operations solved the problem.

Fixes: #44055
Related: #26993
Releases: 6.1, 6.0, 4.7, 4.6, 4.5

Change-Id: I05d81635604a9ea983ab6ee272afde64cfa69526
Reviewed-on: https://review.typo3.org/17495
Reviewed-by: Helmut Hummel
Tested-by: Helmut Hummel

Revision 2736f911 (diff)
Added by Helmut Hummel over 6 years ago

[BUGFIX] ExtJS connection timeout when mass publishing

When doing a mass publish in large installations
it can happen that the ExtJS Ajax request runs into
the 30 Seconds timeout limit which results in weird
error messages.

Increasing the ExJS timeout, like it has been done for
the pagetree copy operations solved the problem.

Fixes: #44055
Related: #26993
Releases: 6.1, 6.0, 4.7, 4.6, 4.5

Change-Id: I05d81635604a9ea983ab6ee272afde64cfa69526
Reviewed-on: https://review.typo3.org/17620
Reviewed-by: Helmut Hummel
Tested-by: Helmut Hummel

Revision f6b81f19 (diff)
Added by Helmut Hummel over 6 years ago

[BUGFIX] ExtJS connection timeout when mass publishing

When doing a mass publish in large installations
it can happen that the ExtJS Ajax request runs into
the 30 Seconds timeout limit which results in weird
error messages.

Increasing the ExJS timeout, like it has been done for
the pagetree copy operations solved the problem.

Fixes: #44055
Related: #26993
Releases: 6.1, 6.0, 4.7, 4.6, 4.5

Change-Id: I05d81635604a9ea983ab6ee272afde64cfa69526
Reviewed-on: https://review.typo3.org/17618
Reviewed-by: Helmut Hummel
Tested-by: Helmut Hummel

History

#1 Updated by Alban Cousinie almost 8 years ago

I confirm the issue.

This is a real problem, as :
- it doubles the count of records created in the database (imagine when you duplicate a big website with thousand objects).
- also the copy /paste performance is currently very bad, so doubling the database copy operations is likely slowing down even more the operation
- there is no indication of the operation's progress status as stated in Bug #25056 (http://forge.typo3.org/issues/25056) so you don't really know when it's finished
- the page tree doesn't refresh during the operation, and you don't know if you can refresh it manually without compromising the operation (I assume as it is an AJAX based operation, you can refresh the operation, but I don't know it for sure).
- when duplicating large trees, often the operation expires because it takes so long to copy (I really mean several minutes) and PHP likely ends killing the script if the max_execution_time has not been extended to a huge value like 1200 secondes.

Overall, copying large page trees with Typo3 has become a tedious and buggy operation.

#2 Updated by Steffen Gebert almost 8 years ago

It didn't get slower with the new Page tree - it only became more unpredictable, what's going on. After 30sec (Ext.Direct timeout) the second request is fired. I already fixed that for the Extension Manager, will try to disable this also for the page tree.

#3 Updated by Alban Cousinie almost 8 years ago

Great news, thanks Steffen.

After having tested, I confirm the page tree view can be reloaded without affecting the copy operation.

I have been working since 6 hours on duplicating a separate working copy of that big website I was talking above and I am not close from being done with it. The performance is really poor on that dedicated server I'm running on, which is a 2.5Ghz intel quad core running only this website. Typo3 is currently duplicating folders of tt_address records at a rate of 272 records per minute only. This is extremely low given the horsepower of the server. (I can see however only one core is being used by the copy operation since scripts cannot be threaded inside web pages).

Also I have noticed when duplicating another website, when there are plugins in pages, instead of being copied, they are re-created as references to the source page tree instead of copies. The problem it raises is when you delete the source tree, you get all the plugins in pages erased from the target tree. I don't know if it is a core or a templavoila related issue.

Regards,

Alban

#4 Updated by Alban Cousinie almost 8 years ago

Here is the result of another test : copying a single folder full of 1230 tt_adress records is done in 4 seconds. So it copies 307 records / sec, while in recursive mode the rate of copy was 4.5 records / sec

Something is really bizarre in the recursive copy algorithm.

While looking at the tt_address records duplicated using the single folder copy, and those duplicated with the recursive copy I noticed in the first case the IDs of the records contained in the folder are unchanged, so the folder is copied, but the records are the same. In the second case (recursive copy) the records ID have are different, which suggests they are truly duplicated. This should explain the performance difference.

#5 Updated by Gerrit Code Review almost 8 years ago

  • Status changed from New to Under Review

Patch set 1 for branch master has been pushed to the review server.
It is available at http://review.typo3.org/8981

#6 Updated by Steffen Gebert almost 8 years ago

  • Category set to Backend User Interface
  • Assignee set to Steffen Gebert

#7 Updated by Thorsten Kahler almost 8 years ago

  • Subject changed from Pagegree: Copying large branches duplicates branch to Pagetree: Copying large branches duplicates branch

#8 Updated by Gerrit Code Review over 7 years ago

Patch set 2 for branch master has been pushed to the review server.
It is available at http://review.typo3.org/8981

#9 Updated by Gerrit Code Review over 7 years ago

Patch set 3 for branch master has been pushed to the review server.
It is available at http://review.typo3.org/8981

#10 Updated by Gerrit Code Review over 7 years ago

Patch set 1 for branch TYPO3_4-7 has been pushed to the review server.
It is available at http://review.typo3.org/11334

#11 Updated by Gerrit Code Review over 7 years ago

Patch set 1 for branch TYPO3_4-6 has been pushed to the review server.
It is available at http://review.typo3.org/11335

#12 Updated by Gerrit Code Review over 7 years ago

Patch set 1 for branch TYPO3_4-5 has been pushed to the review server.
It is available at http://review.typo3.org/11336

#13 Updated by Steffen Gebert over 7 years ago

  • Status changed from Under Review to Resolved
  • % Done changed from 0 to 100

#14 Updated by Xavier Perseguers over 7 years ago

  • Status changed from Resolved to Under Review
  • % Done changed from 100 to 90

#15 Updated by Steffen Gebert over 7 years ago

  • Status changed from Under Review to Resolved
  • % Done changed from 90 to 100

#16 Updated by Gerrit Code Review over 7 years ago

  • Status changed from Resolved to Under Review

Patch set 1 for branch TYPO3_4-7 has been pushed to the review server.
It is available at http://review.typo3.org/13146

#17 Updated by Gerrit Code Review over 7 years ago

Patch set 1 for branch TYPO3_4-6 has been pushed to the review server.
It is available at http://review.typo3.org/13262

#18 Updated by Stefan Galinski about 7 years ago

  • Status changed from Under Review to Resolved

#19 Updated by Christian Holzmann about 4 years ago

Hello!

This ist still not solved in TYPO3 6.2.15!

Until a timeout occours, the page will be duplicated over and over...
If if extend max_execution_time to 1200 i get 3...4.. duplicated...
each of them is not completed...

Regards,
Christian Holzmann

#20 Updated by Jan Greth over 3 years ago

Christian Holzmann wrote:

Hello!

This ist still not solved in TYPO3 6.2.15!

Until a timeout occours, the page will be duplicated over and over...
If if extend max_execution_time to 1200 i get 3...4.. duplicated...
each of them is not completed...

Regards,
Christian Holzmann

Same here with 6.2.19!

Is there any workaround or manual fix to solve this?

Regards,
Jan Greth

#21 Updated by Ulrik H. Kold over 3 years ago

This is also an issue with v7.6.4 which I'm using here now.

#22 Updated by Michal Cygankiewicz over 3 years ago

Same with 6.2.25!

#23 Updated by Benni Mack about 1 year ago

  • Status changed from Resolved to Closed

Also available in: Atom PDF