Project

General

Profile

Actions

Bug #90180

open

DataHandler is running into a 500 Error while importing large data

Added by Luca Braun about 4 years ago.

Status:
New
Priority:
Should have
Assignee:
-
Category:
DataHandler aka TCEmain
Target version:
-
Start date:
2020-01-23
Due date:
% Done:

0%

Estimated time:
TYPO3 Version:
9
PHP Version:
7.2
Tags:
Complexity:
Is Regression:
Sprint Focus:

Description

I have built an extension to import a lot of data from another database into my TYPO3 instance.To prevent running out of memory or reaching the max execution time the import is devided into several chunks and multiple ajax requests until all chunks are imported. After about 65 chunks (96 chunks total) the import running into an internal server error with the response "Houston, we have a problem".
I'm calling the process datamap method with alot of data and don't know the problem. I'm also not able to catch this exception:

// Custom log method to store my logs into separat log files for each chunk in the typo3temp directory.
$this->log( 'Starting DataHandler' );
$this->log( json_encode( $this->data, JSON_PRETTY_PRINT | JSON_UNESCAPED_UNICODE | JSON_UNESCAPED_SLASHES ) );
$this->dataHandler->start( $this->data, [] );

try {
    $this->dataHandler->process_datamap();
} catch ( \Exception $e ) {
    $this->log( $e->getMessage() );
}

$this->log( 'DataHandler finished' );

The exception will be thrown every time and the last log "DataHandler finished" won't be logged.

In the files there is an example of data to import (one of the breaking chunks) formatted in json. It is a lot of data, but in some chunks before there is more data but no errors. I'm trying to fix it since 2 weeks without success.

I'm working with TYPO3 v9.5.13 and PHP 7.2


Files

temp.json (96 KB) temp.json Data for the DataHandler as json Luca Braun, 2020-01-23 09:37

No data to display

Actions

Also available in: Atom PDF