Bug #90180
openDataHandler is running into a 500 Error while importing large data
0%
Description
I have built an extension to import a lot of data from another database into my TYPO3 instance.To prevent running out of memory or reaching the max execution time the import is devided into several chunks and multiple ajax requests until all chunks are imported. After about 65 chunks (96 chunks total) the import running into an internal server error with the response "Houston, we have a problem".
I'm calling the process datamap method with alot of data and don't know the problem. I'm also not able to catch this exception:
// Custom log method to store my logs into separat log files for each chunk in the typo3temp directory.
$this->log( 'Starting DataHandler' );
$this->log( json_encode( $this->data, JSON_PRETTY_PRINT | JSON_UNESCAPED_UNICODE | JSON_UNESCAPED_SLASHES ) );
$this->dataHandler->start( $this->data, [] );
try {
$this->dataHandler->process_datamap();
} catch ( \Exception $e ) {
$this->log( $e->getMessage() );
}
$this->log( 'DataHandler finished' );
The exception will be thrown every time and the last log "DataHandler finished" won't be logged.
In the files there is an example of data to import (one of the breaking chunks) formatted in json. It is a lot of data, but in some chunks before there is more data but no errors. I'm trying to fix it since 2 weeks without success.
I'm working with TYPO3 v9.5.13 and PHP 7.2
Files
No data to display