External Import

Extension "external_import", along with its tutorial ("externalimport_tut") and all stable connector services ("svconnector", "svconnector_csv", "svconnector_feed" and "svconnector_sql") have been released today after their compatibility with TYPO3 CMS 6.0 was verified. The backend modules of "external_import" and "svconnector" had to be updated, but they are still compatible with all TYPO3 versions from 4.5 to 6.0. There is a problem with TYPO3 6.0.0 though, see below for details.

The tutorial was modified to use extension "news" for the RSS feed import example, rather than "tt_news" which was used until now.

New feature¶

A single new feature was added to "external_import": the possibility to directly take the value or an attribute of the reference nodes, rather than being limited to sub-nodes. Consider the following XML structure:

<list>
    <item name="foo">
        bar
    </item>
</list>

If "item" has been defined as "nodetype", you were until now limited to selecting sub-nodes of each "item" node. Now you can leave the "field" property empty for any column configuration and retrieve the value of the "item" node directly (i.e. "bar" in the example above). Furthermore, if you define the "attribute" property, you can get the attribute of the reference node. In the example above, setting "attribute" to "name" would return "foo" for the first "item" node.

Issues with TYPO3 6.0.0¶

The Extension Manager of TYPO3 6.0.0 is plagued by a bug which makes it impossible to save an extension's configuration when it uses sub-categories, which is the case of "external_import". A patch is pending (https://review.typo3.org/#/c/17621/). Hopefully this will be fixed in TYPO3 6.0.1 and "external_import" will then run flawlessly on the 6.x branch.

It's my great pleasure to announce today the release of External Import's new major version. There was a lot of work involved, often complicated by lack of documentation for the tools used (I'm thinking about ExtJS here) and my available time was limited. The work was spread over many months and I'm very happy to have finally completed it.

So what happened:

• all the legacy code was dropped. The BE module contained JS for AJAX calls compatible down to TYPO3 4.1, the automation process still supported Gabriel. All that is gone and the code could be cleaned up quite a bit.
• the BE module was completely rewritten. It's now based on Extbase and mostly relies on ExtJS and ExtDirect for all the logic. It is much more user-friendly in my opinion, and I hope that you will agree ;-) It is also adapted to the current TYPO3 skin.
• compatibility is ensured up to TYPO3 4.7 (but minimum version required is 4.5).
• on top of that are a number of new features:
  • importing date and time information is easier, with the system taking time zones into account automatically
  • imported data can now reference itself, including when importing into the "pages" table
  • cache clearing can be called after a import run
  • handling of XML data supports namespaces

The ground is ready for more features, but I'll take a rest first ;-)

The upcoming version 2.0 of external_import is now feature complete. All existing features have been reimplemented in Extbase and ExtJS for the BE module legacy code was dropped. Some new features were added, some were finally postponed to version 2.1, so that 2.0 could be released without further delay.

All that is now left to do is to brush up the manual, in particular update the screenshots. Version 2.0 should be out soon.

The last release of External Import dates back to nearly 1 year ago and the BE module is looking bad in more recent versions of TYPO3. So what's cooking?

I have started a long while ago revamping the BE module and dropping all legacy code, but I then had to care for much other stuff, TYPO3 Core, documentation, and also taking a bit of time to relax ;-) Last week I started work again on what will eventually be version 2.0. I braced myself and tackled the ExtJS challenge. ExtJS is not easy to learn and the documentation is not very helpful. Still I managed to transform the BE module listing into a ExtJS grid and all AJAX calls were replaced with ExtDirect calls.

• the synchronization forms are not yet displayed
• the listing of non-synchronizable tables has disappeared for now (the listing is the one of synchronizable tables)

Barring these two little problems, the BE module is usable and the rest of the code is functional. I would be glad if some adventurers would check it out and send feedback.

Thanks to Andrey Aksenov, the manuals of "External Import" and "External Import Tutorial" are now available in Russian. You can get them from these extensions' respective file repositories, in both PDF and OpenOffice format:

• http://forge.typo3.org/projects/extension-external_import/files
• http://forge.typo3.org/projects/extension-externalimport_tut/files

Today a new version of external_import was released. It contains several bug fixes and a host of new features. Here are some highlights:

• rich-text fields can be imported properly, i.e. triggering the DB transformation one expects from a RTE field
• the import process can be interrupted by exceptions thrown by the connector service in case of problems with the data source
• enhanced retrieval of information from XML structures: values can be selected from attributes or from nodes selected with XPath queries

On top of this new connector services were released making it possible to import data from many new sources:

• svconnector_sql: relies on ADODB to connect to any database system
• svconnector_feed: can read any local or remote XML file, for example RSS feeds

The external_import tutorial (externalimport_tut) was made even more useful with the addition of a scenario demonstrating the import of an RSS feed into the tt_news table.

This is probably the last version of the 1.x branch (barring critical bug fixes). Work has already started on what will become the 2.x branch. All legacy code will be dropped, the BE module will get a lifting to match the new TYPO3 skin and all JavaScript code will be moved to ExtJS.

During T3DD10 I had the opportunity and the pleasure to record a podcast about External Import with Kasper. If you have never tried this extension yet, this podcast now offers you a nice 10-minute glimpse into it:

http://castor.t3o.punkt.de/files/external_import_720p.mp4

Thanks Kasper!

In the months since the last update, External Import has gone through intensive use in many real-life scenarios. Although this turned up a few bugs - which were all corrected - it mostly proved that the code base was healthy and that the feature set was meeting the vast majority of the demand.

This means that External Import was on the right track to have its status raised to "stable". There were still some improvements that I thought were necessary for that piece of software to reach the quality I expect from something labeled "stable". One important new feature on that road is the ability to view all the "external" information in the BE module. Now for any synchronisation configuration, you can click on the information icon and get a view of all TCA settings related to External Import.

Another point to improve was the handling of user rights. Since External Import relies on TCEmain to save the data, there were no risks of misuse. But the user interface could be improved. This has been achieved this way:

• in the BE module, users can only see tables to which they have read or modify rights
• also in the BE module, users cannot synchronize a table on which they don't have modify rights. They can view its configuration, but not start a sync, nor define an automated sync
• in the Gabriel/Scheduler task, a table is entirely skipped if the CLI user doesn't have enough rights. This avoids having TCEmain throw an error for each record.

These are the most important changes. For the rest, please refer to the ChangeLog and the manual.

Scheduler support¶

This newly released version (0.11.1) provides full support for the Scheduler coming with TYPO3 4.3. This comes on top of Gabriel support. Both tools are supported using some wrapper classes.

Reporting by e-mail¶

It is now possible to receive detailed synchronisation reports by e-mail to be informed of what happened during automatic synchronisations.

Aborting the import¶

A new step has been added in the import process for data validation. The data received from the external source can be checked using hooks and the import process can be stopped by any hook method by simply returning a false value. A built-in test is also available: it will check whether the data has a minimum number of records (defined using the new "minimumRecords" property) and abort if not. This adds extra security for the existing data, in case the external data returned an empty or incomplete dataset, but without explicit failure.

Refined automatic synchronisation¶

It is now possible to define an automatic synchronisation per import configuration and not just for all tables. The interface for this has been cleaned up and support for cron-like syntax is now available (from the BE module and not just from Gabriel/Scheduler).

Roadmap¶

All issues on the road to version 1.0 are now listed in the bug tracker leading to a clear roadmap towards a first complete and stable version of this extension.

I just released a new version of this extension. It represents a major update. Many bugs were fixed, the API was improved and the processes were streamlined.

All these changes came at a price though: there are some compatibility breaks. Since the happen in the extended TCA syntax, it was not possible to fix them using an update script. However all necessary changes are described in the manual, which you are encourage to read online before upgrading.

Also now available is a tutorial to External Import (extension key: externalimport_tut). Configuring the TCA properly may be complicated in some cases, and the tutorial provides both examples and detailed instructions which should help you get started smoothly.