Story #92526

Concept for improved EXT:impexp

Added by Alexander Nitsche about 1 year ago. Updated 10 months ago.

Status:
Accepted
Priority:
Should have
Assignee:
-
Category:
Import/Export (T3D)
Start date:
2020-10-09
Due date:
% Done:

0%

Estimated time:
TYPO3 Version:
11
PHP Version:
Tags:
impexp
Sprint Focus:

Description

Scope

The TYPO3 system extension EXT:impexp seems to be a bit abandoned ("stepmotherly supervised"):
The UI is not intuitive, an official documentation is missing and the support of CLI commands could be extended. The community seems to be eager to use it, but at the moment there has evolved a group of unofficially certified "TYPO3 Impexp Integrators" who know how to do it right.

As this extension is a very well encapsulated, small piece of code which should not rely too much on other extensions, this is maybe a good project for someone who wants to dive into TYPO3 Core development and Clean Code step by step.

The following diagrams and lists are the result of some brainstorming on a revised EXT:impexp. It is open for discussion and in the end this page should contain a final wish list and rough concept which is ready for implementation.

(and I would like to do / participate in the implementation for the above reasons:)

Concept

Entity "Export Preset"

  • Fully-Qualified TCA
    • display records in Web > List module
    • make all fields editable
    • fields:
      • [ctrl] hideTable = false;
      • [ctrl] rootLevel = -1;
      • uid
      • pid
      • title
      • identifier (new)
      • description (new)
      • configuration (former: preset_data)
      • cruser_id (former: user_uid)
      • tstamp
      • crdate
      • hidden (new)
      • deleted (new)
    • remove fields:
      • public (replace by native TYPO3 access management)
      • item_uid (replace by pid)
  • CRUD
    • Creation:
      • create export preset if
        • export preset by this backend user + pid + title does not exist
        • export module accessible by backend user
        • record creation allowed to backend user on this pid
      • pid = root page of exported page subtree
      • title + description = form fields in export module tab "Meta Data"
      • identifier = auto-generated from backend user + pid + title (unique, used for CLI commands)
      • cruser_id = backend user using export module
      • configuration = serialized export configuration
      • tstamp + crdate = creation timestamp
      • hidden = false
      • deleted = false
    • Read:
      • show export preset if
        • pid = current root page of export module
        • export module accessible by backend user
        • page / record reading allowed to backend user on this pid
    • Update:
      • update export preset if
        • export preset by this backend user + pid + title does exist
        • export module accessible by backend user
        • record editing allowed to backend user on this pid
      • description = form field in export module tab "Meta Data"
      • configuration = serialized export configuration
      • tstamp = update timestamp
    • Deletion:
      • delete export preset if
        • export module accessible by backend user
        • record deletion allowed to backend user on this pid
      • deleted = true
  • General: Sync handling of TCEforms and import/export module:
    • soft / hard deletion
    • record history
    • access management

Entity "Import Preset"

  • basic setting as entity "Export Preset" but used for import configurations

Documentation

  • create full extension docs at /Documentation
  • add good explanations for all 3 export configuration fields:
    • "Include tables"
    • "Include relations to tables"
    • "Use static relations for tables" => what happens if selected / is missing / which scenario is it good for?

UI

  • Export module
    • tab "Preset"
      • list all existing presets of page with
        • normal TCA list item
        • action: load (jump to tab "Page Tree" with preset loaded + success message)
      • list all existing presets of subtree with
        • grouped by page ID (page title)
          • normal TCA list item
          • action: load (jump to page in page tree + tab "Page Tree" with preset loaded + success message + info about page tree jump)
      • Button "Next"
    • tab "Page Tree"
      • Page ID
      • Page Tree (former: Tree)
      • Depth (former: Levels)
      • Include tables
      • Include relations to tables (find better wording)
      • Use static relations for tables (find better wording)
      • Show static relations (find better wording)
      • Exclude elements: Add same row as for "Preview export": Include [check] [symbol] [title/table+id if no title]
      • Exclude disabled elements (is it really required or does it add complexity?)
      • Button "Apply" (former: "Update")
      • Button "Prev" + Button "Next"
      • Preview export (single records removable)
    • tab "Files"
      • Exclude media which is linked in HTML / CSS files
      • Button "Apply"
      • Button "Prev" + Button "Next"
      • Preview export (single records removable?)
    • tab "Export"
      • Section "Save configuration as preset"
        • title
        • description
        • button "Save preset" => on save: "Configuration saved to path/file successfully."
      • Section "Save export"
        • File format
        • Subsection "Save export on server"
          • Readonly server path + input field for filename (if empty, suggest something)
          • Button "Save export on server" => on save: "Export saved to path/file successfully."
        • Subsection "Download export"
          • Button "Download export"
      • Button "Prev"
      • Preview export

  • Import module
    • tab "Preset"
      • list all existing presets of page with
        • normal TCA list item
        • action: load (jump to tab "Load" with preset loaded + success message)
      • list all existing presets of subtree with
        • grouped by page ID (page title)
          • normal TCA list item
          • action: load (jump to page in page tree + tab "Page Tree" with preset loaded + success message)
      • Button "Next"
    • tab "Load"
      • Select file to import
      • Upload file from local computer
        • Overwrite existing files
      • Button "Load" (former: "Update")
      • Button "Next"
    • tab "Page Tree"
      • Page ID
      • Page Tree (former: Tree)
      • Depth (former: Levels)
      • Include tables (as in view "Export")
      • Method: Insert records / Update records / Update records but ignore the pid (former: Update records)
      • Do not show differences in records => Why, for performance reasons?
      • Force uids of import file (former: Force ALL UIDs values)
      • Log all database actions (former: Write individual DB actions during import to the log)
      • Exclude elements: Add same row as for "Preview import": Include [check] [symbol] [title/table+id if no title]
      • Button "Apply"
      • Button "Prev" + Button "Next"
      • Preview import (single records removable)
    • tab "Import"
      • Section "Save configuration as preset"
        • title
        • description
        • button "Save preset" => on save: "Configuration saved to path/file successfully."
      • Section "Import"
        • Button "Import"
      • Button "Prev"
      • Preview import

  • General: perform database actions by DataHandler
  • General: Sort lists alphabetically, if no other sorting is more appropriate.
  • General: Add info overlays with detailed explanations for all fields that are not super obvious.
  • General: Always return to the current tab when submitting a form.
  • General: Are there UI elements available for progress / proceeding with a multistep form?
  • General: Add more apply buttons to UI, maybe after each configuration section
  • General: Apply button maybe with loading overlay, maybe with enabled and disabled state (depending on change in formular?)
  • General: Prev and next buttons maybe contain titles of prev + next tabs

CLI

  • Export command (https://forge.typo3.org/issues/84718)
    • params
      • filepath (required) (The path and filename to export to)
      • filetype (required) (t3d/t3dz/xml)
      • preset (optional) (The preset identifier to use for export)
      • pid (optional) (The pid indicates root page of export)
      • depth (optional) (Traversal depth, same as levels in export module)
      • includeTable (optional, multiple) (if added, only records of these tables will be exported, same as in export module)
      • includeRelated (optional, multiple) (if added, records of these tables will be exported only if linked by another record, same as in export module)
      • includeStatic (optional, multiple) (if added, foreign keys will be kept in records of these tables, same as in export module)
      • excludeRecord (optional, multiple) (if added, this particular record (table:uid) will be exluded from export, same as in export module)
      • includeRecord (optional, multiple) (if added, this particular record (table:uid) will be included into export, same as in export module)
      • dry-run (optional) (if added, the command returns all queries instead of executing them, same as "Export Preview" in export module)
    • preset and other params can be used together: they get merged in a sensible and well documented way
  • Import command
    • params
      • file -> filepath (filetype gets auto-detected)
      • preset (optional) (The preset identifier to use for import)
      • pid (optional)
      • depth (optional) (Traversal depth, same as levels in import module)
      • includeTable (optional, multiple) (if added, only records of these tables will be imported, same as in import module)
      • excludeRecord (optional, multiple) (if added, this particular record (table:uid) will be exluded from import, same as in import module)
      • method [insert(default)/update/updateButIgnorePid] (optional) (former: updateRecords, ignorePid)
      • forceUid (optional) (if added, uids of imported records will be forced)
      • enableLog (optional) (if added, database queries get logged)
      • dry-run (optional) (if added, the command returns all queries instead of executing them, same as "Import Preview" in import module)
    • preset and other params can be used together: they get merged in a sensible and well documented way
  • General: Add shortcuts for each param
  • General: Add summary log with configuration and statistics of import/export
  • General: Make all module configuration fields available as CLI command params (such that an integrator can fully switch from TYPO3 backend to CLI for large TYPO3 instances which exceed PHP maximum execution time)

Wordings

  • levels -> depth (see e.g. DeletedRecordsCommand)
  • pageId -> pid (see e.g. DeletedRecordsCommand)
  • No tree exported - only tables on the page. -> No pages exported - only records on this page.

Misc

  • Check and maybe adjust indentation of entries in section "Preview import" / "Preview export"
  • General: Use as many core functionality as possible
  • General: Refactor to smaller yet better testable portions of code (e.g. small controllers)
  • Check and maybe add missing TCA configurations for all existing TYPO3 entities to make them exportable, e.g. scheduler tasks

Questions

  • Merge tabs "Page Tree" and "Files" into "Configuration" to reduce tabbing?
  • File formats of preset and dump files still state of the art?
    • How to best visualize TCA field "configuration"? If visualization for XML or JSON or YAML or FlexForm available, maybe switching from serialized object to that format is preferred?
    • Is there any advantage of T3D over XML? If not, could we skip support?
    • Is there any advantage of XML over JSON? Maybe having a scheme? If not, should we switch to the slim JSON format?
  • How to best support a user-friendly copy & paste / instant preset, e.g. User A -> User B: "Please export your TYPO3 instance with this preset snippet and paste in here the export .."
  • How to best replace access field "public" by official TYPO3 Core access management
  • Are there already TYPO3 Core UI elements for showing progress and handling a multistep form in a linear direction?

Files

impexp_export_module.png (43.8 KB) impexp_export_module.png Alexander Nitsche, 2020-10-09 09:49
impexp_import_module.png (42 KB) impexp_import_module.png Alexander Nitsche, 2020-10-09 09:49

Related issues

Related to TYPO3 Core - Story #85430: Improve usability of impexp and add documentationClosed2018-06-29

Actions
Related to TYPO3 Core - Feature #84718: Add CLI export command to impexpClosed2018-04-13

Actions
Related to TYPO3 Core - Bug #92974: Re-import of website with form partially failsAccepted2020-12-02

Actions
Related to TYPO3 Core - Bug #93470: XML export data does not include page translations.Under Review2021-02-09

Actions
Related to TYPO3 Core - Feature #94939: Keep UIDs when installing distributionsNew2021-08-20

Actions
Related to TYPO3 Core - Feature #94940: Manually excluded UIDs in EXT:impexp rely naivly on volatile UIDsNew2021-08-20

Actions
Related to TYPO3 Core - Feature #94941: Exports via export tools sometimes contain previous exportsNew2021-08-20

Actions
Related to TYPO3 Core - Bug #94972: PHP 8 warnings: Undefined array key in EXT:impexpClosed2021-08-23

Actions
Related to TYPO3 Core - Bug #83580: GeneralUtility::xml2array() can't parse bigger files (> 10MB)Under Review2018-01-16

Actions
Related to TYPO3 Core - Task #95135: Create an extension manual for EXT:impexpClosedLina Wolf2021-09-07

Actions
Related to TYPO3 Core - Bug #95007: Checking the file checksum on export or import may fail due to sanitizersNew2021-08-26

Actions
Related to TYPO3 Core - Task #95471: Add more details to the manual of EXT:impexpResolved2021-10-05

Actions
#1

Updated by Alexander Nitsche about 1 year ago

  • Related to Story #85430: Improve usability of impexp and add documentation added
#2

Updated by Alexander Nitsche about 1 year ago

#3

Updated by Alexander Nitsche about 1 year ago

  • Description updated (diff)
#4

Updated by Alexander Nitsche about 1 year ago

  • File impexp_export_module.svg added
  • File impexp_import_module.svg added
  • Description updated (diff)
#5

Updated by Alexander Nitsche about 1 year ago

  • Description updated (diff)
#7

Updated by Alexander Nitsche about 1 year ago

  • File deleted (impexp_export_module.svg)
#8

Updated by Alexander Nitsche about 1 year ago

  • File deleted (impexp_import_module.svg)
#9

Updated by Alexander Nitsche about 1 year ago

#10

Updated by Alexander Nitsche about 1 year ago

#11

Updated by Georg Ringer about 1 year ago

  • Status changed from New to Accepted
#12

Updated by Alexander Nitsche about 1 year ago

  • Description updated (diff)
#13

Updated by Markus Klein about 1 year ago

I take the freedom to dump some notes here I once wrote down in my personal notes:

  • Must be compatible with multiple Core versions: Allow to import exports of older TYPO3 versions. (Aka put some kind of "export-format-version-number" into the data structure.)
  • CLI support
  • Clear logging / reporting of the import/export result
  • Allow language mapping during import: Import content with L=1 in the export file to L=17 in the current instance
#14

Updated by Simon Gilli 12 months ago

  • Related to Bug #92974: Re-import of website with form partially fails added
#15

Updated by Matthias Druske 10 months ago

  • Tracker changed from Story to Bug
  • TYPO3 Version changed from 11 to 10
  • PHP Version set to 7.3
  • Tags set to multilingual translation page
#16

Updated by Matthias Druske 10 months ago

  • Related to Bug #93470: XML export data does not include page translations. added
#17

Updated by Alexander Nitsche 10 months ago

  • Tracker changed from Bug to Story
  • TYPO3 Version changed from 10 to 11
  • PHP Version deleted (7.3)
  • Tags changed from multilingual translation page to impexp
#18

Updated by Alexander Nitsche 3 months ago

  • Related to Feature #94939: Keep UIDs when installing distributions added
#19

Updated by Alexander Nitsche 3 months ago

  • Related to Feature #94940: Manually excluded UIDs in EXT:impexp rely naivly on volatile UIDs added
#20

Updated by Alexander Nitsche 3 months ago

  • Related to Feature #94941: Exports via export tools sometimes contain previous exports added
#21

Updated by Alexander Nitsche 3 months ago

  • Related to Bug #94972: PHP 8 warnings: Undefined array key in EXT:impexp added
#22

Updated by Alexander Nitsche 3 months ago

  • Related to Bug #83580: GeneralUtility::xml2array() can't parse bigger files (> 10MB) added
#23

Updated by Alexander Nitsche 3 months ago

  • Related to Task #95135: Create an extension manual for EXT:impexp added
#24

Updated by Alexander Nitsche 2 months ago

  • Related to Bug #95007: Checking the file checksum on export or import may fail due to sanitizers added
#25

Updated by Alexander Nitsche about 2 months ago

  • Related to Task #95471: Add more details to the manual of EXT:impexp added

Also available in: Atom PDF