Story #92526
openConcept for improved EXT:impexp
0%
Description
Scope¶
The TYPO3 system extension EXT:impexp seems to be a bit abandoned ("stepmotherly supervised"):
The UI is not intuitive, an official documentation is missing and the support of CLI commands could be extended. The community seems to be eager to use it, but at the moment there has evolved a group of unofficially certified "TYPO3 Impexp Integrators" who know how to do it right.
As this extension is a very well encapsulated, small piece of code which should not rely too much on other extensions, this is maybe a good project for someone who wants to dive into TYPO3 Core development and Clean Code step by step.
The following diagrams and lists are the result of some brainstorming on a revised EXT:impexp. It is open for discussion and in the end this page should contain a final wish list and rough concept which is ready for implementation.
(and I would like to do / participate in the implementation for the above reasons:)
Concept¶
Entity "Export Preset"¶
- Fully-Qualified TCA
- display records in Web > List module
- make all fields editable
- fields:
- [ctrl] hideTable = false;
- [ctrl] rootLevel = -1;
- uid
- pid
- title
- identifier (new)
- description (new)
- configuration (former: preset_data)
- cruser_id (former: user_uid)
- tstamp
- crdate
- hidden (new)
- deleted (new)
- remove fields:
- public (replace by native TYPO3 access management)
- item_uid (replace by pid)
- CRUD
- Creation:
- create export preset if
- export preset by this backend user + pid + title does not exist
- export module accessible by backend user
- record creation allowed to backend user on this pid
- pid = root page of exported page subtree
- title + description = form fields in export module tab "Meta Data"
- identifier = auto-generated from backend user + pid + title (unique, used for CLI commands)
- cruser_id = backend user using export module
- configuration = serialized export configuration
- tstamp + crdate = creation timestamp
- hidden = false
- deleted = false
- create export preset if
- Read:
- show export preset if
- pid = current root page of export module
- export module accessible by backend user
- page / record reading allowed to backend user on this pid
- show export preset if
- Update:
- update export preset if
- export preset by this backend user + pid + title does exist
- export module accessible by backend user
- record editing allowed to backend user on this pid
- description = form field in export module tab "Meta Data"
- configuration = serialized export configuration
- tstamp = update timestamp
- update export preset if
- Deletion:
- delete export preset if
- export module accessible by backend user
- record deletion allowed to backend user on this pid
- deleted = true
- delete export preset if
- Creation:
- General: Sync handling of TCEforms and import/export module:
- soft / hard deletion
- record history
- access management
Entity "Import Preset"¶
- basic setting as entity "Export Preset" but used for import configurations
Documentation¶
- create full extension docs at /Documentation
- add good explanations for all 3 export configuration fields:
- "Include tables"
- "Include relations to tables"
- "Use static relations for tables" => what happens if selected / is missing / which scenario is it good for?
UI¶
- Export module
- tab "Preset"
- list all existing presets of page with
- normal TCA list item
- action: load (jump to tab "Page Tree" with preset loaded + success message)
- list all existing presets of subtree with
- grouped by page ID (page title)
- normal TCA list item
- action: load (jump to page in page tree + tab "Page Tree" with preset loaded + success message + info about page tree jump)
- grouped by page ID (page title)
- Button "Next"
- list all existing presets of page with
- tab "Page Tree"
- Page ID
- Page Tree (former: Tree)
- Depth (former: Levels)
- Include tables
- Include relations to tables (find better wording)
- Use static relations for tables (find better wording)
- Show static relations (find better wording)
- Exclude elements: Add same row as for "Preview export": Include [check] [symbol] [title/table+id if no title]
- Exclude disabled elements (is it really required or does it add complexity?)
- Button "Apply" (former: "Update")
- Button "Prev" + Button "Next"
- Preview export (single records removable)
- tab "Files"
- Exclude media which is linked in HTML / CSS files
- Button "Apply"
- Button "Prev" + Button "Next"
- Preview export (single records removable?)
- tab "Export"
- Section "Save configuration as preset"
- title
- description
- button "Save preset" => on save: "Configuration saved to path/file successfully."
- Section "Save export"
- File format
- Subsection "Save export on server"
- Readonly server path + input field for filename (if empty, suggest something)
- Button "Save export on server" => on save: "Export saved to path/file successfully."
- Subsection "Download export"
- Button "Download export"
- Button "Prev"
- Preview export
- Section "Save configuration as preset"
- tab "Preset"
- Import module
- tab "Preset"
- list all existing presets of page with
- normal TCA list item
- action: load (jump to tab "Load" with preset loaded + success message)
- list all existing presets of subtree with
- grouped by page ID (page title)
- normal TCA list item
- action: load (jump to page in page tree + tab "Page Tree" with preset loaded + success message)
- grouped by page ID (page title)
- Button "Next"
- list all existing presets of page with
- tab "Load"
- Select file to import
- Upload file from local computer
- Overwrite existing files
- Button "Load" (former: "Update")
- Button "Next"
- tab "Page Tree"
- Page ID
- Page Tree (former: Tree)
- Depth (former: Levels)
- Include tables (as in view "Export")
- Method: Insert records / Update records / Update records but ignore the pid (former: Update records)
- Do not show differences in records => Why, for performance reasons?
- Force uids of import file (former: Force ALL UIDs values)
- Log all database actions (former: Write individual DB actions during import to the log)
- Exclude elements: Add same row as for "Preview import": Include [check] [symbol] [title/table+id if no title]
- Button "Apply"
- Button "Prev" + Button "Next"
- Preview import (single records removable)
- tab "Import"
- Section "Save configuration as preset"
- title
- description
- button "Save preset" => on save: "Configuration saved to path/file successfully."
- Section "Import"
- Button "Import"
- Button "Prev"
- Preview import
- Section "Save configuration as preset"
- tab "Preset"
- General: perform database actions by DataHandler
- General: Sort lists alphabetically, if no other sorting is more appropriate.
- General: Add info overlays with detailed explanations for all fields that are not super obvious.
- General: Always return to the current tab when submitting a form.
- General: Are there UI elements available for progress / proceeding with a multistep form?
- General: Add more apply buttons to UI, maybe after each configuration section
- General: Apply button maybe with loading overlay, maybe with enabled and disabled state (depending on change in formular?)
- General: Prev and next buttons maybe contain titles of prev + next tabs
CLI¶
- Export command (https://forge.typo3.org/issues/84718)
- params
- filepath (required) (The path and filename to export to)
- filetype (required) (t3d/t3dz/xml)
- preset (optional) (The preset identifier to use for export)
- pid (optional) (The pid indicates root page of export)
- depth (optional) (Traversal depth, same as levels in export module)
- includeTable (optional, multiple) (if added, only records of these tables will be exported, same as in export module)
- includeRelated (optional, multiple) (if added, records of these tables will be exported only if linked by another record, same as in export module)
- includeStatic (optional, multiple) (if added, foreign keys will be kept in records of these tables, same as in export module)
- excludeRecord (optional, multiple) (if added, this particular record (table:uid) will be exluded from export, same as in export module)
- includeRecord (optional, multiple) (if added, this particular record (table:uid) will be included into export, same as in export module)
- dry-run (optional) (if added, the command returns all queries instead of executing them, same as "Export Preview" in export module)
- preset and other params can be used together: they get merged in a sensible and well documented way
- params
- Import command
- params
- file -> filepath (filetype gets auto-detected)
- preset (optional) (The preset identifier to use for import)
- pid (optional)
- depth (optional) (Traversal depth, same as levels in import module)
- includeTable (optional, multiple) (if added, only records of these tables will be imported, same as in import module)
- excludeRecord (optional, multiple) (if added, this particular record (table:uid) will be exluded from import, same as in import module)
- method [insert(default)/update/updateButIgnorePid] (optional) (former: updateRecords, ignorePid)
- forceUid (optional) (if added, uids of imported records will be forced)
- enableLog (optional) (if added, database queries get logged)
- dry-run (optional) (if added, the command returns all queries instead of executing them, same as "Import Preview" in import module)
- preset and other params can be used together: they get merged in a sensible and well documented way
- params
- General: Add shortcuts for each param
- General: Add summary log with configuration and statistics of import/export
- General: Make all module configuration fields available as CLI command params (such that an integrator can fully switch from TYPO3 backend to CLI for large TYPO3 instances which exceed PHP maximum execution time)
Wordings¶
- levels -> depth (see e.g. DeletedRecordsCommand)
- pageId -> pid (see e.g. DeletedRecordsCommand)
- No tree exported - only tables on the page. -> No pages exported - only records on this page.
Misc¶
- Check and maybe adjust indentation of entries in section "Preview import" / "Preview export"
- General: Use as many core functionality as possible
- General: Refactor to smaller yet better testable portions of code (e.g. small controllers)
- Check and maybe add missing TCA configurations for all existing TYPO3 entities to make them exportable, e.g. scheduler tasks
Questions¶
- Merge tabs "Page Tree" and "Files" into "Configuration" to reduce tabbing?
- File formats of preset and dump files still state of the art?
- How to best visualize TCA field "configuration"? If visualization for XML or JSON or YAML or FlexForm available, maybe switching from serialized object to that format is preferred?
- Is there any advantage of T3D over XML? If not, could we skip support?
- Is there any advantage of XML over JSON? Maybe having a scheme? If not, should we switch to the slim JSON format?
- How to best support a user-friendly copy & paste / instant preset, e.g. User A -> User B: "Please export your TYPO3 instance with this preset snippet and paste in here the export .."
- How to best replace access field "public" by official TYPO3 Core access management
- Are there already TYPO3 Core UI elements for showing progress and handling a multistep form in a linear direction?
Files
Updated by Alexander Nitsche almost 4 years ago
- Related to Story #85430: Improve usability of impexp and add documentation added
Updated by Alexander Nitsche almost 4 years ago
- Related to Feature #84718: Add CLI export command to impexp added
Updated by Alexander Nitsche almost 4 years ago
- File impexp_export_module.svg added
- File impexp_import_module.svg added
- Description updated (diff)
Updated by Alexander Nitsche almost 4 years ago
- File impexp_export_module.png impexp_export_module.png added
- File impexp_import_module.png impexp_import_module.png added
- Description updated (diff)
Updated by Alexander Nitsche almost 4 years ago
- File deleted (
impexp_export_module.svg)
Updated by Alexander Nitsche almost 4 years ago
- File deleted (
impexp_import_module.svg)
Updated by Alexander Nitsche almost 4 years ago
- Related to Epic #72584: sysext:impexp added
Updated by Alexander Nitsche almost 4 years ago
- Related to deleted (Epic #72584: sysext:impexp)
Updated by Markus Klein over 3 years ago
I take the freedom to dump some notes here I once wrote down in my personal notes:
- Must be compatible with multiple Core versions: Allow to import exports of older TYPO3 versions. (Aka put some kind of "export-format-version-number" into the data structure.)
- CLI support
- Clear logging / reporting of the import/export result
- Allow language mapping during import: Import content with L=1 in the export file to L=17 in the current instance
Updated by Simon Gilli over 3 years ago
- Related to Bug #92974: Re-import of website with form partially fails added
Updated by Matthias Druske over 3 years ago
- Tracker changed from Story to Bug
- TYPO3 Version changed from 11 to 10
- PHP Version set to 7.3
- Tags set to multilingual translation page
Updated by Matthias Druske over 3 years ago
- Related to Bug #93470: XML export data does not include page translations. added
Updated by Alexander Nitsche over 3 years ago
- Tracker changed from Bug to Story
- TYPO3 Version changed from 10 to 11
- PHP Version deleted (
7.3) - Tags changed from multilingual translation page to impexp
Updated by Alexander Nitsche almost 3 years ago
- Related to Feature #94939: Keep UIDs when installing distributions added
Updated by Alexander Nitsche almost 3 years ago
- Related to Feature #94940: Manually excluded UIDs in EXT:impexp rely naivly on volatile UIDs added
Updated by Alexander Nitsche almost 3 years ago
- Related to Feature #94941: Exports via export tools sometimes contain previous exports added
Updated by Alexander Nitsche almost 3 years ago
- Related to Bug #94972: PHP 8 warnings: Undefined array key in EXT:impexp added
Updated by Alexander Nitsche almost 3 years ago
- Related to Bug #83580: GeneralUtility::xml2array() can't parse bigger files (> 10MB) added
Updated by Alexander Nitsche almost 3 years ago
- Related to Task #95135: Create an extension manual for EXT:impexp added
Updated by Alexander Nitsche almost 3 years ago
- Related to Bug #95007: Checking the file checksum on export or import may fail due to sanitizers added
Updated by Alexander Nitsche almost 3 years ago
- Related to Task #95471: Add more details to the manual of EXT:impexp added
Updated by Alexander Nitsche over 2 years ago
- Related to Task #94772: Cleanup impexp extension added