Task #92346
closedEnable ext:impexp to include export presets in export
Added by Alexander Nitsche almost 4 years ago. Updated over 3 years ago.
100%
Description
This task should improve the comfort of exporting and importing TYPO3 export presets along with the main export.
Status Quo¶
In order to export TYPO3 database and files and the export presets one currently has to
A) TYPO3 Source Instance¶
1) configure export (EXT:impexp)
2) save the export configuration as preset (EXT:impexp)
3) export (EXT:impexp)
4) export the presets separately (phpmyadmin et al)
and
B) TYPO3 Target Instance¶
5) import database and files in the new TYPO3 instance (EXT:impexp)
6) import the presets separately (phpmyadmin et al)
Quo Vadis¶
This should be reduced to
A) TYPO3 Source Instance¶
1) configure export including database table "tx_impexp_presets" (EXT:impexp)
2) save the export configuration as preset (EXT:impexp)
3) export (EXT:impexp)
and
B) TYPO3 Target Instance¶
4) import database and files including presets in the new TYPO3 instance (EXT:impexp)
Implementation¶
The export configuration presets are missing a TCA and thus are not listed as exportable table in EXT:impexp. Adding the TCA will provide it there. Side effects are that the export presets become visible and maintainable by admins in the Web->List module (pid=0). The presets currently contain these fields:
- title (preset title)
- public (preset usable by non-priviledged users?)
- user_uid (preset creator)
- item_uid (export page root)
- preset_data (full export configuration)
I would set title and public as editable and the other fields as read only.
Question¶
Are there any other (negative) side effects from adding a TCA for tx_impexp_presets?
Files
impexp_preset_export_1_include_tables.png (72.3 KB) impexp_preset_export_1_include_tables.png | Alexander Nitsche, 2020-09-19 16:16 | ||
impexp_preset_export_2_save_preset.png (50.1 KB) impexp_preset_export_2_save_preset.png | Alexander Nitsche, 2020-09-19 16:16 | ||
impexp_preset_export_3_include_presets.png (69.9 KB) impexp_preset_export_3_include_presets.png | Alexander Nitsche, 2020-09-19 16:16 | ||
impexp_preset_export_4_list_preset_record.png (68.4 KB) impexp_preset_export_4_list_preset_record.png | Alexander Nitsche, 2020-09-19 16:16 | ||
impexp_preset_export_5_show_preset_record.png (78.9 KB) impexp_preset_export_5_show_preset_record.png | Alexander Nitsche, 2020-09-19 16:16 | ||
copyPreset.php (1.21 KB) copyPreset.php | example script for duplicating an impexp preset | Rémy DANIEL, 2020-10-02 23:04 |
Updated by Alexander Nitsche almost 4 years ago
- Related to Story #85430: Improve usability of impexp and add documentation added
Updated by Gerrit Code Review almost 4 years ago
- Status changed from New to Under Review
Patch set 1 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/65770
Updated by Alexander Nitsche almost 4 years ago
- File impexp_preset_export_1_include_tables.png impexp_preset_export_1_include_tables.png added
- File impexp_preset_export_2_save_preset.png impexp_preset_export_2_save_preset.png added
- File impexp_preset_export_3_include_presets.png impexp_preset_export_3_include_presets.png added
- File impexp_preset_export_4_list_preset_record.png impexp_preset_export_4_list_preset_record.png added
- File impexp_preset_export_5_show_preset_record.png impexp_preset_export_5_show_preset_record.png added
Updated by Gerrit Code Review almost 4 years ago
Patch set 2 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/65770
Updated by Gerrit Code Review almost 4 years ago
Patch set 3 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/65770
Updated by Gerrit Code Review almost 4 years ago
Patch set 4 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/65770
Updated by Alexander Nitsche almost 4 years ago
I suggest to make the creator editable in the TCA configuration as the creator is also the owner of the preset. As long as access control and creator of the preset are the same for presets, the administrator should always be allowed to change the owner of this preset from one - who might not exist anymore - to another.
Maybe all fields should be editable even if the configuration field is not nice formatted but could give the opportunity to change it slightly by an admin who knows the expected data structure and it would also give the opportunity to create a new export configuration from an existing one but with slightly changed parameters (different owner, one table less, different page root, etc.). I would add a note to the context sensitive help of item_uid and preset_data that the admin should know what he is doing and should switch to the export module in case of having doubts.
Finally, i wonder if i should add the option adminOnly=1 or rootLevel=1 to the TCA configuration to make sure that only administrators can change it and hide it from non priviledged users who could mess around with it but should maybe stick to the export module only?
Updated by Gerrit Code Review almost 4 years ago
Patch set 5 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/65770
Updated by Alexander Nitsche almost 4 years ago
Georg Ringer
is there a reason why swiching soft/hard delete? tca allows both, so this is not the rason
Alexander Nitsche
mh, i was not aware of this; as one creates and deletes probably only a few presets in one instance, i would tend to prefer the double-layer-safety of a soft deletion over the clean-sweeping-benefits of hard deletion, but if there is a reason for hard deletion i would switch it, of course
Georg Ringer
if no delete field is defined in ctrl, there is no soft delete. just to be consitent I would not change that. it is fine for master to change that only but for backports I would like to avoid this
Alexander Nitsche
ok, so i keep soft deletion in the current review process, right? And what would you say about making all fields editable and allow access only to admin?
Georg Ringer
Then this is a breaking change and master only, tbh wouldn't like that admin only
Alexander Nitsche
ok, so in your opinion: keep hard deletion + access for everybody. and what about making all fields editable?
Georg Ringer
imo ok
Alexander Nitsche
ok
Updated by Gerrit Code Review almost 4 years ago
Patch set 6 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/65770
Updated by Gerrit Code Review almost 4 years ago
Patch set 7 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/65770
Updated by Gerrit Code Review almost 4 years ago
Patch set 8 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/65770
Updated by Alexander Nitsche almost 4 years ago
I will change the name "creator" to "owner" of the export preset as this name is used already in some error messages.
Updated by Gerrit Code Review almost 4 years ago
Patch set 9 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/65770
Updated by Alexander Nitsche almost 4 years ago
Christian Kuhn
In case you decide to push a patch for this, it would be awesome if the docs on the introduction package and t3 core-api would have an update for the simplified process, too:
(1) https://github.com/FriendsOfTYPO3/introduction/blob/master/Documentation/Maintenance.rst
(2) https://docs.typo3.org/m/typo3/reference-coreapi/master/en-us/ExtensionArchitecture/CreateNewDistribution/Index.html
Updated by Alexander Nitsche almost 4 years ago
Updated by Alexander Nitsche almost 4 years ago
Since this is now a TCA table, we should/would consider:
- Permission Handling (create/update possibly done via DataHandler?)
- RootLevel Restrictions in TCA (what "pid" should be used)
- Reference Index (table should probably excluded IMHO)
Updated by Alexander Nitsche almost 4 years ago
Sybille Peters
Shouldn't this be a feature and include a changelog?
(We had that before in another patch which was also declared as TASK and where it was pointed out and changed to BUGFIX. And it seems to be the way to go now to avoid the "this is a feature but we have a feature freeze in 10.4" dilemma.) I find this is messy.
Also, do you really want to document this under "Distribution" in "TYPO3 Explained"? https://github.com/TYPO3-Documentation/TYPO3CMS-Reference-CoreApi/issues/999 I think the impexp should rather have it's own documentation inside the system extension Documentation folder.
Updated by Alexander Nitsche almost 4 years ago
Hi Benni, hi Sybille,
thanks for your feedback. I would like to handle this task "as small as possible" and "as large as required". I know there are more tasks out there regarding the impexp extension very well listed in Sybille's story https://forge.typo3.org/issues/85430 , but i would like to handle them in different issues, such that
Benni: Since this is now a TCA table, we should/would consider:
- Permission Handling (create/update possibly done via DataHandler?)
I would like to move that into a new issue and combine it with the fixing of the current preset update which is broken: you can currently not update a preset in the export module. Furthermore this would introduce the requirement to have a different pid than pid=0 for the presets otherwise the non-admin users would not be able to create or update or delete a preset anymore in the export module (what they currently can).
- RootLevel Restrictions in TCA (what "pid" should be used)
I have not enough knowledge here.
- Reference Index
I have not enough knowledge here either.
Sybille:
- Shouldn't this be a feature and include a changelog?
I would like to have it in 10.4 for current distribution development, but i understand that it could be seen as a feature too. I still lack an idea of what is a task and what is a feature.
- Also, do you really want to document this under "Distribution" in "TYPO3 Explained"? https://github.com/TYPO3-Documentation/TYPO3CMS-Reference-CoreApi/issues/999 I think the impexp should rather have it's own documentation inside the system extension Documentation folder.
In the end i would like to have a real chapter about EXT:impexp and only refer to it from the "Distribution" chapter, but as long as there is no such chapter, i add that info to the distribution chapter directly and would move it later to the EXT:impexp chapter. Maybe we could add the documentation of EXT:impexp to the todos of https://forge.typo3.org/issues/85430.
In the end i want to integrate all your proposals but i am not sure what this issue should include and what should be moved to another. Furthemore i am not sure, if this should be a larger feature being integrated only into master or a small task which should be applied also to 10.4.
Greetings
Alex
Updated by Alexander Nitsche almost 4 years ago
Rémy DANIEL
I am agree with Benni: since the table is now with a TCA, the DataHandler should be used by the PresetRepository for inserting/updating the impexp preset.
Actually if I edit the preset within impexp, the records history is not maintained, the permission are not checked, etc. But they are if I edit the record with the list module.
Updated by Rémy DANIEL almost 4 years ago
- File copyPreset.php copyPreset.php added
Hi Alexander
I read again the initial demand: "include export presets in export". Here are my thoughts, given my knowledge of ext:impexp.
Personally I never had the need to export the preset of an export, but you obviously do. I just can't imagine what is your usecase.
When I export a project (a whole site or a subtree), I import it on the target and I do not need the "export" preset on the target:
I will not export the imported project from the target to another target.
One usecase I had, is that I needed to duplicate a preset on the same TYPO3 instance, for exporting the same set of data from multiple sites.
Duplicating a preset is not possible and I had to code a little script to do that. The script duplicates the DB row, loads it, unserializes the data, replaces some PIDs and saves the row back in the DB (I attached it to this issue).
Also, a preset is tied to a page (item_uid) and contains serialized data, which is mainly the list of tables to export, relationships, exclusions, starting pid, etc.
This serialized data holds record's ID, and those record's ID are "hardcoded": there will be no "softref" parsing on it, the remapping of IDs will not be done when importing the project on the target.
Thus, even if the proposed feature allows to include the preset in the export, the preset will actually be useless when imported on the target: the serialized data has been imported without any parsing or remapping of its internal record's ID, and maybe item_uid now points to something completely different, or exclusions are now excluding the wrong record, etc.
In order to be fully functional, the feature requires some "magic", based on what the softref parser does when exporting/importing.
Kind regards
Updated by Alexander Nitsche almost 4 years ago
Hi Rémy
thanks for your insights.
The use case would be to facilitate distribution development. Currently the setup of a distribution for further development requires two steps:
- install the distribution which automagically sets up the page tree
- execute a sql query to import the according page tree export configuration of the distribution (see e.g. https://github.com/FriendsOfTYPO3/introduction/blob/master/Resources/Private/ImportExportPreset.sql)
The second step is not only one step more but requires also a different tool, a database client. I think that this second step should not be necessary and all that should be required is the import/export tool and one step to continue with the distribution development and any further page tree export.
If the exported tx_impexp_preset table does not hold reliable export configurations due to hard links, then this should be no regression as the current sql statement provides no soft links either. So this could be another issue to add some magic to the export configuration export - having the export configuration export in the impexp module could be a first step into that direction as it gives all control to the impexp module.
On the other side, is there any reason to keep the export configuration preset as separate sql import?
Greetings
Alex
Updated by Alexander Nitsche almost 4 years ago
To make this task as small as possible and as backwards compatible as possible i suggest to reduce this change to
- adding a TCA with all fields being of type "passthrough"
- hiding the new TCA table via "hideTable=true"
such that the only way to manage presets remains the export module and we stick to the current procedure of creating / updating / deleting presets as a switch to the DataHandler would introduce further work on access management side. I will provide a next patch which reflects this rollback and hopefully makes the change small enough and backwards compatible to be introduced in 10.4 as well.
Thanks Rémy for your thoughts on it.
Updated by Gerrit Code Review almost 4 years ago
Patch set 10 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/65770
Updated by Alexander Nitsche almost 4 years ago
Rémy DANIEL
Just to be sure: at the end you want to include the export preset in the introduction package's data.xml and get rid of the extra sql insert statement, right?
Alexander Nitsche
yes, and not only in the introduction package but in all distribution packages: it should be normal to include the used export preset into the export dump such that everybody can easily continue distribution package development, adding and changing pages and exporting them repeatedly with the same export preset; and if the export configuration changes one day, it will be automatically included into the next export dump
Updated by Gerrit Code Review almost 4 years ago
Patch set 11 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/65770
Updated by Alexander Nitsche almost 4 years ago
Sorry guys, i tested the import of presets on root level manually and i missed obviously some points and fixed them with the latest patch:
- TCA bad structure: missed config => []
around type => 'passthrough'
- TCA: missing permission to create presets on root level as well: rootLevel => -1
- TCA: some leftovers of the removed hidden field
Greetings
Alex
Updated by Gerrit Code Review almost 4 years ago
Patch set 1 for branch 10.4 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/66582
Updated by Alexander Nitsche almost 4 years ago
- Status changed from Under Review to Resolved
- % Done changed from 0 to 100
Applied in changeset 9cddf73bc0ffd9fe26af11b880b19919c78b8c26.
Updated by Oliver Hader over 3 years ago
- Related to Bug #92985: Cannot show record history of pages anymore in TYPO3 v10.4.10 added
Updated by Oliver Hader over 3 years ago
Just stumbled over this ticket here...
Is/Was the original intention to export just the one preset that was used for exporting - or was it really about exporting all presets.
The current implementation is doing a lot to fake/simulate TCA which is not used in the end. My suggestion probably would have been either one of
- export preset definition with export-bundle - in terms of meta-data like "that was the configuration this export file was created with"
- use a dedicated handling for export preset records without applying TCA (and all its consequences)
Updated by Oliver Hader over 3 years ago
Besides that, it looks like item_uid
was storing the corresponding page ID already - now we have an additional pid
always being 0
since usually not modified using DataHandler - expect for importing an export preset.
Updated by Alexander Nitsche over 3 years ago
The initial intention was to export the preset along with the export dump. It should be minimal invasive and use existing processes of the EXT:impexp. The easiest and yet most flexible way was to introduce the TCA configuration, which gives you the ability to include no, one specific, several or all presets. The TCA should not provide yet a real backend form and thus configured all preset fields as passthrough. In this patch there should be no fields removed (like item_uid) but only added (like pid), and the existing presets should work flawlessly after having applied this patch and no manual steps should be required. Besides the immediate effect of providing the export of presets along with the dump this should serve as preparation for a real TCA configuration of presets in future TYPO3 releases and thus making them more visible and easier manageable.
The first use case of this patch was the distribution page tree development: Creating a distribution, exporting the page tree with its preset, letting someone else import the distribution, changing the page tree and exporting with the same export settings without having the configure the export setting manually.
About the impact of using the upgrade wizard: I was not aware of the requirement, that a patch of an LTS should not include a database update. If so, do you suggest to remove this patch again with the next TYPO3 10.4 release?