Bug #94729
Updated by Stephan Jorek over 3 years ago
Hello Everybody, as already indicated in [[http://forge.typo3.org/issues/91236]] under certain conditions the method … > @\TYPO3\CMS\Core\Utility\ArrayUtility::replaceAndAppendScalarValuesRecursive@ … sometimes re-numbers associative keys, which look like plain consecutively numbered (YAML-)list keys. Before TYPO3 version 10 this was not as obvious as it became in version 10 and above, because previous versions of the Form-Framework did not use the … > @\TYPO3\CMS\Core\Configuration\Loader\YamlFileLoader@ … as documented in [[https://docs.typo3.org/c/typo3/cms-core/master/en-us/Changelog/10.2/Feature-84203-UnifyFormSetupYAMLLoading.html]]. But of course it was/is also an issue in version below 10, but not in the form-framework. *Before getting into detail, here is detail - I already prepared a working solution: [[https://review.typo3.org/c/Packages/TYPO3.CMS/+/70246]]* fix, which will be committed and linked here, immediately after posting this issue.* h1. How can this be explained and reproduced? First of all I will describe the State of YAML-Loading in the Form-Framework before TYPO3 Version 10. Let's assume we have the following (simplified) setup: h2. Typoscript <pre> plugin.tx_form.settings.yamlConfigurations { 10 = EXT:form/Configuration/Yaml/FormSetup.yaml 20 = EXT:my_custom_extension/Configuration/Yaml/FormSetup.yaml } </pre> h2. @EXT:form/Configuration/Yaml/FormSetup.yaml@ <pre><code class="yaml"> TYPO3: CMS: Form: ########### FORMEDITOR CONFIGURATION ########### prototypes: standard: ########### DEFAULT FORM ELEMENT DEFINITIONS ########### formElementsDefinition: Form: formEditor: editors: 900: selectOptions: 10: value: '' label: 'formEditor.elements.Form.editor.finishers.EmptyValue.label' 20: value: 'EmailToSender' label: 'formEditor.elements.Form.editor.finishers.EmailToSender.label' 30: # … 40: # … 50: # … 60: # … # … </code></pre> h2. @EXT:my_custom_extension/Configuration/Yaml/FormSetup.yaml@ <pre><code class="yaml"> TYPO3: CMS: Form: prototypes: standard: formElementsDefinition: Form: formEditor: editors: 900: # Example: # Hide the EmailToSender finisher, as # the customer never wants to use it, # even not by accident 20: null </code></pre> h2. Processing of the Form-Framework YAML-files in TYPO3 ≤ v9 The Yaml-Parser turns both YAML-files into Arrays and both arrays are merged together using @array_replace_recursive@, see [[https://github.com/TYPO3/typo3/blob/9.5/typo3/sysext/form/Classes/Mvc/Configuration/YamlSource.php#L90-L94]] The result is something like: <pre><code class="yaml"> TYPO3: CMS: Form: prototypes: standard: formElementsDefinition: Form: formEditor: editors: 900: selectOptions: 10: value: '' label: 'formEditor.elements.Form.editor.finishers.EmptyValue.label' # Here is the 20: null from above 20: null 30: # … 40: # … 50: # … 60: # … </code></pre> h2. Processing of the Form-Framework YAML-files in TYPO3 ≥ v10 When we assume the exact same setup as explained above, nothing really changes, and everything works as expected, see [[https://github.com/TYPO3/typo3/blob/master/typo3/sysext/form/Classes/Mvc/Configuration/YamlSource.php#L69-L77]]. The only difference is, that this time the @YAML::parse@ is not used directly, but indirectly through the @YamlLoader@, which introduces the @import@ feature, see [[https://github.com/TYPO3/typo3/blob/master/typo3/sysext/form/Classes/Mvc/Configuration/YamlSource.php#L136-L139]]. Hence the result is the same as shown above. h1. A slightly different setup in TYPO3 ≥ v10 triggering the error h2. Typoscript <pre> # same as above plugin.tx_form.settings.yamlConfigurations { 10 = EXT:form/Configuration/Yaml/FormSetup.yaml 20 = EXT:my_custom_extension/Configuration/Yaml/FormSetup.yaml } </pre> h2. @EXT:form/Configuration/Yaml/FormSetup.yaml@ <pre><code class="yaml"> # the same as above TYPO3: CMS: Form: ########### FORMEDITOR CONFIGURATION ########### prototypes: standard: ########### DEFAULT FORM ELEMENT DEFINITIONS ########### formElementsDefinition: Form: formEditor: editors: 900: selectOptions: 10: value: '' label: 'formEditor.elements.Form.editor.finishers.EmptyValue.label' 20: value: 'EmailToSender' label: 'formEditor.elements.Form.editor.finishers.EmailToSender.label' 30: # … 40: # … 50: # … 60: # … # … </code></pre> h2. @EXT:my_custom_extension/Configuration/Yaml/FormSetup.yaml@ <pre><code class="yaml"> # Now we're using an import here! import: - { resource: "./Finishers/EmailToSender.yaml" } TYPO3: CMS: Form: prototypes: … </code></pre> h2. @EXT:my_custom_extension/Configuration/Yaml/Finishers/EmailToSender.yaml@ <pre><code class="yaml"> # The same content, as used in FormSetup.yaml previously TYPO3: CMS: Form: prototypes: standard: formElementsDefinition: Form: formEditor: editors: 900: # Example: # Hide the EmailToSender finisher, as # the customer never wants to use it, # even not by accident 20: null </code></pre> h2. The (broken) Result The result is now neither the same as above, nor correct anymore. It looks like: <pre><code class="yaml"> # the same as above TYPO3: CMS: Form: prototypes: standard: formElementsDefinition: Form: formEditor: editors: 900: selectOptions: # Here is the 20: null from above 0: null 10: value: '' label: 'formEditor.elements.Form.editor.finishers.EmptyValue.label' 20: value: 'EmailToSender' label: 'formEditor.elements.Form.editor.finishers.EmailToSender.label' 30: # … 40: # … 50: # … 60: # … </code></pre> h2. Processing of the Form-Framework YAML-file @imports@ in TYPO3 ≥ v10 So what happened? The reason for this, is that the @YamlLoader@ treats the @import@ differently, as the @YamlSource@ implementation does. The latter uses, as mentioned above, @array_replace_recursive@, whereas the @import@ of the @EmailToSender.yaml@ fragment-file into the @FormSetup.yaml@ file is using @ArrayUtility::replaceAndAppendScalarValuesRecursive@ to merge the two arrays, see [[https://github.com/TYPO3/typo3/blob/master/typo3/sysext/core/Classes/Configuration/Loader/YamlFileLoader.php#L164-L166]]. For understanding, it is important to know, that the array of the @FormSetup.yaml@ is merged into the array of the @EmailToSender.yaml@-file, so the array of the @EmailToSender.yaml@ is the first argument in the @ArrayUtility::replaceAndAppendScalarValuesRecursive@ method-call, see previous link. h1. @ArrayUtility::replaceAndAppendScalarValuesRecursive@ The purpose of this method is to merge two arrays like @array_replace_recursive@ does, with one exception - if the first argument (target array or one if it's subpaths) is a simple (YAML-) list, the key does not get replaced. Instead the source-value-list (array) is appended to the target-value-list (array) using a simple @array_merge($target, $source)@, see [[https://github.com/TYPO3/typo3/blob/master/typo3/sysext/core/Classes/Utility/ArrayUtility.php#L940-L943]]. And as mentioned in [[https://www.php.net/manual/en/function.array-merge.php]]: > Values in the input arrays with numeric keys will be renumbered with incrementing keys starting from zero in the result array. This is where the renumbered keys are created. <pre><code class="php"> // … // Simple lists get merged / added up if (!self::isAssociative($array1)) { return array_merge($array1, $array2); } // … </code></pre> Remember: @$array1@ is the array from to import, so in the @ArrayUtility::isAssociative@ the branch containing the @20: null@ declaration, looks like not being associative. h1. @ArrayUtility::isAssociative@ The current implementation of this method, assumes an array to be associative, when at least one key is a string, see [[https://github.com/TYPO3/typo3/blob/master/typo3/sysext/core/Classes/Utility/ArrayUtility.php#L924-L927]] <pre><code class="php"> // … public static function isAssociative(array $array): bool { return count(array_filter(array_keys($array), 'is_string')) > 0; } // … </code></pre> Is the YAML-declaration @20: null@ associative? Yes - we want to override exactly this key. Does the array contain a string as the key? No … now one could argument, to declare the key simply as a string, like @'20': null@. But this also does not work, due to special treatment of integer-like string-keys in arrays, which automatically are turned into integers. Here is an example to demonstrate this behaviour in the PHP 7.0 to 8.0: <pre><code class="shell"> for PHP in php{70,71,72,73,74,80} ; do echo "$PHP:" $PHP -r 'var_dump(array_keys(["10" => null]));' done php70: Command line code:1: array(1) { [0] => int(10) } php71: Command line code:1: array(1) { [0] => int(10) } php72: Command line code:1: array(1) { [0] => int(10) } php73: Command line code:1: array(1) { [0] => int(10) } php74: Command line code:1: array(1) { [0] => int(10) } php80: Command line code:1: array(1) { [0] => int(10) } </code></pre> So somehow, we must find a way to identify this case as being associative, while not losing the ability to identify simple lists. What is the difference between the simple list … <pre><code class="php"> [ 'value 1', 'value 2', 'value 3', 'value 4' ] </code></pre> … and the following associative list, with integer-only keys ? <pre><code class="php"> [ 10 => 'value', 20 => 'value', 30 => 'value', 40 => 'value' ] </code></pre> Obviously the keys in the latter array are not numbered consecutively. So the solution to the bug, is to assume an array to be associative, when either one of its keys is a string or all keys are not numbered consecutively (or when the array is empty). The first condition can even be even left out, as real string keys are no integers. So the final solution is: <pre><code class="php"> // … public static function isAssociative(array $array): bool { return !empty($array) && array_keys($array) !== range(0, count($array) - 1); } // … </code></pre> Someone working on the TYPO3-Core had the same idea, see [[https://github.com/TYPO3/typo3/blob/master/typo3/sysext/backend/Classes/Form/Container/SingleFieldContainer.php#L327]]. The updated UNIT-Tests for the ArrayUtility Unit-Tests in the intially linked Gerrit-review prove, that it is sufficient to just determine if all keys are not numbered consecutively. Thanks, for your patience! Stephan