Feature #86803

Possibility to import other yaml files into a yaml site configuration

Added by Jan Kornblum 11 months ago. Updated 6 days ago.

Status:
Resolved
Priority:
Must have
Assignee:
-
Category:
Link Handling, Site Handling & Routing
Start date:
2018-10-30
Due date:
% Done:

100%

PHP Version:
7.2
Tags:
url,yaml,routing,configuration
Complexity:
Sprint Focus:

Description

Having multi domain instances with same plugins (e.g. news, events etc.) located inside each instance it is currently neccessary to write down the same site configuration (here especially: route enhancers) for each domain. It would be really a nice feature to add the possibility to include / import other yaml (partial) configurations into a yaml site configuration file. Like ckeditor rte configuration does by...

imports:
    - { resource: "EXT:rte_ckeditor/Configuration/RTE/Processing.yaml" }
    - { resource: "EXT:rte_ckeditor/Configuration/RTE/Editor/Base.yaml" }
    - { resource: "EXT:rte_ckeditor/Configuration/RTE/Editor/Plugins.yaml" }

...the same might be possible for site configurations:

imports:
    - { resource: "typo3conf/sites/global/route-enhancers-news.yaml" }
    - { resource: "typo3conf/sites/gloabl/route-enhancers-events.yaml" }

And the referenced files might contain some partial configuration like:

routeEnhancers:
  NewsPlugin:
    type: Extbase
    limitToPages: [182,190,211,309] <- here all news detailpids inside a multidomain environment could be set
    extension: Extension
    plugin: News
    routes:
      - { routePath: '/{news_title}', _controller: 'News::show', _arguments: {'news_title': 'news'} }
    defaultController: 'News::show'
    aspects:
      news_title:
        type: PersistedAliasMapper
        tableName: 'tx_ext_domain_model_news'
        routeFieldName: 'title'

Related issues

Related to TYPO3 Core - Bug #89116: Selective writing of site config broken for nested structures Resolved 2019-09-09

Associated revisions

Revision 71fda451 (diff)
Added by Susanne Moog 9 days ago

[BUGFIX] Allow selective saving in SiteConfiguration

YAML SiteConfiguration allows imports and custom configuration
that is not configurable via UI to be written directly into the
configuration files, however, when saving with the UI these
settings are overwritten and resolved. This change changes the
SiteConfiguration in a way, that only modified values are written
and imports are completely left as they are.

Resolves: #86803
Releases: master, 9.5
Change-Id: I92ece4959d57f41535e6fc8c3c7c9d2894d7f920
Reviewed-on: https://review.typo3.org/c/Packages/TYPO3.CMS/+/61573
Tested-by: TYPO3com <>
Tested-by: Torben Hansen <>
Tested-by: Richard Haeser <>
Reviewed-by: Torben Hansen <>
Reviewed-by: Richard Haeser <>

Revision e3574986 (diff)
Added by Andreas Fernandez 6 days ago

[BUGFIX] Fix configuration composition for nested structures

When a new site configuration gets written, removed values are
determined first and changed values are determined afterwards to keep
nested structures intact.

Resolves: #89116
Related: #86803
Releases: master
Change-Id: I6deae2a3f6eed121aa2753f2ba9a5cafa058a54d
Reviewed-on: https://review.typo3.org/c/Packages/TYPO3.CMS/+/61646
Tested-by: TYPO3com <>
Tested-by: Susanne Moog <>
Tested-by: Benni Mack <>
Reviewed-by: Susanne Moog <>
Reviewed-by: Benni Mack <>

Revision 9c5f0333 (diff)
Added by Susanne Moog 6 days ago

[BUGFIX] Allow selective saving in SiteConfiguration

YAML SiteConfiguration allows imports and custom configuration
that is not configurable via UI to be written directly into the
configuration files, however, when saving with the UI these
settings are overwritten and resolved. This change changes the
SiteConfiguration in a way, that only modified values are written
and imports are completely left as they are.

Resolves: #86803
Related: #89116
Releases: master, 9.5
Change-Id: I92ece4959d57f41535e6fc8c3c7c9d2894d7f920
Reviewed-on: https://review.typo3.org/c/Packages/TYPO3.CMS/+/61603
Tested-by: TYPO3com <>
Tested-by: Andreas Fernandez <>
Reviewed-by: Andreas Fernandez <>

History

#1 Updated by Jan Kornblum 11 months ago

  • Category set to Link Handling, Site Handling & Routing

#2 Updated by Wouter Wolters 11 months ago

  • Target version changed from Candidate for patchlevel to Candidate for Major Version

#3 Updated by Felix Nagel 8 months ago

According to the blog extension (created by the TYPO3 GmbH), this should already work. See here: https://bitbucket.typo3.com/projects/EXT/repos/blog/browse/Configuration/Routes/Default.yaml

I'm able to confirm this is working for TYPO3 9.5.3.

#4 Updated by Ricky Mathew 7 months ago

I just managed to do it by tweaking the load() function of YAML loader class(TYPO3\CMS\Core\Configuration\Loader\YamlFileLoader) in a way it don't affect the configuration writing or reading of yaml files other than site configuration yaml file(config.yaml).see the changes:

   public function load(string $fileName, int $flags = self::PROCESS_PLACEHOLDERS | self::PROCESS_IMPORTS): array
   {
         $content = $this->getFileContents($fileName);
         $content = Yaml::parse($content);
         if (!is_array($content)) {
             throw new \RuntimeException('YAML file "' . $fileName . '" could not be parsed into valid syntax, probably empty?', 1497332874);
         }
+        else{
+           //checking whether it is a routing configuration file of a specific domain.
+           $siteConfiguration = isset($content['rootPageId'])?1:0;
+        }
         if (($flags & self::PROCESS_IMPORTS) === self::PROCESS_IMPORTS) {
-            $content = $this->processImports($content);
+            //patch to avoid writing import file configuration into config.yaml of site configuration
+            if(TYPO3_MODE == 'BE'){
+                //Ensures it's not a routing configuration file.
+                if($siteConfiguration == 0){
+                  $content = $this->processImports($content);
+                }
+            }
+            else{
+               $content = $this->processImports($content);
+            }
         }
         if (($flags & self::PROCESS_PLACEHOLDERS) === self::PROCESS_PLACEHOLDERS) {
              // Check for "%" placeholders
            $content = $this->processPlaceholders($content, $content);
        }

        return $content;
    }

Before this patch i was able to import files into config.yaml as follows:

imports:
  -
    resource: fileadmin/pluginconfig/news.yaml

But the problem is , on saving the configuration via 'sites' module in backend for a specific domain, the configurations in news.yaml file got directly written into config.yaml instead of preserving 'import' directinve in config.yaml since reading and writing of yaml files are dealt with the same load() functionality in YamlFileLoader.

After applying this patch i achieved what i intended.ie preserving 'import' directive in the config.yaml itself(by checking whether the context of loading yaml is front end or backend) even after saving the configuration via sites module.

If this finds helpful and sensible,can anyone push this to typo3 gerrit review?

#5 Updated by Ricky Mathew 7 months ago

  • Priority changed from Should have to Must have
  • Target version changed from Candidate for Major Version to Candidate for patchlevel

#6 Updated by Ricky Mathew 7 months ago

  • Tags set to url,yaml,routing,configuration

#7 Updated by hhjsfe no-lastname-given 7 months ago

I think the Problem is not the YamlFileLoader but the SiteConfiguration. I find for me a temporary solution. I'm override the SiteConfiguration with XCLASS

localconf.php

$GLOBALS['TYPO3_CONF_VARS']['SYS']['Objects'][\TYPO3\CMS\Core\Configuration\SiteConfiguration::class] = array(
    'className' => \Vendor\Ext\Configuration\SiteConfiguration::class
);

Extend SiteConfiguration

<?php
declare(strict_types = 1);

namespace Vendor\Ext\Configuration;

use Symfony\Component\Finder\Finder;
use TYPO3\CMS\Core\Configuration\Loader\YamlFileLoader;
use TYPO3\CMS\Core\Utility\GeneralUtility;

class SiteConfiguration extends \TYPO3\CMS\Core\Configuration\SiteConfiguration
{

    /**
     * Read the site configuration from config files.
     *
     * @return array
     * @throws \TYPO3\CMS\Core\Cache\Exception\NoSuchCacheException
     */
    protected function getAllSiteConfigurationFromFiles(): array
    {
        // Check if the data is already cached
        if (TYPO3_MODE != 'BE' && $siteConfiguration = $this->getCache()->get($this->cacheIdentifier)) {
            // Due to the nature of PhpFrontend, the `<?php` and `#` wraps have to be removed
            $siteConfiguration = preg_replace('/^<\?php\s*|\s*#$/', '', $siteConfiguration);
            $siteConfiguration = json_decode($siteConfiguration, true);
        }
        // Nothing in the cache (or no site found)
        if (empty($siteConfiguration)) {
            $finder = new Finder();
            try {
                $finder->files()->depth(0)->name($this->configFileName)->in($this->configPath . '/*');
            } catch (\InvalidArgumentException $e) {
                // Directory $this->configPath does not exist yet
                $finder = [];
            }
            /** @var YamlFileLoader $loader */
            $loader = GeneralUtility::makeInstance(YamlFileLoader::class);
            $siteConfiguration = [];
            $siteConfigurationCache = [];
            foreach ($finder as $fileInfo) {
                $configuration = $loader->load(GeneralUtility::fixWindowsFilePath((string)$fileInfo), 0);
                $configurationCache = $loader->load(GeneralUtility::fixWindowsFilePath((string)$fileInfo));
                $identifier = basename($fileInfo->getPath());

                $siteConfiguration[$identifier] = $this->handleDeprecated($configuration);
                $siteConfigurationCache[$identifier] = $this->handleDeprecated($configurationCache);;
            }
            $this->getCache()->set($this->cacheIdentifier, json_encode($siteConfigurationCache));
        }
        return $siteConfiguration ?? [];
    }

    /**
     * @param $configuration array
     * @return array
     */
    private function handleDeprecated($configuration)
    {
        if (isset($configuration['site'])) {
            trigger_error(
                'Site configuration with key \'site\' has been deprecated, remove indentation level and site key.',
                E_USER_DEPRECATED
            );
            $configuration = $configuration['site'];

        }
        return $configuration;
    }

}

I render the Yaml File twice. One time for the config.yaml without processing the imports. And the second time for the Cache file with processing the imports.

#8 Updated by Michael Stopp 4 months ago

I would go as far as calling the current implementation for saving a SiteConfiguration buggy, because it basically deactivates the imports feature of config.yaml. This can't possibly be the intended behaviour...

#9 Updated by Gerrit Code Review 18 days ago

  • Status changed from New to Under Review

Patch set 4 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/61573

#10 Updated by Gerrit Code Review 10 days ago

Patch set 5 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/61573

#11 Updated by Gerrit Code Review 10 days ago

Patch set 6 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/61573

#12 Updated by Gerrit Code Review 10 days ago

Patch set 7 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/61573

#13 Updated by Gerrit Code Review 10 days ago

Patch set 8 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/61573

#14 Updated by Gerrit Code Review 10 days ago

Patch set 9 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/61573

#15 Updated by Gerrit Code Review 9 days ago

Patch set 1 for branch 9.5 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/61603

#16 Updated by Susanne Moog 9 days ago

  • Status changed from Under Review to Resolved
  • % Done changed from 0 to 100

#17 Updated by Gerrit Code Review 7 days ago

  • Status changed from Resolved to Under Review

Patch set 2 for branch 9.5 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/61603

#18 Updated by Andreas Fernandez 7 days ago

  • Related to Bug #89116: Selective writing of site config broken for nested structures added

#19 Updated by Gerrit Code Review 6 days ago

Patch set 3 for branch 9.5 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/61603

#20 Updated by Susanne Moog 6 days ago

  • Status changed from Under Review to Resolved

Also available in: Atom PDF