TYPO3 Core

In composer mode in composer.json:

"extra": {
      "typo3/cms": {
          "app-dir": "custom",
          "web-dir": "public" 
      }
  },

In this case Environment::$projectPath would be /path/to/root/custom and Environment::$publicPath would be /path/to/root/public.

The typo3conf folder is then generated in /path/to/root/custom/typo3conf instead of /path/to/root/public/typo3conf.

This is actually by accident, because "custom" and "public" have the same length.
Reason is the code in https://git.typo3.org/Packages/TYPO3.CMS.git/blob/9fb677f6f3b3a1cd584b9ef183b35da771d3e25d:/typo3/sysext/install/Classes/FolderStructure/DefaultFactory.php#l134

$publicPath = substr(Environment::getPublicPath(), strlen(Environment::getProjectPath())+1);

which tries to substract the project path from the public path, while assuming that project path is an ancestor, which results in a public path "" to be calculate.
The result should rather be "../public".

We should rather generate a relative path and adapt the FolderStructur Factory to handle relative paths, include partent dots ("..").