TYPO3 Core

this can only happen for 11.

Why? v9 is still LTS and marketed as an "enterprise CMS". Most big setups (the ones that are affected by such behaviour most) are starting just now to migrate now from 8 to 9 - hundreds-of-domain setups with loads of stakeholders and budgets of several 100k can't simply "early-adopt" or "upgarde to next version". The release schedule and time span for giving this kind of feedback for real-life setups (with a chance of fixing/implementing) of TYPO3 got way to tight lateley...

The workround is to implement a custom Middleware between these two:

'after' => [
    'typo3/cms-frontend/site'
],
'before' => [
    'typo3/cms-frontend/maintenance-mode'
]

In this Middleware you have to check whether you have a NullSite and react accordingly:

public function process(ServerRequestInterface $request, RequestHandlerInterface $handler): ResponseInterface
{
    if (!$request->getAttribute('site') instanceof \TYPO3\CMS\Core\Site\Entity\NullSite) {
        return $handler->handle($request);
    }

    return new \TYPO3\CMS\Core\Http\Response(
        'path/to/404.file',
        404
    );
}

Still, I don't get why the core can't do this small step by itself anymore (only maybe after the redirect middleware in the general case)!? You could simply have returned the setting in FE|pageNotFound_handling whenever there is a NullSite instead of removing these settings and exit with an "Ooops". Why were these settings dropped, then, if a Middleware can handle them flawlessly? The changelogs sound like this was not possible anymore, that's what made me write this ticket in the first place. But it turns out, it's still possible with not much effort.

For TYPO3 Core: We'd need to have a custom error handler within the SiteResolver middleware when no site was found. Question is how to configure it in a flexible way like we do with site configurations?