Turns out there IS an event that can be utilized, making the whole issue obsolete. Documented here, in case anyone ever wants to do that (HINT: Use your time to convert this hook to a PSR-14 event instead!).

So this hook is doing the magic:

$GLOBALS['TYPO3_CONF_VARS']['SC_OPTIONS']['t3lib/class.t3lib_pagerenderer.php']['render-postProcess']

It is executed as a last step within the PageRenderer class, passing all rendered parts of a page to the event.

So you can provide an extension that does the following.

Register the hook, in ext_localconf.php¶

<?php
        $GLOBALS['TYPO3_CONF_VARS']['SC_OPTIONS']['t3lib/class.t3lib_pagerenderer.php']['render-postProcess'][] =
            \YourVendor\YourExtension\Hooks\PostRenderHook::class . '->postProcessKillBackendCSS';

Create a hook class in Classes/Hooks/PostRenderHook.php¶

<?php
namespace YourVendor\YourExtension\Hooks;

class PostRenderHook
{
    public function postProcessKillBackendCSS(&$params, $object) {
        if (isset($GLOBALS['MY_EXTENSION']['removeBackendCss'])) {
            $params['cssFiles'] =
                preg_replace(
                    '@<link rel="stylesheet" href="[^"]+backend.css[^>]+>@imsU',
                    '<!-- removed TYPO3 Backend via popular demand -->',
                    $params['cssFiles']
                );

            // You could do other hacks here as well, i.e. operate on $params['bodyContent']
        }
    }
}

Adjust your backend Controller, i.e. in a global initializeAction method:¶

$GLOBALS['MY_EXTENSION']['removeBackendCss'] = true;

The last step is important to instruct the hook that you only want to remove the backend CSS on your specific backend module. Else, you'd kill the backend.css file in EVERY backend module.

Abusing a global variable for this is the easiest approach, but a bit dirty. You might create a singleton object at that place which you later query, if you want to solve it "properly". Or you might let the hook introspect the body of the rendered page and get triggered by specific classes or whatnot.