TYPO3 Core

This would be a very helpful addition.
Just recently I created a very similar solution by overriding the YouTubeRenderer, but other video media types would also benefit from this feature. I will try to create a patch later.

my current solution is: https://extensions.typo3.org/extension/hh_video_extender/

currently not the best code, but it does what it should

No, there is no native "defer" attribute for html videos, so the browser wouldn't know from that information alone, that the video needs to be "lazyloaded".
For doing so, you would also need to override the "src" attribute to something like "data-src" and handle the lazyloading logic, that replaces "data-src" with "src" in some custom javascript. Otherwise the src would always be fetched

Question is: Should the core viewhelper support that (to override the attribute name for "src") or should this not be done in the core and better be done by third-party extension code, that implement their own video viewhelper solutions?
https://developers.google.com/web/fundamentals/performance/lazy-loading-guidance/images-and-video#for_video_acting_as_an_animated_gif_replacement
Lazy loading videos is still a bit "tricky" to support all browsers and scenarios well enough.

Yes, unfortunately you're right, especially now with all the privacy data stuff and the needed javascript.
Maybe it's not wise to put something like that into the core - it could be a bit tricky with the CookieConsent plugins, at least for users in the EU.
Hmm stupid... or?!

Chrome supports lazy loading for iframes https://caniuse.com/?search=loading

I made a patch witch includes that:
https://forge.typo3.org/issues/96477

We integrated "loading=lazy" for images, we could add this for videos via iframe now as well, as <f:media> VH can deal with this already.