TYPO3 Core

that seems a problem of Safari with Iframes. I was able to fix it adding to its parent ( .modal-type-iframe .modal-body):

position:absolute;
width:100%;
height:100%;

I have not yet tested the effect on other browsers

Same issue in Chrome 59
.modal-iframe needs

position: absolute;

No more reproducible on latest master (I think it has been solved with https://review.typo3.org/#/c/53184/ that should be soon backported to 8.7, too)