Project

General

Profile

Actions

Task #87528

open

VimeoHelper/AbstractOEmbedHelper must send Referer Header to fetch proper oEmbed.json for videos with restricted embedding.

Added by Leonie Philine almost 6 years ago. Updated 5 months ago.

Status:
New
Priority:
Should have
Assignee:
-
Category:
File Abstraction Layer (FAL)
Target version:
-
Start date:
2019-01-23
Due date:
% Done:

0%

Estimated time:
TYPO3 Version:
11
PHP Version:
7.4
Tags:
Complexity:
Sprint Focus:
On Location Sprint

Description

VideoHelper uses an oEmbed request, as documented here: https://developer.vimeo.com/api/oembed/videos

curl 'https://vimeo.com/api/oembed.json?width=2048&url=https%3A%2F%2Fvimeo.com%2F123456789' | python -m json.tool
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   387  100   387    0     0   1646      0 --:--:-- --:--:-- --:--:--  1646
{
    "domain_status_code": 403,
    "height": 1152,
    "html": "<iframe src=\"https://player.vimeo.com/video/123456789?app_id=122963\" width=\"2048\" height=\"1152\" frameborder=\"0\" allow=\"autoplay; fullscreen\" allowfullscreen></iframe>",
    "provider_name": "Vimeo",
    "provider_url": "https://vimeo.com/",
    "type": "video",
    "uri": "/videos/123456789",
    "version": "1.0",
    "video_id": 123456789,
    "width": 2048
}

Note:
Due to privacy reasons, the video ID was replaced by 123456789.
To test this, create a Vimeo video, set the privacy settings to "Who can watch?" to "Anyone" and "Where can this be embedded?" to "Specific domains" and choose your domain.

The API docs (https://developer.vimeo.com/api/oembed/videos) state the following:

NOTE: In the case of a video with domain-level privacy, 
the oEmbed request returns a simplified response containing the embed code only and no private metadata. 
To get the complete response, send the Referer header along with the request, 
and set its value to the video's whitelisted domain.

This is the case here. As you see, the JSON response contains no title.

When adding the video in the TYPO3 filelist, it will be successfully saved as ".vimeo" ($oEmbed['title'] . '.' . $fileExtension in \TYPO3\CMS\Core\Resource\OnlineMedia\Helpers\AbstractOEmbedHelper::transformMediaIdToFile):

            if (!empty($oEmbed)) {
                $fileName = $oEmbed['title'] . '.' . $fileExtension;
            } else {
                $fileName = $mediaId . '.' . $fileExtension;
            }

Actually, $oEmbed['title'] is undefined (!isset()) here, but the script only checks if $oEmbed is empty, which is insufficient. (This is a bug on its own!)

Following, a success flash message is shown, but the file will not be visible because of the default filterHiddenFilesAndFolders filter defined in defaultFilterCallbacks.
When trying to create the vimeo file once more, \TYPO3\CMS\Core\Resource\OnlineMedia\Helpers\AbstractOnlineMediaHelper::findExistingFileByOnlineMediaId() will find the existing file, but not tell the user about it.
So again a green success flash message is shown. (This is another bug. The user should be informed that this video exists already in the folder.)

The actual solution, however, would be to send a referer header with the hostname, like:

curl 'https://vimeo.com/api/oembed.json?width=2048&url=https%3A%2F%2Fvimeo.com%2F123456789' -H 'Referer: https://example.com' | python -m json.tool 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   950  100   950    0     0   4166      0 --:--:-- --:--:-- --:--:--  4166
{
    "account_type": "pro",
    "author_name": "Redacted",
    "author_url": "https://vimeo.com/redacted",
    "description": "",
    "domain_status_code": 200,
    "duration": 235,
    "height": 1152,
    "html": "<iframe src=\"https://player.vimeo.com/video/123456789?app_id=122963\" width=\"2048\" height=\"1152\" frameborder=\"0\" title=\"Redacted video title\" allow=\"autoplay; fullscreen\" allowfullscreen></iframe>",
    "is_plus": "0",
    "provider_name": "Vimeo",
    "provider_url": "https://vimeo.com/",
    "thumbnail_height": 540,
    "thumbnail_url": "https://i.vimeocdn.com/video/redacted_960.jpg",
    "thumbnail_url_with_play_button": "https://i.vimeocdn.com/filter/overlay?src0=https%3A%2F%2Fi.vimeocdn.com%2Fvideo%2Fredacted_960.jpg&src1=http%3A%2F%2Ff.vimeocdn.com%2Fp%2Fimages%2Fcrawler_play.png",
    "thumbnail_width": 960,
    "title": "Redacted video title",
    "type": "video",
    "upload_date": "2019-01-01 00:00:00",
    "uri": "/videos/123456789",
    "version": "1.0",
    "video_id": 123456789,
    "width": 2048
}

You can see the full response being sent due to the -H 'Referer: https://example.com' header. (For testing, use the hostname you applied as embed restriction in the vimeo privacy settings.)

Only now is a title sent, and $oEmbed['title'] . '.' . $fileExtension no longer results in .vimeo but, in this example case, in Redacted video title.vimeo.

A little gotcha:
Unfortunately the filelist module does not "know" which of the TYPO3 installation's possibly many domain records match the video's privacy restrictions.
Therefore, TYPO3 would have no choice but to try sys_domain records until the oEmbed request returns a video title.
Meaning: If only one of multiple sites within a TYPO3 installation is whitelisted for the video, then the video can be added.
Otherwise you might want to let the request fail or succeed with a warning to the user (using the video ID as filename and remaining unable to set a preview image (thumbnail).

Target:
Same in TYPO3 8 and 9.

Actions #1

Updated by Susanne Moog over 5 years ago

Hey, thanks for the detailed report.

Do you think you could prepare a patch for this and push it to our review system? See https://docs.typo3.org/typo3cms/ContributionWorkflowGuide/Index.html for more information.

Actions #2

Updated by Gerrit Code Review almost 4 years ago

  • Status changed from New to Under Review

Patch set 1 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/67495

Actions #3

Updated by Gerrit Code Review almost 4 years ago

Patch set 2 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/67495

Actions #4

Updated by Gerrit Code Review almost 4 years ago

Patch set 3 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/67495

Actions #5

Updated by Gerrit Code Review almost 4 years ago

Patch set 4 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/67495

Actions #6

Updated by Gerrit Code Review almost 4 years ago

Patch set 5 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/67495

Actions #7

Updated by Benni Mack almost 3 years ago

  • Status changed from Under Review to New
Actions #8

Updated by Leonie Philine over 2 years ago

  • TYPO3 Version changed from 9 to 11
  • PHP Version changed from 7.3 to 7.4

A little help for anyone who struggles with FAL handling Vimeo videos that have set privacy settings "Who can watch?" to "Anyone" and "Where can this be embedded?" to "Specific domains":

Step 1:
Override the Vimeo online media helper:

ext_tables.php

<?php

call_user_func(function () {
    $GLOBALS['TYPO3_CONF_VARS']['SYS']['fal']['onlineMediaHelpers']['vimeo'] = \Vendor\ExtensionName\Resource\OnlineMedia\Helpers\VimeoDomainRestrictionAwareHelper::class;
});

Step 2:
Let TYPO3 try all base variant URIs of all configured sites as Referer header, and stop trying once extended metadata (see original issue description above!) was returned by Vimeo.
The extended metadata is here recognized by the presence of the title attribute, but other heuristics could be used just as well.

EXT:extension_name/Classes/Resource/OnlineMedia/Helpers/VimeoDomainRestrictionAwareHelper.php

<?php

namespace Vendor\ExtensionName\Resource\OnlineMedia\Helpers;

use TYPO3\CMS\Core\Configuration\SiteConfiguration;
use TYPO3\CMS\Core\Http\RequestFactory;
use TYPO3\CMS\Core\Resource\OnlineMedia\Helpers\VimeoHelper;
use TYPO3\CMS\Core\Site\SiteFinder;
use TYPO3\CMS\Core\Utility\GeneralUtility;

class VimeoDomainRestrictionAwareHelper extends VimeoHelper
{
    /**
     * Get OEmbed data - overriding `\TYPO3\CMS\Core\Resource\OnlineMedia\Helpers\AbstractOEmbedHelper::getOEmbedData()` to try `Referer` headers
     * until, if the Referer matches, unrestricted oEmbed data is returned.
     *
     * Vimeo uses the `Referer` header for videos with domain-embed-restriction.
     *
     * @param string $mediaId
     * @return array|null
     */
    protected function getOEmbedData($mediaId)
    {
        $oEmbed = null;

        $siteConfiguration = GeneralUtility::makeInstance(SiteConfiguration::class);
        $siteFinder = GeneralUtility::makeInstance(SiteFinder::class);
        foreach ($siteFinder->getAllSites() as $site) {
            // Try all sites and break if the 'title' attribute is set.

            $configuration = $siteConfiguration->load($site->getIdentifier());

            $variants = [$configuration['base'], ...array_reduce($configuration['baseVariants'], function($accumulatedVariants, $baseVariant) {
                $accumulatedVariants[] = $baseVariant['base'];
                return $accumulatedVariants;
            }, [])];

            foreach ($variants as $variant) {

                $oEmbed = GeneralUtility::makeInstance(RequestFactory::class)
                    ->request($this->getOEmbedUrl($mediaId), 'GET', ['headers' => ['Referer' => $variant]])
                    ->getBody()->getContents();

                if ($oEmbed) {
                    $oEmbed = json_decode($oEmbed, true);

                    if (isset($oEmbed['title'])) {
                        break 2;
                    }
                }
            }

        }

        return $oEmbed;
    }
}

Actions #9

Updated by Georg Ringer 5 months ago

  • Sprint Focus set to On Location Sprint
Actions

Also available in: Atom PDF