Bug #73567
closed
cURL Proxy HTTP header not cleaned correctly
100%
Description
After the TER changes from http:// to https://, my environment failed to update the extension list and the languages from TER. I'm using cURL and a cURL proxy server and could identify the following problem:
In GeneralUtility::getUrl(), the headers are loaded to detect redirects and cleaned up again. But with my proxy, I have an extra header block from the proxy.
HTTP/1.0 200 Connection established
HTTP/1.1 200 OK
Server: nginx/1.9.9
Date: Fri, 19 Feb 2016 13:17:32 GMT
Content-Type: application/xml
Content-Length: 6758
Connection: keep-alive
Last-Modified: Fri, 19 Feb 2016 02:00:13 GMT
X-Cacheable: YES: Static resources are always cached
cache-control: public, max-age=86400
X-Varnish: 1094220381 1093945158
Age: 9442
Via: 1.1 varnish
X-Cache: HIT
Strict-Transport-Security: max-age=15768000; preload;
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
Content-Security-Policy-Report-Only: script-src 'self' 'unsafe-inline' 'unsafe-eval' https://piwik.typo3.org https://maps.google.com https://*.googleapis.com https://cdn.jquerytools.org; style-src 'self' 'unsafe-inline' https://fonts.googleapis.com; report-uri https://report-uri.io/report/90234a82d771236e837f46bebcbe46a5/reportOnly
<?xml version="1.0" standalone="yes" ?>
<TERlanguagePackIndex>
<meta>
<timestamp>1455...
So, only the first header block (proxy) will be cleaned up and the function will return the original header as content. As long the TER was defined with http:// in the core and redirected to https:// by server rewrite rules, I did not see any problem. Maybe the GeneralUtility::getUrl() called itself recursively to process the redirect and cleaned the header up correctly. Now, without the redirects, it failed.
The proxy headers must be removed separately.
Updated by Gerrit Code Review almost 9 years ago
- Status changed from New to Under Review
Patch set 1 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/46779
Updated by Morton Jonuschat almost 9 years ago
I don't see this as a bug in TYPO3 handling the response. The response you are receiving is broken according to current HTTP Standards. HTTP 1.0 and HTTP 1.1 Specs define the response format as
Response = Status-Line
*(( general-header
| response-header
| entity-header ) CRLF)
CRLF
[ message-body ]
Since the first blank line (CRLF) is the separator between the Header and the Message Body TYPO3 is handling the response according to standards and treating the second "header" as part of the message body. It seems to me your proxy is doing something funny with the HTTPS connections (or your curl version is), as you also have the problem that the status lines in the example are conflicting - is the response in HTTP 1.0 or 1.1 format?
Updated by Morton Jonuschat almost 9 years ago
Reading further through the code it might be related to the setting $GLOBALS['TYPO3_CONF_VARS']['SYS']['curlProxyTunnel'], could you check what's your configuration value is for this parameter and maybe try toggling it?
Updated by Albrecht Köhnlein over 8 years ago
$GLOBALS['TYPO3_CONF_VARS']['SYS']['curlProxyTunnel'] is disabled but even when I enable it, nothing changes. Response is still ...
HTTP/1.0 200 Connection established HTTP/1.1 200 OK Server: nginx/1.9.9 Date: Mon, 22 Feb 2016 07:59:42 GMT Content-Type: application/xml Content-Length: 6758 Connection: keep-alive ...
And about your question if the response is HTTP 1.0 or 1.1: As you can see in the response headers, the original server's response is HTTP/1.1 and the proxy says HTTP/1.0.
Updated by Albrecht Köhnlein over 8 years ago
I've got some new information ... And it really seems to be a problem with HTTPS requests through a proxy.
There's an interesting discussion about that toppic on http://stackoverflow.com/a/34537988 including links to specifications:
- http://tools.ietf.org/rfc/rfc2817 > 5.3 Establishing a Tunnel with CONNECT
- http://www.web-cache.com/Writings/Internet-Drafts/draft-luotonen-web-proxy-tunneling-01.txt > 3.2. Proxy Response
So I still think, it should be solved in TYPO3 as it's a valid response with the two header blocks. But maybe I have to change my patch to differ between HTTP and HTTPS requests.
Updated by Gerrit Code Review over 8 years ago
Patch set 2 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/46779
Updated by Gerrit Code Review over 8 years ago
Patch set 3 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/46779
Updated by Albrecht Köhnlein over 8 years ago
- Status changed from Under Review to Resolved
- % Done changed from 0 to 100
Applied in changeset 09f1f5632e05a9140a7d65ceca0f38caa5f335eb.
Updated by Gerrit Code Review over 8 years ago
- Status changed from Resolved to Under Review
Patch set 1 for branch TYPO3_7-6 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/47294
Updated by Gerrit Code Review over 8 years ago
Patch set 1 for branch TYPO3_6-2 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/47308
Updated by Albrecht Köhnlein over 8 years ago
- Status changed from Under Review to Resolved
Applied in changeset d81e40d414346c2a758a66cea8d7068e026b2cd5.
Updated by