Bug #56484
closed
INCLUDE_TYPOSCRIPT: Content of non-UTF8-files does not show at all
0%
Description
In the overview TYPO3 says that it detected a certain number of lines in a template. However opening that template in the editor (including the content of external files) does not show any content at all - not even BEGIN/END-markers.
Files
Updated by Mathias Schreiber over 9 years ago
- Status changed from New to Needs Feedback
- Assignee set to Mathias Schreiber
Can you walk me through the steps here sometime?
Updated by Stefan Neufeind over 9 years ago
- Status changed from Needs Feedback to Accepted
Create a file like described below, save it in Latin1 encoding, do an INCLUDE_TYPOSCRIPT on it, open template-editor in the backend and tell it to "Include TypoScript file content" (checkbox below).
While of course usually those files should be utf8, it sometimes happens people get the encoding (or just one character through copy/paste) wrong. And then not displaying anything is ... sub-optimal :-)
Demo:
page = PAGE
page.10 = TEXT
page.10.value = Viele Grüße
(Edit: Tried it on current master. Still fails.)
Updated by Helmut Hummel over 9 years ago
- File encodingtest.php encodingtest.php added
- Status changed from Accepted to Needs Feedback
- PHP Version set to 5.4
Stefan Neufeind wrote:
While of course usually those files should be utf8, it sometimes happens people get the encoding (or just one character through copy/paste) wrong. And then not displaying anything is ... sub-optimal :-)
Well, the reason is, that htmlspecialchars() returns an empty string if it encounters invalid encoding since PHP 5.4
Just test it with attached file. How should we fix this?
Updated by Stefan Neufeind almost 9 years ago
Thanks for investigating. I've also taken another look in the php-docs for htmlspecialchars() as well as the comments there. The returning of empty strings is "new" since PHP 5.4.
Do I guess right that the inclusion of files works fine but it finally fails when the output is sent to the editor (and therefor htmlspecialchars is used, at the end of the process)? There seems to be no easy way to let PHP either pass or filter out problematic characters.
Some options I see:
option 1) Try to come up with a function to filter out "problematic" utf8-characters before calling htmlspecialchars(). (I was under the impression we had a tool-function in GeneralUtility - but no.)
=> Will "break" some things as soon as content is loaded and saved back to the files. If it's "just" comments in the code that might be "okay" but otherwise it might break content.
option 2) Try htmlspecialchars() on each individual file when it is included. If the string was not empty before, but is empty afterwards, don't include it and leave the include-line in the code.
=> We can only do that when we want to display code. For including and evaluating TypoScript there currently seems to be no problem, and leaving the include-lines in there would prevent using certain portions of code that worked until now.
=> For displaying TS it would at least display the rest of code and it would not break when somebody tries to save back the TS to files.
option 3) Maybe don't load the file-content in html-notation as part of the page but retrieve it in "binary" form through a XHR-call and find a way on the client-side (in JS) to feed it into the codeeditor?
=> That would prevent errors like this on the serverside. But I'm not sure what other problems we might trigger on the client in JavaScript.
Updated by Riccardo De Contardi over 8 years ago
I tried with both 6.2 and 7.5-dev latest master, and I was not able to reproduce it.
I even tried with 2 different installations (on MAC, with PHP 5.5.23 and on Windows, with 5.5.12)
I tried saving the files as ISO 8859-2, ANSI, Windows-1252 and even WIndows-1251 Cyrillic
every time I clicked on the checkbox "Include TypoScript file content", its content were included.
Do you think this issue can be closed, or should I perform a different test?
Updated by Riccardo De Contardi over 8 years ago
- Status changed from Needs Feedback to Closed
- Assignee deleted (
Mathias Schreiber)
No feedback within the last 90 days => closing this issue.
If you think that this is the wrong decision or experience this issue again, then please write to the mailing list typo3.teams.bugs with issue number and an explanation or open a new ticket and add a relation to this ticket number.
Updated by Riccardo De Contardi over 8 years ago
I reopen this; I am sorry that I forgot to test the file with special characters; I tested again using the attached file (Windows-1251 encoded) and the result is that the t3editor shows nothing (I mean, it is completely blank)
Tested on both 6.2.16 and 7.6.1-dev (latest master on a Windows enviroment with PHP 5.6.12)
Updated by Stefan Neufeind over 8 years ago
- Status changed from New to Accepted
Sorry for not providing feedback early. I wanted to retest this on current master and must have forgotten to do so. Glad you were able to reproduce the problem. So I guess that makes it kind of "accpeted" then. Imho we need some workaround here.
Updated by Mathias Schreiber over 8 years ago
I don't see any way how to fix this.
The JS gets false (non-UTF8) data to process, this breaks the editor.
The only thing we can add is an exception message that the editor wasn't able to load the data.
Updated by Riccardo De Contardi over 8 years ago
- Category changed from TypoScript to t3editor
- Target version set to Candidate for patchlevel
An error message like "The editor has encountered a problem loading the content of an external typoscript file; please check that the encoding of the typoscript files included is UTF-8" could be sufficient?
Updated by Mathias Schreiber about 8 years ago
- Status changed from Accepted to Rejected
We discussed this internally.
TYPO3 supports UTF8 and UTF8 only.