Bug #43027

Problems on case-sensitive filesystems

Added by Andreas Wolf almost 9 years ago. Updated almost 3 years ago.

Status:
Closed
Priority:
Must have
Assignee:
Category:
File Abstraction Layer (FAL)
Target version:
-
Start date:
2012-11-16
Due date:
% Done:

100%

Estimated time:
TYPO3 Version:
6.0
PHP Version:
Tags:
Complexity:
medium
Is Regression:
Sprint Focus:

Description

On case-sensitive file systems, files with the same name but different casing (like foobar.jpg and FOOBAR.jpg) will cause problems in different areas. From a quick test, the element browser displays the same preview image for both and cannot add one of them to the IRRE list.

The root cause seems to be some weirdness in the file list module; the index record is - for whatever reason - not loaded during the file list building and also not during the file processing. The problem could thus also come from the file record loading process or some weirdness in the indexer.


Files

2013-01-11_typo3-fal-list-case.png (14.8 KB) 2013-01-11_typo3-fal-list-case.png Christian Weiske, 2013-01-11 15:16

Related issues

Related to TYPO3 Core - Feature #44991: DB table COLLATION: disabled for ext_tables.sql but not for ext_tables_static+adt.sqlClosedMorton Jonuschat2013-01-30

Actions
Related to TYPO3 Core - Task #46553: Resolve issues with case-sensitive filesClosed2013-03-22

Actions
Related to TYPO3 Core - Bug #45504: Case sensitive filenames lead to file list problemsClosed2013-02-15

Actions
Related to TYPO3 Core - Bug #53575: Language module: Wrong argument passed to File::updateProperties()ClosedNicole Cordes2013-11-12

Actions
Related to TYPO3 Core - Task #53604: CleanUp Canonicalize and Unique Identifier in DriversClosedSteffen Ritter2013-11-13

Actions
#1

Updated by Andreas Wolf almost 9 years ago

The problem is that the default configuration of MySQL is to be case-insensitive in comparisons, for most of the available collations. See http://dev.mysql.com/doc/refman/5.0/en/case-sensitivity.html for more details.

The solution is to change the collation of the columns "identifier" and "name" in the tables sys_file and sys_file_processedfile to "utf8_bin", which is case-sensitive.

#2

Updated by Christian Weiske over 8 years ago

I cannot reproduce this issue; see my screenshot.

This is with current TYPO3 master, last commit 7ac01fe7.

#3

Updated by Andreas Wolf over 8 years ago

  • Status changed from Accepted to Needs Feedback

Christian Weiske wrote:

I cannot reproduce this issue; see my screenshot.

This is with current TYPO3 master, last commit 7ac01fe7.

Could you check what collation the filename and identifier columns in your sys_file table have? You could check this with e.g. phpMyAdmin in the table structure view.

#4

Updated by Christian Weiske over 8 years ago

Both are utf8_general_ci:

identifier    varchar(200)    utf8_general_ci
name          tinytext        utf8_general_ci

#5

Updated by Andreas Wolf over 8 years ago

Christian Weiske wrote:

Both are utf8_general_ci:
[...]

Ok, that's really strange... What is your MySQL version?

#6

Updated by Christian Weiske over 8 years ago

MySQL 5.5.28-1 - (Debian)

#7

Updated by Andreas Wolf over 8 years ago

I don't really know what is going on here... According to the mentioned MySQL doc page, this behaviour has not really changed between MySQL 5.0 and 5.5.

Could you test what the output of these two SQL statements:

SELECT 'FOOBAR' COLLATE utf8_general_ci = 'foobar' COLLATE utf8_general_ci;
SELECT 'FOOBAR' COLLATE utf8_bin = 'foobar' COLLATE utf8_bin;

It should be 1 (for general) and 0 (for bin), at least that's true on my machine (5.5.29 on Fedora 17). UPDATE: I had mixed up "1" and "0" as results, now it is correct... Sorry for that.

#8

Updated by Christian Weiske over 8 years ago

I get 1 for general, 0 for bin:

mysql> SELECT 'FOOBAR' COLLATE utf8_general_ci = 'foobar' COLLATE utf8_general_ci;
+---------------------------------------------------------------------+
| 'FOOBAR' COLLATE utf8_general_ci = 'foobar' COLLATE utf8_general_ci |
+---------------------------------------------------------------------+
|                                                                   1 |
+---------------------------------------------------------------------+
1 row in set (0.00 sec)

mysql> SELECT 'FOOBAR' COLLATE utf8_bin = 'foobar' COLLATE utf8_bin;
+-------------------------------------------------------+
| 'FOOBAR' COLLATE utf8_bin = 'foobar' COLLATE utf8_bin |
+-------------------------------------------------------+
|                                                     0 |
+-------------------------------------------------------+
1 row in set (0.00 sec)

#9

Updated by Alexander Opitz over 8 years ago

mysql> SELECT 'FOOBAR' COLLATE utf8_general_ci = 'foobar' COLLATE utf8_general_ci;
+---------------------------------------------------------------------+
| 'FOOBAR' COLLATE utf8_general_ci = 'foobar' COLLATE utf8_general_ci |
+---------------------------------------------------------------------+
|                                                                   1 |
+---------------------------------------------------------------------+
1 row in set (0.03 sec)

mysql> SELECT 'FOOBAR' COLLATE utf8_bin = 'foobar' COLLATE utf8_bin;
+-------------------------------------------------------+
| 'FOOBAR' COLLATE utf8_bin = 'foobar' COLLATE utf8_bin |
+-------------------------------------------------------+
|                                                     0 |
+-------------------------------------------------------+
1 row in set (0.00 sec)

mysql> show variables like "version%";
+-------------------------+---------------------+
| Variable_name           | Value               |
+-------------------------+---------------------+
| version                 | 5.1.61              |
| version_comment         | Source distribution |
| version_compile_machine | x86_64              |
| version_compile_os      | redhat-linux-gnu    |
+-------------------------+---------------------+

Same results for:

mysql> show variables like "version%";
+-------------------------+-------------------------------------------------------------+
| Variable_name           | Value                                                       |
+-------------------------+-------------------------------------------------------------+
| version                 | 5.0.95                                                      | 
| version_comment         | Source distribution                                         | 
| version_compile_machine | i386                                                        | 
| version_compile_os      | redhat-linux-gnu                                            | 
+-------------------------+-------------------------------------------------------------+
mysql> show variables like "version%";
+-------------------------+---------------------+
| Variable_name           | Value               |
+-------------------------+---------------------+
| version                 | 5.5.29-MariaDB-log  |
| version_comment         | Source distribution |
| version_compile_machine | x86_64              |
| version_compile_os      | Linux               |
+-------------------------+---------------------+
#10

Updated by Alexander Opitz over 8 years ago

I spoke with Christian Weiske, while testing the problem.

The posted screenshot shows the filelist module, which gets his data from the directory listening => it shows both files. But a look in the database table sys_file shows, that only one record was inserted.

A click on the (i) button opens a popup and on both images you get the same sha1 key => same record, so the problem can be reproduced by Christian and me.

#11

Updated by Andreas Wolf over 8 years ago

Alexander Opitz wrote:

I spoke with Christian Weiske, while testing the problem.

The posted screenshot shows the filelist module, which gets his data from the directory listening => it shows both files. But a look in the database table sys_file shows, that only one record was inserted.

A click on the (i) button opens a popup and on both images you get the same sha1 key => same record, so the problem can be reproduced by Christian and me.

Damn it, I completely forgot where exactly this bug is visible... You're right, you can only see it in the info module or the database directly. OTOH I don't quite understand why some people were able to reproduce this even in the list module and some only in info.

Nonetheless, we cannot really fix this easily; we will need to change the collation of the fields or implement case-sensitivity support in some other way.

#12

Updated by Thorben Nissen over 8 years ago

I ran into this problem as I tried to migrate images from 4.4 to 6.0. The files were imported correctly as "kay.jpg" and "Kay.jpg". Both show in file module. The migration of the images in the content element inserted "Kay.jpg" instead of "kay.jpg" which could not be changed manually because the "kay.jpg" inserts record for "Kay.jpg". I changed the collation of the field "identifier" and it worked fine.

Changing the collation for this field wouldn't be too hard, would it?

#13

Updated by Alexander Opitz over 8 years ago

Andreas Otto † Wolf

The double listing in the list module may be caused by renaming and/or in combination with a non-case sensitive filesystem.

I think changing collation with a hint in the release notes should be sufficient. If there is a script which can remove old meta data informations (from files which don't exist) like in DAM then this hint should be added to the note, so you can clean up your database.

Implementing case-sensitivity in another way would be more problematic IMHO.

#14

Updated by Thomas Fersterer over 8 years ago

I had the same problem (TYPO3 6.0.4), but with the file extension. Our customer uploaded images with the names 1.jpg and 1.JPG.
A quick fix for me was to update the getUniqueName() function (TYPO3\CMS\Core\Utility\File\BasicFileUtility). I changed the $origFileInfo['file'] to lowercase, so the uploaded file get a lowercase filename.
Worked for me, but I don't know if it affects other parts.

#15

Updated by Jigal van Hemert about 8 years ago

  • Status changed from Needs Feedback to Accepted
  • Target version deleted (6.0.0-RC2)
  • Complexity set to medium

The cause is of course that the path+file name is used as "identifier" in the Local Driver. While there are case sensitive and case insensitive file systems up till now we treat database values as case insensitive. The entire core completely ignores collation issues (sorting, comparing).
Requiring a case sensitive collation might result in problems with other DBMSs.

The best solution would be if the "identifier" in the Local Driver uses something else than the path+file name. Changing this in a patch level release would be problematic, but the current implementation of the "identifier" field is flawed IMO.

#16

Updated by Gerrit Code Review about 8 years ago

  • Status changed from Accepted to Under Review

Patch set 1 for branch master has been pushed to the review server.
It is available at https://review.typo3.org/23398

#17

Updated by Gerrit Code Review about 8 years ago

Patch set 2 for branch master has been pushed to the review server.
It is available at https://review.typo3.org/23398

#18

Updated by Gerrit Code Review about 8 years ago

Patch set 3 for branch master has been pushed to the review server.
It is available at https://review.typo3.org/23398

#19

Updated by Gerrit Code Review almost 8 years ago

Patch set 4 for branch master has been pushed to the review server.
It is available at https://review.typo3.org/23398

#20

Updated by Gerrit Code Review almost 8 years ago

Patch set 5 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/23398

#21

Updated by Gerrit Code Review almost 8 years ago

Patch set 6 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/23398

#22

Updated by Gerrit Code Review almost 8 years ago

Patch set 7 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/23398

#23

Updated by Gerrit Code Review almost 8 years ago

Patch set 8 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/23398

#24

Updated by Gerrit Code Review almost 8 years ago

Patch set 9 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/23398

#25

Updated by Andreas Wolf almost 8 years ago

  • Status changed from Under Review to Resolved
  • % Done changed from 0 to 100
#26

Updated by Benni Mack almost 3 years ago

  • Status changed from Resolved to Closed

Also available in: Atom PDF