TYPO3 Core

The problem is that the default configuration of MySQL is to be case-insensitive in comparisons, for most of the available collations. See http://dev.mysql.com/doc/refman/5.0/en/case-sensitivity.html for more details.

The solution is to change the collation of the columns "identifier" and "name" in the tables sys_file and sys_file_processedfile to "utf8_bin", which is case-sensitive.

Both are utf8_general_ci:

identifier    varchar(200)    utf8_general_ci
name          tinytext        utf8_general_ci

Christian Weiske wrote:

Both are utf8_general_ci:
[...]

Ok, that's really strange... What is your MySQL version?

MySQL 5.5.28-1 - (Debian)

I don't really know what is going on here... According to the mentioned MySQL doc page, this behaviour has not really changed between MySQL 5.0 and 5.5.

Could you test what the output of these two SQL statements:

SELECT 'FOOBAR' COLLATE utf8_general_ci = 'foobar' COLLATE utf8_general_ci;
SELECT 'FOOBAR' COLLATE utf8_bin = 'foobar' COLLATE utf8_bin;

It should be 1 (for general) and 0 (for bin), at least that's true on my machine (5.5.29 on Fedora 17). UPDATE: I had mixed up "1" and "0" as results, now it is correct... Sorry for that.

I get 1 for general, 0 for bin:

mysql> SELECT 'FOOBAR' COLLATE utf8_general_ci = 'foobar' COLLATE utf8_general_ci;
+---------------------------------------------------------------------+
| 'FOOBAR' COLLATE utf8_general_ci = 'foobar' COLLATE utf8_general_ci |
+---------------------------------------------------------------------+
|                                                                   1 |
+---------------------------------------------------------------------+
1 row in set (0.00 sec)

mysql> SELECT 'FOOBAR' COLLATE utf8_bin = 'foobar' COLLATE utf8_bin;
+-------------------------------------------------------+
| 'FOOBAR' COLLATE utf8_bin = 'foobar' COLLATE utf8_bin |
+-------------------------------------------------------+
|                                                     0 |
+-------------------------------------------------------+
1 row in set (0.00 sec)

mysql> SELECT 'FOOBAR' COLLATE utf8_general_ci = 'foobar' COLLATE utf8_general_ci;
+---------------------------------------------------------------------+
| 'FOOBAR' COLLATE utf8_general_ci = 'foobar' COLLATE utf8_general_ci |
+---------------------------------------------------------------------+
|                                                                   1 |
+---------------------------------------------------------------------+
1 row in set (0.03 sec)

mysql> SELECT 'FOOBAR' COLLATE utf8_bin = 'foobar' COLLATE utf8_bin;
+-------------------------------------------------------+
| 'FOOBAR' COLLATE utf8_bin = 'foobar' COLLATE utf8_bin |
+-------------------------------------------------------+
|                                                     0 |
+-------------------------------------------------------+
1 row in set (0.00 sec)

mysql> show variables like "version%";
+-------------------------+---------------------+
| Variable_name           | Value               |
+-------------------------+---------------------+
| version                 | 5.1.61              |
| version_comment         | Source distribution |
| version_compile_machine | x86_64              |
| version_compile_os      | redhat-linux-gnu    |
+-------------------------+---------------------+

Same results for:

mysql> show variables like "version%";
+-------------------------+-------------------------------------------------------------+
| Variable_name           | Value                                                       |
+-------------------------+-------------------------------------------------------------+
| version                 | 5.0.95                                                      | 
| version_comment         | Source distribution                                         | 
| version_compile_machine | i386                                                        | 
| version_compile_os      | redhat-linux-gnu                                            | 
+-------------------------+-------------------------------------------------------------+

mysql> show variables like "version%";
+-------------------------+---------------------+
| Variable_name           | Value               |
+-------------------------+---------------------+
| version                 | 5.5.29-MariaDB-log  |
| version_comment         | Source distribution |
| version_compile_machine | x86_64              |
| version_compile_os      | Linux               |
+-------------------------+---------------------+

I spoke with Christian Weiske, while testing the problem.

The posted screenshot shows the filelist module, which gets his data from the directory listening => it shows both files. But a look in the database table sys_file shows, that only one record was inserted.

A click on the (i) button opens a popup and on both images you get the same sha1 key => same record, so the problem can be reproduced by Christian and me.

Alexander Opitz wrote:

I spoke with Christian Weiske, while testing the problem.

The posted screenshot shows the filelist module, which gets his data from the directory listening => it shows both files. But a look in the database table sys_file shows, that only one record was inserted.

A click on the (i) button opens a popup and on both images you get the same sha1 key => same record, so the problem can be reproduced by Christian and me.

Damn it, I completely forgot where exactly this bug is visible... You're right, you can only see it in the info module or the database directly. OTOH I don't quite understand why some people were able to reproduce this even in the list module and some only in info.

Nonetheless, we cannot really fix this easily; we will need to change the collation of the fields or implement case-sensitivity support in some other way.

I ran into this problem as I tried to migrate images from 4.4 to 6.0. The files were imported correctly as "kay.jpg" and "Kay.jpg". Both show in file module. The migration of the images in the content element inserted "Kay.jpg" instead of "kay.jpg" which could not be changed manually because the "kay.jpg" inserts record for "Kay.jpg". I changed the collation of the field "identifier" and it worked fine.

Changing the collation for this field wouldn't be too hard, would it?

@Andreas Otto † Wolf

The double listing in the list module may be caused by renaming and/or in combination with a non-case sensitive filesystem.

I think changing collation with a hint in the release notes should be sufficient. If there is a script which can remove old meta data informations (from files which don't exist) like in DAM then this hint should be added to the note, so you can clean up your database.

Implementing case-sensitivity in another way would be more problematic IMHO.

I had the same problem (TYPO3 6.0.4), but with the file extension. Our customer uploaded images with the names 1.jpg and 1.JPG.
A quick fix for me was to update the getUniqueName() function (TYPO3\CMS\Core\Utility\File\BasicFileUtility). I changed the $origFileInfo['file'] to lowercase, so the uploaded file get a lowercase filename.
Worked for me, but I don't know if it affects other parts.

Patch set 2 for branch master has been pushed to the review server.
It is available at https://review.typo3.org/23398

Patch set 3 for branch master has been pushed to the review server.
It is available at https://review.typo3.org/23398

Patch set 4 for branch master has been pushed to the review server.
It is available at https://review.typo3.org/23398

Patch set 5 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/23398

Patch set 6 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/23398

Patch set 7 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/23398

Patch set 8 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/23398

Patch set 9 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/23398