TYPO3 Core

RTE uses expandFolder from base class browselinks.

The files are sorted in increasing alphabetic order by t3lib_div::getFilesInDir.

Perhaps the problem is due to locale sorting?

In what language are the filenames?

t3lib_div::getFilesInDir uses asort. Perhaps it should use uasort($sortarray, 'strcoll') ?

Moving this issue to general core issues as it is not specific to htmlArea RTE.

The Backend language isset to German, but the problem occurs also with BE language set to English.
The filenames are english and german mixed.

I attached a screenshot, where you can see the wrong sorting in the link popup.

In your screenshot sorting is following ASCII order, with uppercase characters leading lowercase ones.
This is what t3lib_div.getFilesInDir() does when using asort() without a sorting order flag: http://typo3.org/fileadmin/typo3api-4.0.0/d6/d99/class_8t3lib__div_8php-source.html#l02552

To get files sorted in an case insensitive order, asort needs an additional flag called SORT_LOCALE_STRING

- asort($sortarray);
+ asort($sortarray, SORT_LOCALE_STRING);

Please try, because I didn't.

Wheater the sorting is wrong or not is a matter of taste. So I'd plead for an optional change.

While we are at it:
Another option to first sort by filename extension and afterwards by filename would be appreciated.

Bug is fixed in 4.3.1 so for me, this bug report could be closed.