Feature #59312

No possibility to remove missing indexed files from DB

Added by Robert Vock over 5 years ago. Updated 9 months ago.

Status:
Accepted
Priority:
Should have
Assignee:
-
Category:
File Abstraction Layer (FAL)
Target version:
-
Start date:
2014-06-03
Due date:
% Done:

0%

PHP Version:
7.2
Tags:
Complexity:
easy
Sprint Focus:

Description

There currently is no possibility to remove indexed files from DB, when they are missing.

The FileStorageIndexingTask marks the files as missing and the FalStatus-Report shows which files are missing. This report tells the user to restore the files and rerun the indexer to clear the flag.

This means it is not possible to delete files using FTP or otherwise the entries in the DB won't be cleared. It would be nice if the Indexer could be configured to delete missing files in the database, instead of only marking them as missing.

But I can imagine that it gets quite complicated, if the files are still in use somewhere and were deleted using FTP or some other direct access.


Related issues

Related to TYPO3 Core - Task #50876: Handling of deleted files in FAL Closed 2013-08-06
Related to TYPO3 Core - Bug #78473: "Fileadmin garbage collection" scheduler task doesn't remove entries from sys_file Closed 2016-10-27

History

#1 Updated by Frans Saris over 5 years ago

  • Status changed from New to Accepted
  • Priority changed from Could have to Should have

Indeed something we need to add. Maybe something we can add to System -> DB check and can link to from reports module.

And then some overview of all missing files and where they are linked + the possibility to remove some you have selected or all.

#2 Updated by Marcin Sągol over 5 years ago

+1 for some solution for this issue.

#3 Updated by Wolfgang Wagner over 5 years ago

This would be a nice feature indeed!

#4 Updated by Armin Vieweg about 5 years ago

+1

#5 Updated by Frans Saris almost 5 years ago

  • Tracker changed from Feature to Task
  • Target version set to 7.1 (Cleanup)
  • TYPO3 Version set to 6.2
  • Sprint Focus set to On Location Sprint

A new BE (sub) module needs to be created where a admin can view a list of all missing files with the posibility to remove all/only some records from database.

#6 Updated by Ingo Schmitt almost 5 years ago

  • Complexity set to hard

#7 Updated by Benni Mack over 4 years ago

  • Target version changed from 7.1 (Cleanup) to 7.4 (Backend)

#8 Updated by Susanne Moog over 4 years ago

  • Sprint Focus deleted (On Location Sprint)

#9 Updated by Susanne Moog over 4 years ago

  • Target version changed from 7.4 (Backend) to 7.5

#10 Updated by Benni Mack about 4 years ago

  • Target version deleted (7.5)

#11 Updated by Clemens Riccabona over 3 years ago

Upvote.
It is getting more and more anoying ...

#12 Updated by Frank Gerards over 3 years ago

It is indeed a dearly missing feature, which shouldnt be that hard to implement.
I would see this as a flag in the scheduler Task for updating the file reference index as this is the point,
where u are sure the "missing" state is most accurate.

Anyways you could check for file_exists before deleting the "missing" DB entries.

Clemens Riccabona wrote:

Upvote.
It is getting more and more anoying ...

#13 Updated by Markus Klein over 3 years ago

Keep in mind that you have to do a bit more when cleaning this.
You have to remove all references to that very file as well.

#14 Updated by Jörg Wagner over 3 years ago

Markus Klein wrote:

Keep in mind that you have to do a bit more when cleaning this.
You have to remove all references to that very file as well.

Not really. The major problem are files that are NOT referenced but still listed by the indexer als missing.
Any file that the indexer has seen in his lifetime will create an error when it is removed from the file system using FTP or shell access – even if there is no references at all!

Referenced files that are deleted can be considered operator errors and the request to bring them back is correct.

But I would be happy to have an option to remove the error messages for unreferenced files.
IMHO the indexer records for these files could even be removed without further notice.

#15 Updated by Philipp Gampe about 3 years ago

Until this is fixed, you can run the following SQL on regular base:

DELETE FROM sys_file
WHERE sys_file.missing=1
AND sys_file.uid NOT IN (SELECT sys_file_reference.uid_local FROM sys_file_reference WHERE sys_file_reference.uid_local=sys_file.uid)

#16 Updated by Jörg Wagner about 3 years ago

Steffen Müller has written a TYPO3 scheduler task class that does exactly what Philipp Gampe is proposing:
https://gist.github.com/stmllr/da259f62428cb0b499c48fbaedc932e3

#17 Updated by Viktor Livakivskyi about 3 years ago

I think, it would also make sense to run same query in Fileadmin garbage collection scheduler task, because after a while the Files flagged as missing report is filled with tons of un-useful /_recycler_/* entries. It makes impossible to see if there are really some problems with missing files, because of such spam entries.

#18 Updated by Philipp Gampe about 3 years ago

@Viktor sounds like another bug

#19 Updated by Viktor Livakivskyi about 3 years ago

@Phillip, done #78473 :)

#20 Updated by Leonie Philine Bitto 10 months ago

  • TYPO3 Version changed from 6.2 to 9
  • PHP Version set to 7.2
  • Complexity changed from hard to easy

This is still an issue in TYPO3 9.

Could https://gist.github.com/stmllr/da259f62428cb0b499c48fbaedc932e3 be included in the core?

#21 Updated by Susanne Moog 9 months ago

  • Tracker changed from Task to Feature

Also available in: Atom PDF