Bug #55298

sys_history broken because of sys_log task

Added by Stefan Froemken over 5 years ago. Updated 8 months ago.

Status:
Closed
Priority:
Should have
Assignee:
-
Category:
Logging
Target version:
Start date:
2014-01-24
Due date:
% Done:

100%

TYPO3 Version:
6.2
PHP Version:
5.4
Tags:
Complexity:
Is Regression:
No
Sprint Focus:

Description

Hello Coreteam,

we have activated scheduler task to clean up sys_log, so only last 90 days are in log.
Now we have found out that the history of some older records were deleted, but still available in sys_history.
I have searched the core and found method getHistoryData in RecordHistory.php. There you build following Query:

$res = $GLOBALS['TYPO3_DB']->exec_SELECTquery('sys_history.*, sys_log.userid', 'sys_history, sys_log', 'sys_history.sys_log_uid = sys_log.uid
    AND sys_history.tablename = ' . $GLOBALS['TYPO3_DB']->fullQuoteStr($table, 'sys_history') . '
    AND sys_history.recuid = ' . intval($uid), '', 'sys_log.uid DESC', $this->maxSteps);

As you can see, this query can only build entries from sys_history where a relation to a sys_log entry exists. The entry from sys_log was deleted, so output of module history is incomplete.

So what to do?
- When deleting sys_log entries also delete related sys_history
- Modify task to only delete non related entries.

Stefan


Related issues

Related to TYPO3 Core - Bug #71950: sys_log doesn't show latest sys_history Entry if there are multiply with the same sys_log_uid Closed 2015-11-29

Associated revisions

Revision d047b314 (diff)
Added by Benni Mack over 1 year ago

[!!!][BUGFIX] Separate sys_history from sys_log db entries

Before, the history module fetched info about "modified records" from
sys_history+the authoritive user from a coupled sys_log entry.

Info about "insert" and "delete" was fetched from sys_log solely.

However, when using a scheduled cleanup task to truncate sys_log
then all history information is useless (see bug report).

The patch introduces a new RecordHistoryStore as an abstraction
for adding history entries (currently done solely within DataHandler).

It adds some additional, necessary SQL fields to sys_history to
store all information in there and creates an update wizard
to migrate all coupled sys_history/sys_log entries to a
new sys_history entry itself.

Additionally, the whole existing "RecordHistory" class is
now only necessary for fetching the so-called ChangeLog,
for a page or a specific record, and to do rollbacks, preparing
the history records so they can be worked on.

The whole logic for fetching the GET/POST parameters is moved
into the "ElementHistoryController", everything that is only possible
via Fluid is moved from the RecordHistory object and the
ElementHistoryController into the view.

Referencing from sys_log (Log module) into sys_history is
now done the other way around, storing information about
the corresponding history entry inside sys_log.
As a side-effect, sys_log should load faster.

Abstraction basis:
- sys_history is the only source of truth about the history of a record
- sys_log contains a reference to an history entry now
(inside sys_log.log_data) to link from the backend log module
- RecordHistoryStore exists for tracking changes to records
- RecordHistory is for retrieving, compiling the history/changelog and rollbacks
- ElementHistoryController is doing PSR-7 style request/response
handling and preparing data for the view
- Fluid is handling more view functionality now, removing
the need for doing <f:format.raw> everywhere in the templates.

Sidenotes:
  • Data within sys_history is now stored as JSON, not serialized anymore
  • Adding/deleting was previously stored in sys_log only, is now within sys_history
  • Moving records is now tracked (but not evaluated yet)
  • Highlight/Snapshot functionality within the Backend Module
    was removed

This functionality is built so it can also be used within Extbase
persistence and in FE in general in a future iteration.

Resolves: #55298
Resolves: #71950
Releases: master
Change-Id: I354317609099bac10c264b9932e331fa908c98be
Reviewed-on: https://review.typo3.org/53195
Reviewed-by: Andreas Fernandez <>
Tested-by: Andreas Fernandez <>
Tested-by: TYPO3com <>
Reviewed-by: Joerg Kummer <>
Tested-by: Joerg Kummer <>
Reviewed-by: Benni Mack <>
Tested-by: Benni Mack <>

History

#1 Updated by Steffen Müller over 5 years ago

sys_history and sys_log are tightly coupled. I would love to see this decoupled.
Nevertheless this story will not be solved with 6.2.

To solve this for 6.2, I would vote for
a) When deleting sys_log entries also delete related sys_history
and add a note to the task. Not sure how to handle this for 6.1.

What do others think?

#2 Updated by Steffen Müller over 5 years ago

I had a quick look at the backend. It looks like no history record is shown without a corresponding sys_log entry.
If that's true, I would suggest to leave things like they are for now. Once we have decoupled sys_history from sys_log, this can be be solved in a more clean way.

Would it work for you if you set up two tasks, one to clear sys_log, the other to clear sys_history?

#3 Updated by Stefan Froemken over 5 years ago

I think it would be better to add a col to sys_history called "be_user". So we can delete sys_log entries without losing the information WHO has changed what and when.

#4 Updated by Georg Kühnberger about 5 years ago

I highly support Stefan Froemken's suggestion to decouple the two tables; as the current situation is a) Double-entry bookkeeping and b) sys_log is somehow illnamed asuming that all tables with _log in the name might be deleted without loosing the BE-functionality to see the changes & diffs & being able to restore a change.

#5 Updated by Alexander Opitz over 3 years ago

  • Target version changed from 6.2.0 to 8 LTS

#6 Updated by Andreas Allacher over 3 years ago

Sys Log also has an icon to sys history which I think should be kept.
That one, most likely still would to have use the sys_log_uid but as mentioned in #71950 that could actually be easily fixed by retrieving the latest entry for the sys_log_uid in HistoryEntryViewHelper

#7 Updated by Benni Mack about 2 years ago

  • Target version changed from 8 LTS to next-patchlevel

#8 Updated by Gerrit Code Review almost 2 years ago

  • Status changed from New to Under Review

Patch set 15 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/53195

#9 Updated by Gerrit Code Review almost 2 years ago

Patch set 16 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/53195

#10 Updated by Gerrit Code Review almost 2 years ago

Patch set 17 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/53195

#11 Updated by Gerrit Code Review almost 2 years ago

Patch set 18 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/53195

#12 Updated by Gerrit Code Review over 1 year ago

Patch set 19 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/53195

#13 Updated by Gerrit Code Review over 1 year ago

Patch set 20 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/53195

#14 Updated by Gerrit Code Review over 1 year ago

Patch set 21 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/53195

#15 Updated by Gerrit Code Review over 1 year ago

Patch set 22 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/53195

#16 Updated by Gerrit Code Review over 1 year ago

Patch set 23 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/53195

#17 Updated by Gerrit Code Review over 1 year ago

Patch set 24 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/53195

#18 Updated by Gerrit Code Review over 1 year ago

Patch set 25 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/53195

#19 Updated by Gerrit Code Review over 1 year ago

Patch set 26 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/53195

#20 Updated by Gerrit Code Review over 1 year ago

Patch set 27 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/53195

#21 Updated by Gerrit Code Review over 1 year ago

Patch set 28 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/53195

#22 Updated by Benni Mack over 1 year ago

  • Status changed from Under Review to Resolved
  • % Done changed from 0 to 100

#23 Updated by Benni Mack 8 months ago

  • Status changed from Resolved to Closed

Also available in: Atom PDF