TYPO3 Core

Specifically, are any custom cache configs involved or maybe mutagen/ddev that could "lag" the cache?

It's true: after flushing all caches the problem is gone.

But still, why does a new field (that's not been created via "the system") disrupt the DataHandler this much?

Update: and I'm able to reproduce it with tt_content as well.

But still, why does a new field (that's not been created via "the system") disrupt the DataHandler this much?

Again, the DataHandler works with a cache of the database schema. If you add or change the database, the cache data does not reflect these changes. In some places, these information are required to do data conversions.

However, if you open a record ALL fields are retrieved from the database, for example with the "added" field. When you click safe, the value is passed to the DataHandler. As it is not included within TCA, code paths scanning this would not do TCA based things with it. That specific code path uses the cached database information, and the field is not contained for that table in the cache. The exception thrown is from the classes populated from cache and not from real database, and thus emitting the exception.

Will have a look to harden that in some way, now that it is clear how this happend. Should not crash, still need tho thing about it how to deal with it which may be not that easy. Using uncached schema information is not reasonable, as schema information is quite happy and redoing that for multiple fields and tables is a real performance hog and the reason the schema has been cached (and cleared on TYPO3 controller database changes).

In general, I still postulate that schema should not changed outside / without TYPO3. But we will see to harden it at that specific place.

Clearing caches on system changes is a common required task, and changeing database should be taken as a system change anyway.

To summarize this:

• Database Analyzer changes are safe as the caches are correctly cleared.
• This specific (rare) case happens if database structure is changed with plain DDL either by code or from the outside without clearing the database structure cache.
• We need decide how we can deal with this at the concrete place WITHOUT reading uncached schema information at that place. Not sure if data could be dropped silently, which may introduce other headache (for example if field is not nullabe etc).

Thanks for the feedback (and yes, it is clear that this is not bound to a specific table, just for changing database structure from outside without lushing the database schema cache.

Which should reduces the occurance of this issue a little bit, as most system would use the database analyzer to change the database scheme.

@Stefan Bürk, a big thank you for taking time to explain this matter!

However, if you open a record ALL fields are retrieved from the database, ...

It's happening as well when just toggling the hidden flag of a record btw.

Philipp Kitzberger wrote in #note-14:

@Stefan Bürk, a big thank you for taking time to explain this matter!

However, if you open a record ALL fields are retrieved from the database, ...

It's happening as well when just toggling the hidden flag of a record btw.

Again, it does not matter which field is changed - the whole retrieved record is transfered witht the field added without updateing the cached database schema information.
Trying to mitigate that in some way without loosing the cache and circumstance the fact that someone is altering the TYPO3 managed database from the outside without taking care to clear the caches afterwards.

How could we proceed with this? Add a note in the DB analyzer that says "Analysis works on internally cached information. Caches need to be flushed to recognize external changes to the database."?