Feature #61110
open
Epic #77562: Misbehaviors with datetime values and timezones
Support for timezones in all date fields in TYPO3 BE
Added by Nikola Stojiljković over 9 years ago. Updated about 2 years ago.
0%
Description
Johannes Feustel:
As an editor and as a dev
AC
i want a documentation about how datetime filed acutally behave in typo3
so that i understand what values (TZ) to enter and how to test
there should be a document/wiki page listing all relevant datetime fields like
- access stop start of content and pages and records
- admin panel
- scheduler task
- publish date for workspaces
Define meaningful use cases covering each of those like:
beeing in a specific timezone like CEST while server is in PST, entering a datetime into start field of a page
what will be save to the db? local time? server time?
when will it go live acutally?
How to test? what do i need to emter into the mock date filed of the admin panel (server time?)?Description
All time fields in TYPO3 should be easy understandable (format) and lead to reliable result.
Format should ideally be based on a user setting and for all the places below TYPO3 should display the timezoneFollowing cases should be covered
See also
- admin panel
- schedule for content desired for display in different timezone compared to server time
- date time of a workspace to auto publish
- parse datetime string possible like "today", "MM:HH DD-MM-YYYY", ISO 8601: "YYYY-MM-DD HH:MM", "Tue, 24 Nov 09 16:00:00 -0500" in the client?
- make it possible for the editor to change format of datetime fields in user settings?
- http://forge.typo3.org/issues/27919 Preview in Admin Panel does not handle Timezone correctly
- http://forge.typo3.org/issues/32081 document timezone handling / eval datetime
- http://forge.typo3.org/issues/20088 In BE time and datetime fields GMT-0 will be shown by ja
Files
| be_user_date_settings.png (85.2 KB) be_user_date_settings.png | Nikola Stojiljković, 2014-08-25 22:18 | ||
| sample_fields_with_timezone.png (28.1 KB) sample_fields_with_timezone.png | Nikola Stojiljković, 2014-08-25 22:18 | ||
| sample_fields_with_timezone.png (14.7 KB) sample_fields_with_timezone.png | Nikola Stojiljković, 2014-08-25 22:20 | ||
| be_user_date_settings.png (48.5 KB) be_user_date_settings.png | Nikola Stojiljković, 2014-08-25 22:20 | ||
| sample_kolkata.png (10.6 KB) sample_kolkata.png | Nikola Stojiljković, 2014-08-26 11:18 | ||
| sample_belgrade.png (11.6 KB) sample_belgrade.png | Nikola Stojiljković, 2014-08-26 11:18 | ||
| sample_chatam.png (10.6 KB) sample_chatam.png | Nikola Stojiljković, 2014-08-26 11:22 |
Updated by Nikola Stojiljković over 9 years ago
- add "Date format" setting in the user settings
- add "Date-time format" setting in the user settings
- add "Time format" setting in the user settings
- add "Time zone" setting in the user settings
- add "Show timezone in date pickers" setting in the user settings
- refactor jsfunc.evalfield.js so it can be easily extendable (and maintainable)
- my idea is to go with Prototype.js OOP since it's already loaded and available
- Integrate date.js parsing logic
- add support for for overriding php.ini timezone setting in LocalConfiguration.php (since it's a bad practice to store dates in "server timezone") and document best use case scenarios. Additional useful info on storing dates in MySQL: http://cdn.oreillystatic.com/en/assets/1/event/36/Time%20Zones%20and%20MySQL%20Presentation.pdf
Updated by Markus Klein over 9 years ago
Please refrain from creating anything new with Prototype. We try to get rid of it, please use jQuery.
Updated by Nikola Stojiljković over 9 years ago
jQuery is not my first choice when working with "OOP" JS. And this refactoring has very little to do with DOM (apart from maybe using jQuery to read data attributes of a DOM element - a feature I will add for detail customization of the eval logic).
Is there an suggested alternative for the TYPO3 BE? I can always use something simple like this http://ejohn.org/blog/simple-javascript-inheritance/ (which is in fact based on Prototype.js).
Updated by Markus Klein over 9 years ago
Hi!
Please discuss this on slack with others (I just Stucki to invite you), I'm not so much into the JS part.
Maybe talk to Andreas Wolf or Benni Mack.
Thanks
Updated by Gerrit Code Review over 9 years ago
- Status changed from New to Under Review
Patch set 1 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/32360
Updated by Gerrit Code Review over 9 years ago
Patch set 2 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/32360
Updated by Gerrit Code Review over 9 years ago
Patch set 3 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/32360
Updated by Tymoteusz Motylewski over 9 years ago
Great contribution! Indeed problems with TZ were quite annoying.
I did not have time yet to go through the patch yet, but just wanted to confirm that with your patch dates and time should Always be stored in UTC, no metter what timezone user is in, and how server is configured.
Updated by Nikola Stojiljković over 9 years ago
- File be_user_date_settings.png be_user_date_settings.png added
- File sample_fields_with_timezone.png sample_fields_with_timezone.png added
OK, so this is a big change, lets start with the concepts.
Currently, TYPO3 stores dates (and datetimes) from TCEforms using web server's PHP timezone setting. Quite extensive description is written here: https://forge.typo3.org/issues/32081.
This is wrong for a lot of reasons:- Using data on servers with different timezone will yield wrong results,
- TCEforms is saving in server's timezone, while scheduler tasks are saved in UTC (which is correct way),
- Examine the following case: server's system timezone is America/Los_Angeles, php.ini timezone is Europe/Berlin. Calling PHP time() function will return current timestamp in UTC timezone.
So in order to not break the existing installations there's a new setting $GLOBALS['TYPO3_CONF_VARS']['SYS']['storeDatesInUTC'] which defaults to FALSE. It is recommended to set this to TRUE on all new installations.
There's a new tab in the user settings:
Should be self-explanatory. The default setting is to leave the timezone to the client's browser (which will fallback to OS settings).
If you select to show timezone, it will be displayed by default as UTC+XX:00. The setting respects the daylight savings time, so this is for example correct display for CET/CEST:
Behind the scenes, there was a huge rewrite of the jsfunc.evalfield.js. I simply couldn't read through the old code. I wanted to use prototype.js OOP setup, but since there's an intention to drop prototype.js, I introduced similar Class object (with support for defining true JavaScript singletons). The new FieldEvaluation.Utility is what used to be evalFunc, while evalFunc_dummy is now FieldEvaluation.EvaluationSettings. Most of the transformations are actually copy'n'paste. Except for the alpha/alphanum* evaluations which are now done using RegExp and date/datetime fields which are now using moment.js.
The part which uses moment.js assume that moment.js is already loaded using require.js (which will not be the case on the first execution of evalFunc). I opted not to go too much in refactoring here, but could do it if needed... The changed tceforms.js loads rsvp.js and moment.js and formats the dates according to user settings. It expects that the user settings are already in JS memory (see BackendUserAuthentication ->getDateSettings). If not, it will load them using ExtDirect. So there might be a short period of time when the date field will have blank value (noticable in IE8 only from my tests).
Updated by Nikola Stojiljković over 9 years ago
Updated by Nikola Stojiljković over 9 years ago
One more note: date.js has better support in parsing natural language date strings, but moment.js plays much nicer with timezones, so opted for it instead. That's why I added date input field placeholders. Moment.js will parse standard (and ISO) date formats as well as a fallback...
Anyway, if you guys and gals feel like we need natural language processing, we can add date.js as well :) My impression is that it would be too much...
Updated by Markus Klein over 9 years ago
Thanks for your awesome work. I will try to read through the code asap.
One question: I assume the lib supports half hour and quarter hour timezones as well, but does it support DST for all variants that are out there in the wild?
(I guess you know that funny video about time(-zone) programming)
Updated by Gerrit Code Review over 9 years ago
Patch set 4 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/32360
Updated by Nikola Stojiljković over 9 years ago
- File sample_kolkata.png sample_kolkata.png added
- File sample_belgrade.png sample_belgrade.png added
Markus Klein wrote:
Yes, here are the screenshots of the same fields:Thanks for your awesome work. I will try to read through the code asap.
One question: I assume the lib supports half hour and quarter hour timezones as well, but does it support DST for all variants that are out there in the wild?
(I guess you know that funny video about time(-zone) programming)
- Europe/Belgrade timezone
- Asia/Kolkata timezone
- Pacific/Chatham:
Updated by Nikola Stojiljković over 9 years ago
- File sample_chatam.png sample_chatam.png added
Updated by Markus Klein over 9 years ago
As I said, I assumed that, but the question is about daylight saving time.
"most" countries have some certain dates where the clock is changed, but there is an awefull list of exceptions.
Check that video:
http://www.youtube.com/watch?v=-5wpm-gesOY
So maybe we should do the DST setting as a manual setting for the user.
Updated by Nikola Stojiljković over 9 years ago
Markus Klein wrote:
As I said, I assumed that, but the question is about daylight saving time.
"most" countries have some certain dates where the clock is changed, but there is an awefull list of exceptions.Check that video:
http://www.youtube.com/watch?v=-5wpm-gesOYSo maybe we should do the DST setting as a manual setting for the user.
If you select a timezone (based on the usual list), it already defines if DST is in place or not and when (as you can see in the sample above for CET/CEST in Europe/Belgrade timezone). Adding a DST setting will just confuse a regular BE user. Furthermore, moment.js doesn't have a documented way of retrieving DST dates per timezone and I would refrain from hacking through it in order to allow easier upgrades in future.
Updated by Markus Klein over 9 years ago
Ok, I'm confused now.
How does moment.js implement DST?
According to you screenshots it deducts the DST from the timezone, but that is plain wrong. (as explained in the video, DST switching highly depends on the country/location)
I agree we shouldn't hack that on our own, I propose to disable DST handling completely (if that is possible).
Updated by Nikola Stojiljković over 9 years ago
Markus Klein wrote:
Ok, I'm confused now.
How does moment.js implement DST?
According to you screenshots it deducts the DST from the timezone, but that is plain wrong. (as explained in the video, DST switching highly depends on the country/location)
Yes, and selecting a timezone means selecting a location. Please take a look at the new user BE settings. This list is used: http://php.net/manual/en/timezones.php.
I agree we shouldn't hack that on our own, I propose to disable DST handling completely (if that is possible).
Moment.js automatically handles DST, the same way as PHP does.
There might be some confusion regarding this. Selecting a timezone means selecting a location - that defines DST start/end dates per year. When displaying a timezone in the field, you really don't need to see the location but time zone in format (UTC+XX:XX). Moment.js even thrown out the support for displaying timezone in the short format (like CET, CEST, PDT,...) since the browsers do not implement it consistently.
Updated by Nikola Stojiljković over 9 years ago
One more note, I included the moment.js timezone library with data for all years (http://momentjs.com/timezone/).
Updated by Tymoteusz Motylewski over 9 years ago
So in order to not break the existing installations there's a new setting $GLOBALS['TYPO3_CONF_VARS']['SYS']['storeDatesInUTC'] which defaults to FALSE. It is recommended to set this to TRUE on all new installations.
Can't we make it correct (UTC) by default? Especially as there are already inconsistencies in the core. This patch is targeting 6.3 anyway. In the separate patchset we could add a migration wizard to recompute the dates when sb is updating the installation.
I'm opting in for storing dates in UTC, as most people especially newcommers will forget to change this setting.
Updated by Markus Klein over 9 years ago
I think the patch already has this by default for new installations.
Updated by Stefan Neufeind over 9 years ago
Just a side-note: When we have it "correct" in TYPO3-core, let's then look what Extbase stores and reads. I just had a nightmare with Extbase not storing DateTime-objects correctly - and when we have a defined way how to store it in DB, we can look at that part as well.
Updated by Gerrit Code Review over 9 years ago
Patch set 5 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/32360
Updated by Mathias Schreiber over 9 years ago
- Target version changed from 7.0 to 7.1 (Cleanup)
Updated by Tymoteusz Motylewski about 9 years ago
- Assignee changed from Nikola Stojiljković to Tymoteusz Motylewski
- Sprint Focus set to On Location Sprint
Updated by Tymoteusz Motylewski about 9 years ago
- Status changed from Under Review to In Progress
Updated by Gerrit Code Review about 9 years ago
- Status changed from In Progress to Under Review
Patch set 6 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/32360
Updated by Gerrit Code Review about 9 years ago
Patch set 7 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/32360
Updated by Gerrit Code Review about 9 years ago
Patch set 8 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/32360
Updated by Gerrit Code Review about 9 years ago
Patch set 9 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/32360
Updated by Benni Mack almost 9 years ago
- Target version changed from 7.1 (Cleanup) to 7.4 (Backend)
Updated by Susanne Moog over 8 years ago
- Target version changed from 7.4 (Backend) to 7.5
Updated by Anja Leichsenring over 8 years ago
- Sprint Focus changed from On Location Sprint to Remote Sprint
Updated by Tymoteusz Motylewski over 8 years ago
- Assignee deleted (
Tymoteusz Motylewski)
2 good news:
- with forengine refactoring, and removal of jsfunc.evalfield.js implementation of this feature will be easier and cleaner
- the last blocker (timezone support in datetimepicker) is now also gone.
see the development branch (should be released soon as v5)
https://github.com/Eonasdan/bootstrap-datetimepicker/commits/development
Updated by Tymoteusz Motylewski over 8 years ago
even better:
timeozne support was added to the stable branch of date picker 4.17.37
see http://eonasdan.github.io/bootstrap-datetimepicker/Changelog/
Updated by Gerrit Code Review over 8 years ago
Patch set 10 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/32360
Updated by Gerrit Code Review over 8 years ago
Patch set 11 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/32360
Updated by Gerrit Code Review over 8 years ago
Patch set 12 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/32360
Updated by Benni Mack over 8 years ago
- Target version changed from 7.5 to 7 LTS
Updated by Gerrit Code Review over 8 years ago
Patch set 13 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/32360
Updated by Gerrit Code Review over 8 years ago
Patch set 14 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/32360
Updated by Gerrit Code Review over 8 years ago
Patch set 15 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/32360
Updated by Gerrit Code Review over 8 years ago
Patch set 16 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/32360
Updated by Gerrit Code Review over 8 years ago
Patch set 17 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/32360
Updated by Gerrit Code Review almost 8 years ago
Patch set 18 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/32360
Updated by Gerrit Code Review almost 8 years ago
Patch set 19 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/32360
Updated by 
Updated by Riccardo De Contardi almost 7 years ago
- Category set to FormEngine aka TCEforms
Updated by Markus Klein almost 7 years ago
- Status changed from Under Review to On Hold
- Target version deleted (
7 LTS)
This needs to be part of TYPO3 9 as v8 did some first steps into the right direction.
Updated by Markus Klein almost 7 years ago
- Has duplicate Feature #80559: datetime and time handling should always have timezone information added
Updated by Benni Mack over 6 years ago
- Status changed from On Hold to Accepted
- Target version set to Candidate for Major Version
Updated by Susanne Moog over 6 years ago
- Status changed from Accepted to Under Review
Updated by Gerrit Code Review over 6 years ago
Patch set 20 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/32360
Updated by Christian Kuhn over 5 years ago
- Status changed from Under Review to New
resetting this feature back to 'new' since the patch again stalled for a long time in the review system. please feel free to resurrect it if work continues.
Updated by Stefan Neufeind about 2 years ago
Stumbled across this again on a TYPO3 v10-system. DB correctly stores UTC, frontend correctly shows local time (according to phpTimeZone). But backend uses UTC ... which is a pain for editors to work with.
Since a lot of blockers from the past are done, for example we are now using moment.js, is having locale-aware support for the backend something we can take care of? Or are there further renovations pending for FormEngine or so that I'm not aware of, which need to be handled first before continuing this effort here?