T3DD11-Workshop: The 36 chambers of TYPO3 Logging¶

There was a workshop about TYPO3 Logging at the TYPO3 Developer Days 2011 in Sursee. About 20 participants worked together for half a day to analyze, understand and improve Logging in TYPO3.
The workshop was very successful and it turned out to be the kickoff for the first core project for the release of TYPO3 version 4.7.

Workshop participants¶

• Dominique Feyer
• Steffen Gebert
• Steffen Müller
• Sonja Scholz
• Alexander Wahl
• Please add your name here

Why Logging Improvements? About the problem we face with Logging in TYPO3¶

Excerpt from the Workshop abstract:

Logging is a fundamental feature of any IT system. TYPO3 has various kinds of logging features, whereas it's characteristics are determined by different factors: User roles, logging purpose, targets, APIs etc. The logging features in TYPO3 have grown over the years and seem to be scattered all over the core and various extensions. Therefore it is hard even for the experienced users to understand and keep track of it.

Results from the workshop¶

This is a collection of our work, which should be moved to separate pages for continuing to work on it.

Analysis of the actual Logging implementation¶

We investigated the most common Logging possibilities and discussed their pros and cons:

BE LOG¶

API

+ THERE IS NO API, only plain SQL queries
+ Configurable log levels (systemLogLevel in Install Tool)

- Mixing information target groups
- no way to configure targets (mail, etc)
- no localization
- no event triggers (no push, just pull)

UI

+ Filters
+ Overview of TYPO3

- Usability of the Backend module
- Insufficient information / Helpless information
- Separation/Right Management

WEB > INFO > LOG¶

It displays parts of BE log, but with a very limited feature set.

- Filters
- No configuration or not known to us.

DevLog¶

API (t3lib_div::devLog())

+ Way to give additional information in array
+ Add own view, like done with fire_devlog:http://forge.typo3.org/projects/extension-fire_devlog Extension for firebug
+ Works without xDebug
+ Easy to use for developers

- Not streamlined with rest of the logging API
- Path to class not logged/shown
- DLOG constant should be part of devLog() and is confusing

UI (devlog Extension - since this is the most popular one)

+ Filter by extension

- Debug array information is shown inline, bad with big data sets

Deprecation Log¶

+ Educates developers
+ Helps developers to keep code up to date

- Repeats the same message
- Flooding
- Yet another Logging system

BE Console Log (Debug Console)¶

API

+ Allows debugging of ExtJS in BE
+ Easy to use for BE debugging
+ Separated from Page markup

UI

+ Stays open although you reload

- Stays open although you reload
- Flooding
- Where to configure?

System Log (t3lib_div::sysLog())¶

API

+ allows to log to several targets (file, syslogd, email)
+ no_cache trigger detection

- BE log cannot be chosen as a target

UI

- No UI at all

admPanel¶

Link Validator¶

Query Logging¶

Reports module in the Backend¶

caretaker Extension¶

(not discussed, to be done later)

Challenges we face with Logging in TYPO3¶

• Complexity
• Insufficient documentation: Target group specific tutorials (developer, admins, ...), Guides, Best Practices and documentation how Logging works in general are yet missing
• Information overload
• Insufficient possibilities to filter, order and cluster Logging events
• Difficulties to realize common Logging strategies
• Insufficient, unstructured and confusing configuration options
• 3rd party integration
• Confusion: too many logging APIs which work almost the same way

Aspects of Logging¶

• Logging purpose (Why...?)
• Logging target (Where/Who...?)
• Logging severity (How harmfull is...?)
• Logging implementations (What tool/method...?)
• Logging documentation (How to...?)
• Logging events (What to log?)
• 3rd party Logging systems (How to integrate...?)

Ideas for a user interfaces (UI)¶

(Thomas W., Steffen Müller)

General Requirements:

• Sorting
• Filtering
• Example: Extension devlog (trunk version) uses ExtJS grid view for example
• Personalization of the view
• Save the views
• Export the view (to share)
• Categories to filter by certain tasks/role
• Styling of complex data structures (nested arrays)
• More granular filtering – not only global (like severity e.g.)
• Combinations of filters
• Group log sessions (and allow more than one aspect)
• Differentiate between individual sessions (e.g. the sessions of two BE users)

Ideas for Logging configuration¶

(Thomas W., Steffen Müller)

• Provide default Logging configuration sets for widespread purposes (monitoring in production, development, staging, ...)
• Restructure configuration: * Central Logging section in install tool * Cleanup

Ideas for Logging Messages¶

(Thomas W., Steffen Müller)

• Logging guidelines (LGL): All Logging should follow the same rules: core, extensions. Help people to write good Logging messages
• Cleanup: Apply these rules to existing message

Communicate to the Community¶

(Thomas W., Steffen Müller)

• Motivate and help to improve Logging outside of the core.

Current APIs¶

(Please add your name here)

t3lib_div¶

 1  /**
 2  * Logs message to the system log.
 3  * This should be implemented around the source code, including the Core and both frontend and backend, logging serious errors.
 4  * If you want to implement the sysLog in your applications, simply add lines like:
 5  *         t3lib_div::sysLog('[write message in English here]', 'extension_key', 'severity');
 6  *
 7  * @param    string        Message (in English).
 8  * @param    string        Extension key (from which extension you are calling the log) or "Core" 
 9  * @param    integer        Severity: 0 is info, 1 is notice, 2 is warning, 3 is error, 4 is fatal error
10  * @return    void
11  */
12 public static function sysLog($msg, $extKey, $severity = 0) {

1 /**
2 * Writes a message to the deprecation log.
3 *
4 * @param string Message (in English).
5 * @return void
6 */
7 public static function deprecationLog($msg) {

1 /**
2 * Logs a call to a deprecated function.
3 * The log message will be taken from the annotation. 
4 * @return void 
5 */
6 public static function logDeprecatedFunction() {

 1 /**
 2 * Logs message to the development log.
 3 * This should be implemented around the source code, both frontend and backend, logging everything from the flow through an application, messages, results from comparisons to fatal errors. 
 4 * The result is meant to make sense to developers during development or debugging of a site. 
 5 * The idea is that this function is only a wrapper for external extensions which can set a hook which will be allowed to handle the logging of the information to any format they might wish and with any kind of filter they would like. 
 6 * If you want to implement the devLog in your applications, simply add lines like: 
 7 * if (TYPO3_DLOG) t3lib_div::devLog('[write message in english here]', 'extension key'); 
 8 * 
 9 * @param string Message (in english). 
10 * @param string Extension key (from which extension you are calling the log) 
11 * @param integer Severity: 0 is info, 1 is notice, 2 is warning, 3 is fatal error, -1 is "OK" message 
12 * @param array Additional data you want to pass to the logger. 
13 * @return void 
14 */
15 public static function devLog($msg, $extKey, $severity = 0, $dataVar = FALSE) {

 1 /**
 2 * Converts a one dimensional array to a one line string which can be used for logging or debugging output
 3 * Example: "loginType: FE; refInfo: Array; HTTP_HOST: www.example.org; REMOTE_ADDR: 192.168.1.5; REMOTE_HOST:; security_level:; showHiddenRecords: 0;" 
 4 *
 5 * @param array Data array which should be outputted
 6 * @param mixed List of keys which should be listed in the output string. Pass a comma list or an array. An empty list outputs the whole array.
 7 * @param integer Long string values are shortened to this length. Default: 20
 8 * @return string Output string with key names and their value as string
 9 */
10 public static function arrayToLogString(array $arr, $valueList = array(), $valueLength = 20) {
11     $str = '';
12     if (!is_array($valueList)) {
13         $valueList = self::trimExplode(',', $valueList, 1);
14     }
15     $valListCnt = count($valueList);
16     foreach ($arr as $key => $value) {
17         if (!$valListCnt || in_array($key, $valueList)) {
18             $str .= (string) $key . trim(': ' . self::fixed_lgd_cs(str_replace(LF, '|', (string) $value), $valueLength)) . '; ';
19         }
20     }
21     return $str;
22 }

t3lib_utility_Debug¶

1 /**
2  * Opens a debug message inside a popup window
3  *
4  * @param mixed $debugVariable
5  * @param string $header
6  * @param string $group
7 */
8 public static function debugInPopUpWindow($debugVariable, $header = 'Debug', $group = 'Debug') {

1 public static function debug($var = '', $header = '', $group = 'Debug') {

t3lib_userAuthGroup¶

 1 /**
 2  * Writes an entry in the logfile/table
 3  * Documentation in "TYPO3 Core API" 
 4  *
 5  * @param    integer        Denotes which module that has submitted the entry. See "TYPO3 Core API". Use "4" for extensions.
 6  * @param    integer        Denotes which specific operation that wrote the entry. Use "0" when no sub-categorizing applies
 7  * @param    integer        Flag. 0 = message, 1 = error (user problem), 2 = System Error (which should not happen), 3 = security notice (admin)
 8  * @param    integer        The message number. Specific for each $type and $action. This will make it possible to translate errormessages to other languages
 9  * @param    string        Default text that follows the message (in english!). Possibly translated by identification through type/action/details_nr
10  * @param    array        Data that follows the log. Might be used to carry special information. If an array the first 5 entries (0-4) will be sprintf'ed with the details-text
11  * @param    string        Table name. Special field used by tce_main.php.
12  * @param    integer        Record UID. Special field used by tce_main.php.
13  * @param    integer        Record PID. Special field used by tce_main.php. OBSOLETE
14  * @param    integer        The page_uid (pid) where the event occurred. Used to select log-content for specific pages.
15  * @param    string        Special field used by tce_main.php. NEWid string of newly created records.
16  * @param    integer        Alternative Backend User ID (used for logging login actions where this is not yet known).
17  * @return    integer        Log entry ID.
18  */
19 function writelog($type, $action, $error, $details_nr, $details, $data, $tablename = '', $recuid = '', $recpid = '', $event_pid = -1, $NEWid = '', $userId = 0) {

t3lib_DB¶

1 /**
2  * Debug function: Outputs error if any
3  *
4  * @param    string        Function calling debug()
5  * @param    string        Last query if not last built query
6  * @return    void
7  */
8 function debug($func, $query = '') {

1 /**
2  * Checks if recordset is valid and writes debugging inormation into devLog if not.
3  *
4  * @param    resource    $res    Recordset
5  * @return    boolean    <code>FALSE</code> if recordset is not valid
6  */
7 function debug_check_recordset($res) {

Common parameters.. somehow

• $message
• $source (or category, e.g. extension key)
• $severity
• $additionalInformation (array of infos, dependent on the caller)

Ideas popped up

• logrotation (for file, db)
• one log entry with client information during startup (e.g. UA, remote address - dependent on log level, of course)
• request identifier, to easily spot all entries generated during one request