TYPO3 Core

Hi Stephane,

really great that you are working on this!

Can you please attach the updated ext_tables_static+adt.sql to this bug? I will then take care of getting it into the core.

greetings from zurich,
Ingmar

Ps.: BTW, would be great if you could set your real-name in your bug tracker profile.

Here is the file generated from the latest docs, you can see which source data was used in comments in the sql file itself.

Hi Stephane,

you also wrote some new PHP code for extracting the docs from the manuals, didn't you?

I think it would be great if that would be managed in the typo3xdev SVN in Sourceforge. What do you think?

cheers
Ingmar

Committed to trunk and TYPO3_4_0.

Thanks again to Stephane Schitter!

cheers
Ingmar

I would like to keep this issue open until we have an automated solution that keeps track of creating this documentation. Otherwise we will end up at the same stage in a year...

Hi Stephane,

can you attach the scripts you used for extracting the new data from the references to this bug please?

cheers
Ingmar

I am reopening this issue, just to make sure that these build scripts don't get forgotten...

Hello Ingmar, Michael and other readers,

I am now ready to continue work on the script after a long period of inactivity. I would like to be ready to provide updated online TSref help for the release of Typo3 4.1 in the next few weeks.

I still have the intention to rewrite my shell scripts into a proper Typo3 extension. However one issue kinds of prevents me to progress with all of this:

I notice in the different core documents we sometimes have duplicate documentation. This means my conversion script will generate duplicate entries in the sql database. I don't have an example at hand, but will try to find one. Basically the current Typo3 code that displays the TSref help does not handle duplicates and will only display one of them.

So should :
1) documents be checked for duplicates and manually remove them at the source, so we don't have the problem
2) my tsconfig help sql database to detect duplicates and find a way or another to differentiate them (different ID number, etc...)
3) the code that displays TSref be updated to allow for displaying more than one single entry per TSref key.

Does anyone have an idea ?

Cheers,
Stephane

Hi Stephane,

first of all, thanks for getting back to this. In fact, I still had it on my todo list to get in touch with you again... :-)

You say you don't have an example for duplicate keys at hand. Would "file" be such an example? It is a subproperty of many other properties, for example the "->filelink" function and the "FILE" and "IMAGE" cObjects. (I didn't check if these really don't work!)

I don't think that duplicates like this should be removed, because they are essential. When looking at the above example, I think that the key must be a combination of property and path, so the SQL table definition will probably have to be changed.

- michael

Actually, the duplicates I thought about are more for example if one document defines a table for properties for : page:RTE and another document also defines a table for properties for page:RTE, I would add two rows into the static_tsconfig_help table, one row for each page:RTE properties definition.

The tsconfig browser would however only display the last row it will find. I would hope that the 'guide' column would help me dissociate when a similar property table is defined in two different manuals, but I could not find where the tsconfig browser actually uses this guide, and how to use that to have it display (in a "merge" way - display all properties from all guides) all the data even from duplicate columns.

Again, in the standard documentation (tsref, tsconfig, api) I could not find an example of such duplication occuring, but as I would like to extend my script to a php module that would generate the tsconfig sql table based on all the extensions a particular TYPO3 installation has (so it would always match all the TS that an admin would possibly want to use on his own installaiton), it could happen that some RTE documents for example add more data to the page:RTE properties, and we would want to display all of them, not just the one from the last parsed document.

I hope this makes sense.

In the meantime, I can provide for an updated ext_tables_static+adt.sql if that is necessary (not sure however if there were lots of changes since 4.0).

Cheers,
Stephane

Ah, now I understand. So forget my last comment, it was written in a wrong context.

Let's think about your problem in a practical example: If PATH1 : PROP2 is defined in a core document, and an extension adds features to the same PATH1, then the best thing would simply be to merge those features, right?

However, if the extension overwrites PATH1 : PROP2 with a modified functionality, then this would usually overwrite the definition from the core. So both should be defined in the SQL file, but the TSConfig help wizard would have to find out if this extension is loaded, and in this case prefer the extensions version instead of the core's version. (I hope you can follow my explanation... ;-))

This means: the key is a combination of path, property and extension name / core document name. PATH1 : PROP2 can be defined twice, but with different extension names.

- michael

I guess the best is to correctly use the following column from the table defined by tsconfig_help:
guide int(11) DEFAULT '0' NOT NULL,
except I don't know where TYPO3 uses this column when displaying the Typoscript help browser. One thing to keep in mind is that the actual help browser is core code of Typo3. The tsconfig_help extension actually only provides the sql data for it.

So what I was looking at was to provide the data, not how it gets displayed (filtered/merged/etc...), as that latter function is part of the TYPO3 core.

It seems that "guide" is not needed by wizard_tsconfig.php. However, I don't like the name for it. What about removing this key and instead add a new one that simply contains the extension name from where the document comes from?

I understand that you don't need to display the data, but we first need to find a good way before you write the providing part, and someone (I?) the display part...

Hello,

I have done further testing with the tsconfig_help extension, and you can find here attached the current version of my code so far. Sorry for hijacking the tsconfig_help key, but as I am not publishing this to TER yet I thought I'd use the same, as it made sense to me.

When you install it you will get an additional be module in the HELP section. To use, open the extension, select the "Update" function and it will load all manuals (manual.sxw) from loaded extensions and if some TSref/TSconfig tables are present in these manuals it will load it into the online TS reference. You can then browse the results with the "Display" function of the module.

To try this extension to its maximum potential you better load and install the latest doc_core_tsref and doc_core_tsconfig extensions as a minimum before running the "Update" function.

WARNING. Before uploading new manuals into the online TSconfig reference it will erase the current contents of the tsconfig_static_help table, so if you wish to keep current contents make sure to back it up first.

Any feedback is appreciated ! For info, this is based on my previous script that generated the ext_tables_static+adt.sql which I still have available if required.

Cheers,
Stephane

P.S. As the extensions manuals are loaded in the order they appear as loaded extensions in localconf.php, we still face the issue of the wizard still displaying only one amongst the possible many entries it finds in the database, issue discussed earlier. My own proposal would be to show all of the versions it finds in the database.

I forgot to mention that this extension depends on the libunzipped extension (this dependency is not yet part of the dependencies within the ext_emconf.php)

Please see the newly attached T3X extension. I have updated my initial version and this one should work much better - better UTF8 handling, better admin interface, better in-code documentation, now correct dependencies.

The only remaining problem is still when there are many times the same table name defined, in either different documents, or in the same document. For example see the silly example of page:RTE table appearing 4 times in the htmlarea manual.sxw...

The published extension here would assign different "guide" values for different extension manuals, but if in one single manual we have 4 times the table redefined, then in the SQL database we would have 4 entries defining the RTE properties.

Michael,

Is there any way to update the Typo3 Wizard that displays the TSRef to load multiple items (when there are duplicates) when it shows the help ? I am happy to change the "guide" column into something more descriptive, I guess the best would even it to be a string with the name of the extension where the TSRef doc comes from. But then we still need Typo3 to merge all duplicates into a single display in the wizard.

Thanks a lot, Stephane! This sounds very promising, I'll check it out asap.

I consider the missing merging functionality as a bug in the tsconfig_wizard and will try to fix it...

Maybe you should already switch the name of the extension (it is not the help itself, it's just the generator) and put it into TER.

There is a document which explains how extension developers have to make their documents look like. I can't figuer its name yet... But it would be good if you add a little mini-howto that contains a link to that one simply...

Thanks so far!
- michael

Hi Stephane, I have today been playing with this extension and it seems to work quite well!

However, I first encountered some problems with the libunzipped extension. Apparently it was not able to unzip doc_core_tsref.

So what I finally did was to rewrite your extension to make use of the em_unzip.php implementation in the new Extension Manager (on board since 4.0) instead of libunzipped.

Feel free to add it to your version.

More suggestions:
- I would be happy if you could get in touch with Andreas Otto to put the extension into typo3xdev. I would be willing to help improving it over SVN...
- As an alternative suggestion, we might think about replacing the version in core sysext with this one. What do you think?

Michael

Michael,

Thanks for the supportive feedback and suggestions. Either way (typo3xdev or core sysext) would be great. What should I call my extension in this case, something like tshelp_rebuild ?

Also, if necessary I can provide an updated ext_tables_static+adt.sql with the latest CoreDocs data for TYPO3 4.1. But now you can also build it yourself from a stripped down installlation with only the basic doc_core_ extensions loaded (with the latest manual.sxw in the correct places).

Cheers,
Stephane

Following some confusion, Michael asked me at T3DD07 if I could look at problems related to this bug. Actually the problem he showed was not listed here, but was a problem nonetheless. After updating the static_tsconfig_help table with a recent version of the TSRef (tested with 4.1), the description of the CONFIG object was empty. After some searching I found the problem: the size of the appdata field is (just) too small. Changing the field to MEDIUMBLOB solves the problem.

As for the duplicate entry problem, I would vote for updating the manuals. I'm willing to look at the rtehmtmlarea manual and suggest some logic for differentiating the various tables that are all labeled Page:RTE.

Hello everybody,

"As for the duplicate entry problem, I would vote for updating the manuals. I'm willing to look at the rtehmtmlarea manual and suggest some logic for differentiating the various tables that are all labeled Page:RTE."
I just checked out the documentation of rtehtmlarea, and I think as well that no further information is given by the sentence written above every table (see http://typo3.org/documentation/document-library/extension-manuals/rtehtmlarea/1.4.4/view/4/4/ for details).
So cleaning up the manual would be the easiest short-term solution I think, but for the long term, it would make sense to at least output a warning when an override takes place.
Francois, did you use Michael's patch?

Greetings,
Sebastian

Yes, I applied Michael's patch. I have just uploaded the version of the script I have been working on. Lots of debug output, but they are all commented out.

Thanks for the upload! Do you still want to implement a warning when a property is overridden? I think that would be pretty helpful in the beginning.

Btw, in the end, should we package the extension as a sysext or publish it in TER?

Greets,
Sebastian

tsconfig_help indeed is already a system extension!

I think in the end it should remain like this, shipping with a static table of core tsref/tsconfig data, but allowing users to update the table with contents of their own installed extensions...

It's true that a warning would be useful anyway. I'll add that and also look at the HTMLArea manual.

I just had a simple idea tonight: everytime the same object string pops up again I increase a counter and append it to the object string. That way - for example - all the page:RTE from the HTMLArea manual are referenced inside the Page TSConfig as [RTE], [RTE (1)], [RTE (2)], etc. That way they don't overwrite each other and you can at least have a complete reference.

Of course, warnings will be useful too...

Hello Francois,
that solution sounds good to me as well! I'll be happy when we have this issue solved in 4.2.

Greets,
Sebastian
PS: I will not be online next week, just to let you know...

I have uploaded a corrected version of the extension. The changes are:

- changed appdata field of table static_tsconfig_help from blob to mediumblob for large tables
- added a counter for each object string so that similar object strings can be differentiated and their tables do not overwrite each other in the online help
- added informational warnings about the above and also about tables that have no object string (so that extension writers can improve their manuals :-)

As it is the extension is functional and several of the problems have been solved. There are however other issues, which I will list in the next note.

Remaining issues:

- it seems impossible to create unique keys for each object strings. Ideally they should be unique, but there's no guarantee that they are, since extension editors can use any object string in their manual. This means I would suggest removing the "Update" feature, which doesn't work yet and probably never will properly. Let's just keep the "Rebuild".
- many strings (inlcuding the latest ones I added (I was in a hurry)) are not localised.
- it is not very clear how tables are pinpointed as containing references compared to other tables. I have observed some strange behaviors and I would need to dig back into the code to see how that selection process works.
- last but not least, many tables are not properly marked with object strings. This has nothing to do with this extension's code, but it reduces its usefulness (a striking example, not a single reference table from tt_news is found, so there's no online help for that extension).

Hey,

I will take care of the localization issues today, and upload this version here again and post it to the core list.
Besides, I'll remove the "update" feature as it does not work...
Do you think that makes sense?

Thanks again for this great feature :-)

Greets,
Sebastian

Cool that you finish polishing that extension.

Thanks go mostly to the original author, who did most of the work but seems unfortunately to have dropped out of the radar.

Thread revived in core-team ML with updated patch/t3x file.

I'm uploading the t3x file here too for reference.

Note: in this new file, the extension is set up to run as a sysext. It will not run locally.