Apache Solr for TYPO3 CMS

This one cost me a lot of time and grief and constitutes a serious problem for language-localized UTF8-only TYPO3 installations using SOLR.

I had a 100% utf8 solution in TYPO3 and implemented SOLR. Everything was being indexed correctly (I even checked the Lucene index directly in Luke) and unicode chars were recorded with no problem. However, performing a search query with unicode chars consistently returned zero results both in the SOLR FE extension and using the tomcat SOLR admin query.

After debugging this for more than a day I decided to remove the <charFilter>-definitions from schema.xml, cleared cache and rebuilt the index - and the expected results were now immediately returned when querying with unicode characters.

There seems to be some problem in character translation when the ISO/Latin1 charmap file is referenced in the schema, at least when the site is in UTF8. Unfortunately I could not tell if the bug presents during ADDs to SOLR (did not seem so, but unsure since I also had to rebuild the index for results to show) or during SELECTs from SOLR (seems to happen here, since adding a facet which used unicode chars also returns zero results even when the original query had plenty of results).

There was no error given in the tomcat logs and/or devlog in TYPO3.

I did not check the charmap file itself for problems.

I have not tested this on Latin1-encoded sites or Latin1 encoded Lucene indexe contents.

• remove the <charFilter> definitions from solr/conf/schema.xml and rebuild index.

• perhaps adding a condition to the SOLR query request to disallow character translation if renderCharset 'utf-8' (not sure if SOLR can support this...)
• perhaps detecting if site has renderCharset 'utf-8' when running install.sh or adding an option parameter or simple dialog to the install.sh script, prompting the user to select site encoding