crawler

Especially the performance improvements and the enhancements for the configuration of the crawler are the most remarkable parts of the new version.

Regarding the performance improvements we've implemented a full multi-process crawling-model for the crawler. This allows to scale up performance of the crawler with your hardware and push through urgent jobs very quick.

The configuration improvements enable a more flexible way to configure your crawl-jobs through configuration-records instead of TSConfig.

Even though all the changes happened with backwards compatibility in our minds so all your old settings should still work as supposed.

Big kudus for all the good work goes to Timo Schmidt and Fabrizio Branca for AOEmedia.