crawler

In some cases the execution of the crawler run (via scheduler) failes with this error:
Execution of task "Crawler Run (crawler)" failed with the following message: PHP Catchable Fatal Error: Argument 1 passed to tx_crawler_lib::buildRequestHeaderArray() must be an array, boolean givenExecution of task "Crawler Run (crawler)" failed with the following message: PHP Catchable Fatal Error: Argument 1 passed to tx_crawler_lib::buildRequestHeaderArray() must be an array, boolean given

To fix this bug it's necessary to modify the function requestUrl in class.tx_crawler_lib.php so it checks if $url is an array:

function requestUrl($originalUrl, $crawlerId, $timeout=2, $recursion=10)    {

        if(!$recursion) return false;
            // Parse URL, checking for scheme:
        $url = parse_url($originalUrl);

        // Bugfix Begin
        if (!is_array($url)) {
            return FALSE;
        }
        // Bugfix End

        if(!in_array($url['scheme'],array('','http','https'))) {
            if (TYPO3_DLOG) t3lib_div::devLog(sprintf('Scheme does not match for url "%s"', $url), 'crawler', 4, array('crawlerId' => $crawlerId));
            return FALSE;
        }

        $reqHeaders = $this->buildRequestHeaderArray($url, $crawlerId);