AutoIt

A Simple Web Crawler

See more Spider Examples

This demonstrates a very simple web crawler using the Chilkat Spider component. AutoIt

Local $bSuccess = False

$oSpider = ObjCreate("Chilkat.Spider")

$oSeenDomains = ObjCreate("Chilkat.StringArray")
$oSeedUrls = ObjCreate("Chilkat.StringArray")

$oSeenDomains.Unique = True
$oSeedUrls.Unique = True

;  You will need to change the start URL to something else...
$oSeedUrls.Append("http://something.whateverYouWant.com/")

;  Set outbound URL exclude patterns
;  URLs matching any of these patterns will not be added to the 
;  collection of outbound links.
$oSpider.AddAvoidOutboundLinkPattern "*?id=*"
$oSpider.AddAvoidOutboundLinkPattern "*.mypages.*"
$oSpider.AddAvoidOutboundLinkPattern "*.personal.*"
$oSpider.AddAvoidOutboundLinkPattern "*.comcast.*"
$oSpider.AddAvoidOutboundLinkPattern "*.aol.*"
$oSpider.AddAvoidOutboundLinkPattern "*~*"

;  Use a cache so we don't have to re-fetch URLs previously fetched.
$oSpider.CacheDir = "c:/spiderCache/"
$oSpider.FetchFromCache = True
$oSpider.UpdateCache = True

While $oSeedUrls.Count > 0

Local $sUrl = $oSeedUrls.Pop()
    $oSpider.Initialize $sUrl

    ;  Spider 5 URLs of this domain.
    ;  but first, save the base domain in seenDomains
Local $sDomain = $oSpider.GetUrlDomain($sUrl)
    $oSeenDomains.Append($oSpider.GetBaseDomain($sDomain))

Local $i
Local $iNumCrawled = 0
    $bSuccess = True
    While ($bSuccess = True) And ($iNumCrawled < 5)
        $bSuccess = $oSpider.CrawlNext()
        If ($bSuccess = True) Then
            ;  Display the URL we just crawled.
            ConsoleWrite($oSpider.LastUrl & @CRLF)

            ;  If the last URL was retrieved from cache,
            ;  we won't wait.  Otherwise we'll wait 1 second
            ;  before fetching the next URL.
            If ($oSpider.LastFromCache <> True) Then
                $oSpider.SleepMs 1000
            EndIf

            $iNumCrawled = $iNumCrawled + 1
        EndIf

        ;  If CrawlNext fails (no more URLs to crawl in this domain), success is false and the loop exits.
    Wend

    ;  Add the outbound links to seedUrls, except
    ;  for the domains we've already seen.
    For $i = 0 To $oSpider.NumOutboundLinks - 1

        $sUrl = $oSpider.GetOutboundLink($i)
        $sDomain = $oSpider.GetUrlDomain($sUrl)
Local $sBaseDomain = $oSpider.GetBaseDomain($sDomain)
        If ($oSeenDomains.Contains($sBaseDomain) = False) Then
            ;  Don't let our list of seedUrls grow too large.
            If ($oSeedUrls.Count < 1000) Then
                $oSeedUrls.Append($sUrl)
            EndIf

        EndIf

    Next

Wend