DataFlex

Avoid URLs Matching Any of a Set of Patterns

See more Spider Examples

Demonstrates how to use "avoid patterns" to prevent spidering any URL that matches a wildcarded pattern. This example avoids URLs containing the substrings "java", "python", or "perl". DataFlex

Use ChilkatAx-win32.pkg

Procedure Test
    Boolean iSuccess
    Handle hoSpider
    Integer i
    String sTemp1
    Integer iTemp1

    Move False To iSuccess

    Get Create (RefClass(cComChilkatSpider)) To hoSpider
    If (Not(IsComObjectCreated(hoSpider))) Begin
        Send CreateComObject of hoSpider
    End

    // The spider object crawls a single web site at a time.  As you'll see
    // in later examples, you can collect outbound links and use them to 
    // crawl the web.  For now, we'll simply spider 10 pages of chilkatsoft.com
    Send ComInitialize To hoSpider "www.chilkatsoft.com"

    // Add the 1st URL:
    Send ComAddUnspidered To hoSpider "http://www.chilkatsoft.com/"

    // Avoid URLs matching these patterns:
    Send ComAddAvoidPattern To hoSpider "*java*"
    Send ComAddAvoidPattern To hoSpider "*python*"
    Send ComAddAvoidPattern To hoSpider "*perl*"

    // Begin crawling the site by calling CrawlNext repeatedly.

    For i From 0 To 9

        Get ComCrawlNext Of hoSpider To iSuccess
        If (iSuccess = True) Begin
            // Show the URL of the page just spidered.
            Get ComLastUrl Of hoSpider To sTemp1
            Showln sTemp1
            // The HTML is available in the LastHtml property
        End
        Else Begin
            // Did we get an error or are there no more URLs to crawl?
            Get ComNumUnspidered Of hoSpider To iTemp1
            If (iTemp1 = 0) Begin
                Showln "No more URLs to spider"
            End
            Else Begin
                Get ComLastErrorText Of hoSpider To sTemp1
                Showln sTemp1
            End

        End

        // Sleep 1 second before spidering the next URL.
        Send ComSleepMs To hoSpider 1000
    Loop



End_Procedure