DataFlex

Fetch robots.txt for a Site

See more Spider Examples

The Chilkat Spider library is robots.txt compliant. It automatically fetches a site's robots.txt file and adheres to it. It will not download pages denied by robots.txt. Pages excluded by robots.txt will not appear in the Spider's "unspidered" list. This example shows how to explicitly download and review the robots.txt for a given site. DataFlex

Use ChilkatAx-win32.pkg

Procedure Test
    Handle hoSpider
    String sRobotsText

    Get Create (RefClass(cComChilkatSpider)) To hoSpider
    If (Not(IsComObjectCreated(hoSpider))) Begin
        Send CreateComObject of hoSpider
    End

    Send ComInitialize To hoSpider "www.chilkatsoft.com"

    Get ComFetchRobotsText Of hoSpider To sRobotsText

    Showln sRobotsText


End_Procedure