PureBasic

Fetch robots.txt for a Site

See more Spider Examples

The Chilkat Spider library is robots.txt compliant. It automatically fetches a site's robots.txt file and adheres to it. It will not download pages denied by robots.txt. Pages excluded by robots.txt will not appear in the Spider's "unspidered" list. This example shows how to explicitly download and review the robots.txt for a given site. PureBasic

IncludeFile "CkSpider.pb"

Procedure ChilkatExample()

    spider.i = CkSpider::ckCreate()
    If spider.i = 0
        Debug "Failed to create object."
        ProcedureReturn
    EndIf

    CkSpider::ckInitialize(spider,"www.chilkatsoft.com")

    robotsText.s
    robotsText = CkSpider::ckFetchRobotsText(spider)

    Debug robotsText


    CkSpider::ckDispose(spider)


    ProcedureReturn
EndProcedure