Chilkat HOME .NET Core C# Android™ AutoIt C C# C++ Chilkat2-Python CkPython Classic ASP DataFlex Delphi ActiveX Delphi DLL Go Java Lianja Mono C# Node.js Objective-C PHP ActiveX PHP Extension Perl PowerBuilder PowerShell PureBasic Ruby SQL Server Swift 2 Swift 3,4,5... Tcl Unicode C Unicode C++ VB.NET VBScript Visual Basic 6.0 Visual FoxPro Xojo Plugin
(Tcl) Avoid URLs Matching Any of a Set of PatternsDemonstrates how to use "avoid patterns" to prevent spidering any URL that matches a wildcarded pattern. This example avoids URLs containing the substrings "java", "python", or "perl".
load ./chilkat.dll set spider [new_CkSpider] # The spider object crawls a single web site at a time. As you'll see # in later examples, you can collect outbound links and use them to # crawl the web. For now, we'll simply spider 10 pages of chilkatsoft.com CkSpider_Initialize $spider "www.chilkatsoft.com" # Add the 1st URL: CkSpider_AddUnspidered $spider "http://www.chilkatsoft.com/" # Avoid URLs matching these patterns: CkSpider_AddAvoidPattern $spider "*java*" CkSpider_AddAvoidPattern $spider "*python*" CkSpider_AddAvoidPattern $spider "*perl*" # Begin crawling the site by calling CrawlNext repeatedly. for {set i 0} {$i <= 9} {incr i} { set success [CkSpider_CrawlNext $spider] if {$success == 1} then { # Show the URL of the page just spidered. puts [CkSpider_lastUrl $spider] # The HTML is available in the LastHtml property } else { # Did we get an error or are there no more URLs to crawl? if {[CkSpider_get_NumUnspidered $spider] == 0} then { puts "No more URLs to spider" } else { puts [CkSpider_lastErrorText $spider] } } # Sleep 1 second before spidering the next URL. CkSpider_SleepMs $spider 1000 } delete_CkSpider $spider |
© 2000-2024 Chilkat Software, Inc. All Rights Reserved.