![]() |
Chilkat • HOME • Android™ • AutoIt • C • C# • C++ • Chilkat2-Python • CkPython • Classic ASP • DataFlex • Delphi DLL • Go • Java • Node.js • Objective-C • PHP Extension • Perl • PowerBuilder • PowerShell • PureBasic • Ruby • SQL Server • Swift • Tcl • Unicode C • Unicode C++ • VB.NET • VBScript • Visual Basic 6.0 • Visual FoxPro • Xojo Plugin
(Xojo Plugin) Avoid URLs Matching Any of a Set of PatternsDemonstrates how to use "avoid patterns" to prevent spidering any URL that matches a wildcarded pattern. This example avoids URLs containing the substrings "java", "python", or "perl".
Dim spider As New Chilkat.Spider // The spider object crawls a single web site at a time. As you'll see // in later examples, you can collect outbound links and use them to // crawl the web. For now, we'll simply spider 10 pages of chilkatsoft.com spider.Initialize "www.chilkatsoft.com" // Add the 1st URL: spider.AddUnspidered "http://www.chilkatsoft.com/" // Avoid URLs matching these patterns: spider.AddAvoidPattern "*java*" spider.AddAvoidPattern "*python*" spider.AddAvoidPattern "*perl*" // Begin crawling the site by calling CrawlNext repeatedly. Dim i As Int32 For i = 0 To 9 Dim success As Boolean success = spider.CrawlNext() If (success = True) Then // Show the URL of the page just spidered. System.DebugLog(spider.LastUrl) // The HTML is available in the LastHtml property Else // Did we get an error or are there no more URLs to crawl? If (spider.NumUnspidered = 0) Then System.DebugLog("No more URLs to spider") Else System.DebugLog(spider.LastErrorText) End If End If // Sleep 1 second before spidering the next URL. spider.SleepMs 1000 Next |
© 2000-2025 Chilkat Software, Inc. All Rights Reserved.