Chilkat HOME .NET Core C# Android™ AutoIt C C# C++ Chilkat2-Python CkPython Classic ASP DataFlex Delphi ActiveX Delphi DLL Go Java Lianja Mono C# Node.js Objective-C PHP ActiveX PHP Extension Perl PowerBuilder PowerShell PureBasic Ruby SQL Server Swift 2 Swift 3,4,5... Tcl Unicode C Unicode C++ VB.NET VBScript Visual Basic 6.0 Visual FoxPro Xojo Plugin
(PHP ActiveX) Avoid URLs Matching Any of a Set of PatternsDemonstrates how to use "avoid patterns" to prevent spidering any URL that matches a wildcarded pattern. This example avoids URLs containing the substrings "java", "python", or "perl".
<?php // For versions of Chilkat < 10.0.0, use new COM('Chilkat_9_5_0.Chilkat.Spider') $spider = new COM("Chilkat.Spider"); // The spider object crawls a single web site at a time. As you'll see // in later examples, you can collect outbound links and use them to // crawl the web. For now, we'll simply spider 10 pages of chilkatsoft.com $spider->Initialize('www.chilkatsoft.com'); // Add the 1st URL: $spider->AddUnspidered('http://www.chilkatsoft.com/'); // Avoid URLs matching these patterns: $spider->AddAvoidPattern('*java*'); $spider->AddAvoidPattern('*python*'); $spider->AddAvoidPattern('*perl*'); // Begin crawling the site by calling CrawlNext repeatedly. for ($i = 0; $i <= 9; $i++) { $success = $spider->CrawlNext(); if ($success == 1) { // Show the URL of the page just spidered. print $spider->LastUrl . "\n"; // The HTML is available in the LastHtml property } else { // Did we get an error or are there no more URLs to crawl? if ($spider->NumUnspidered == 0) { print 'No more URLs to spider' . "\n"; } else { print $spider->LastErrorText . "\n"; } } // Sleep 1 second before spidering the next URL. $spider->SleepMs(1000); } ?> |
© 2000-2024 Chilkat Software, Inc. All Rights Reserved.