Chilkat HOME .NET Core C# Android™ AutoIt C C# C++ Chilkat2-Python CkPython Classic ASP DataFlex Delphi ActiveX Delphi DLL Go Java Lianja Mono C# Node.js Objective-C PHP ActiveX PHP Extension Perl PowerBuilder PowerShell PureBasic Ruby SQL Server Swift 2 Swift 3,4,5... Tcl Unicode C Unicode C++ VB.NET VBScript Visual Basic 6.0 Visual FoxPro Xojo Plugin
(C#) Getting Started Spidering a SiteThis is a very simple "getting started" example for spidering a web site. As you'll see in future examples, the Chilkat Spider library can be used to crawl the Web. For now, we'll concentrate on spidering a single site.
Chilkat.Spider spider = new Chilkat.Spider(); // The spider object crawls a single web site at a time. As you'll see // in later examples, you can collect outbound links and use them to // crawl the web. For now, we'll simply spider 10 pages of chilkatsoft.com spider.Initialize("www.chilkatsoft.com"); // Add the 1st URL: spider.AddUnspidered("http://www.chilkatsoft.com/"); // Begin crawling the site by calling CrawlNext repeatedly. int i; for (i = 0; i <= 9; i++) { bool success; success = spider.CrawlNext(); if (success == true) { // Show the URL of the page just spidered. Debug.WriteLine(spider.LastUrl); // The HTML is available in the LastHtml property } else { // Did we get an error or are there no more URLs to crawl? if (spider.NumUnspidered == 0) { Debug.WriteLine("No more URLs to spider"); } else { Debug.WriteLine(spider.LastErrorText); } } // Sleep 1 second before spidering the next URL. spider.SleepMs(1000); } |
© 2000-2024 Chilkat Software, Inc. All Rights Reserved.