DataFlex

GetBaseDomain

See more Spider Examples

The GetBaseDomain method is a utility function that converts a domain into a "domain base", which is useful for grouping URLs. For example: abc.chilkatsoft.com, xyz.chilkatsoft.com, and blog.chilkatsoft.com all have the same base domain: chilkatsoft.com. Things get more complicated when considering country domains (.au, .uk, .se, .cn, etc.) and government, state, and .us domains. Also, domains such as blogspot, wordpress, etc, are treated specially so that "xyz.blogspot.com" has a base domain of "xyz.blogspot.com". Note: If you find other domains that should be treated similarly to blogspot.com, send a request to support@chilkatsoft.com. DataFlex

Use ChilkatAx-win32.pkg

Procedure Test
    Handle hoSpider
    String sTemp1

    Get Create (RefClass(cComChilkatSpider)) To hoSpider
    If (Not(IsComObjectCreated(hoSpider))) Begin
        Send CreateComObject of hoSpider
    End

    Get ComGetBaseDomain Of hoSpider "www.chilkatsoft.com" To sTemp1
    Showln sTemp1
    Get ComGetBaseDomain Of hoSpider "blog.chilkatsoft.com" To sTemp1
    Showln sTemp1
    Get ComGetBaseDomain Of hoSpider "www.news.com.au" To sTemp1
    Showln sTemp1
    Get ComGetBaseDomain Of hoSpider "blogs.bbc.co.uk" To sTemp1
    Showln sTemp1
    Get ComGetBaseDomain Of hoSpider "xyz.blogspot.com" To sTemp1
    Showln sTemp1
    Get ComGetBaseDomain Of hoSpider "www.heaids.org.za" To sTemp1
    Showln sTemp1
    Get ComGetBaseDomain Of hoSpider "www.hec.gov.pk" To sTemp1
    Showln sTemp1
    Get ComGetBaseDomain Of hoSpider "www.e-mrs.org" To sTemp1
    Showln sTemp1
    Get ComGetBaseDomain Of hoSpider "cra.curtin.edu.au" To sTemp1
    Showln sTemp1

    // Prints: 
    // chilkatsoft.com
    // chilkatsoft.com
    // news.com.au
    // bbc.co.uk
    // xyz.blogspot.com
    // heaids.org.za
    // hec.gov.pk
    // e-mrs.org
    // curtin.edu.a


End_Procedure