Lua Requires Chilkat v9.5.0.80+

Efficiently Process a Huge XML File

See more XML Examples

Demonstrates a technique for processing a huge XML file (can be any size, even many gigabytes). Lua

local chilkat = require("chilkat")

-- This example shows a way to efficiently process a gigantic XML file -- one that may be too large
-- to fit in memory.
--
-- Two types of XML parsers exist: DOM parsers and SAX parsers.

-- A DOM parser is a Document Object Model parser, where the entire XML is loaded into memory
-- and the application has the luxury of interacting with the XML in a convenient, random-access
-- way.  The Chilkat Xml class is a DOM parser.  Because the entire XML is loaded into memory,
-- huge XML files (on the order of gigabytes) are usually not loadable for memory constraints.

-- A SAX parser is such that the XML file is parsed as an input stream.  No DOM exists.
-- Using a SAX parser is generally less palatable than using a DOM parser, for many reasons.
--
-- The technique described here is a hybrid.  It streams the XML file as unstructured text
-- to extract fragments that are individually treated as separate XML documents loaded into
-- the Chilkat Xml parser.
--
-- For example, imagine your XML file is several GBs in size, but has a relatively simple structure, such as:
--
-- <Transactions>
--     <Transaction id="1">
--          ...
--     </Transaction>
--     <Transaction id="2">
--          ...
--     </Transaction>
--     <Transaction id="3">
--          ...
--     </Transaction>
-- ...
-- </Transactions>

-- In the following code, each <Transaction ...> ... </Transaction>
-- is extracted and loaded separately into an Xml object, where it can be manipulated
-- independently.  The entire XML file is never entirely loaded into memory.

local fac = chilkat.FileAccess.new()

local success = fac:open_for_read("qa_data/xml/transactions.xml")
if not success then
    print(fac.last_error_text)
    return
end

local xml = chilkat.Xml.new()
local sb = chilkat.StringBuilder.new()
local firstIteration = true
local retval = 1
local numTransactions = 0

-- The begin marker is "XML tag aware".  If the begin marker begins with "<"
-- and ends with ">", then it is assumed to be an XML tag and it will also match
-- substrings where the ">" can be a whitespace char.
local beginMarker = "<Transaction>"
local endMarker = "</Transaction>"

while retval == 1 do
    sb:clear()
    -- The retval can have the following values:
    -- 0: No more fragments exist.
    -- 1: Captured the next fragment.  The text from beginMarker to endMarker, including the markers, are returned in sb.
    -- -1: Error.
    retval = fac:read_next_fragment(firstIteration, beginMarker, endMarker, "utf-8", sb)
    firstIteration = false

    if retval == 1 then
        numTransactions = numTransactions + 1
        success = xml:load_sb(sb, true)
        -- Your application may now do what it needs with this particular XML fragment...
    end
end

if retval < 0 then
    print(fac.last_error_text)
end

print("numTransactions: " .. numTransactions)