Processing recursive XQuery over XML streams: The raindrop approach

XML stream applications bring the challenge of efficiently processing queries on sequentially accessible tokenbased data. For efficient processing of queries, we need to ensure that memory usage stays low. This in turn requires that we avoid holding data in the query buffer, by outputting it at the earliest possible time. In this paper, we propose a new class of stream algebra operators for efficient recursive XQuery stream processing. In particular we propose two strategies for implementing structural joins: (a) the just-in-Time structural join strategy efficiently processes joins as long as the input XML substreams are non-recursive and (b) the recursive structural join strategy supports structural joins over recursive XML substreams, however at an added cost of tuple-level ID-comparisons. Both structural join strategies are complemented by an automatadriven invocation mechanism that triggers the execution of the join at the first possible moment upon recognizing the end of the targeted input stream subelement. Further, we design this structural join operator itself to be context-Aware. The operator is capable of at run-Time switching from the efficient just-intime join strategy for elements that are recognized to be nonrecursive to the more powerful id-based structural join strategy for elements that are identified to be recursive. In addition, depending on whether the query is recursive, we will generate the plan with cheaper operators whenever possible. We incorporate the proposed techniques into the Raindrop stream engine. We also report on experimental studies we conducted using ToXgene that show that our techniques brings significant performance improvement.