Reconfigurable Web Wrapper Agents for Web Information Integration

In this paper, we presented a tool to exploit online Web data sources using reconfigurable Web wrapper agents. We described how these agents can be rapidly generated and executed based on the script language WNDL and extraction rule generator IEPAD. WNDL is an XML-based language that provides a representation of a Web browsing session. A WNDL script describes how to locate the data, extract the data and combine the data. By executing different WNDL scripts, user can automate virtually all types of Web browsing sessions. We also describe IEPAD, a data extractor based on pattern discovery techniques. IEPAD allows our software agents to automatically discover the extraction rules to extract the contents of a structurally formatted Web page without the need to label a Web page to train a wrapper. With a programming-by-example authoring tool, a user can generate a complete Web wrapper agent by browsing the target Web sites. We have built a variety of applications to demonstrate the feasibility of our approach.