Today world's dependence on the Internet and the emerging of Web 2.0 applications is significantly increasing the requirement of web robots crawling the sites to support services and technologies. Regardless of the advantages of robots, they may occupy the bandwidth and reduce the performance of web servers. Despite a variety of researches, there is no accurate method for classifying huge data ...

industrial-scale r&d it projects depend on many sub-technologies which need to be understood and have their risks analysed before the project can begin for their success. when planning such an industrial-scale project, the list of technologies and the associations of these technologies with each other is often complex and form a network. discovery of this network of technologies is time con...

Several studies examined various features on how to most effectively detect web robots. Based on an insight that most web robots, regardless of specifics, would exhibit focused and therefore monotonous behavior, this paper proposes that monitoring the rate of behavioral change is highly effective in detecting sessions initiated by web robots. Empirical evaluation performed on more than one bill...

The massive and automated access to Web resources through robots has made it essential for Web service providers to make some conclusion about whether the "user" is a human or a robot. A Human Interaction Proof (HIP) like Completely Automated Public Turing test to tell Computers and Humans Apart (CAPTCHA) offers a way to make such a distinction. CAPTCHA is a reverse Turing test used by Web serv...

The emergence of the World Wide Web provides a unique opportunity to connect robots to the Internet, enabling people all over the World to control them and monitor their operation. This paper describes a series of Web interfaces, designed to remotely operate mobile robots in public places through the Web. The design of these interfaces specifically addresses issues such as low bandwidth of Inte...

Understanding the nature and the characteristics of Web robots is an essential step to analyze their impact on caching. Using a multi-layer hierarchical workload model, this paper presents a characterization of the workload generated by autonomous agents and robots. This characterization focuses on the statistical properties of the arrival process and on the robot behavior graph model. A set of...

The Web robots are programs that automatically traverse through networks. Currently, their most visible and familiar application is to provide indices for search engines, such as Lycos and Alta Vista, and semi-automatically maintained topic references or subject directories. In this article, we survey the state-of-art of the Web robots, and the search engines that utilize the results of robot s...

Robots Exclusion standard [4] is a de-facto standard that is used to inform the crawlers, spiders or web robots about the disallowed sections of a web server. Since its inception in 1994, the robots exclusion standard has been extensively used. In this paper, we present our results of the statistical analysis of the usage of robots exclusion standard. Based on the results obtained, we propose t...