Hidden-web Induced by Client-side Scripting: An Empirical Study (ICWE 2013)

Zahra Behfarshad and Ali Mesbah. Hidden-web induced by client-side scripting: An empirical study. In Proceedings of the International Conference on Web Engineering (ICWE), 15 pages. Springer, 2013.

Abstract: Client-side JavaScript is increasingly used for enhancing web
application functionality, interactivity, and responsiveness. Through the
execution of JavaScript code in browsers, the DOM tree representing a
webpage at runtime, can be incrementally updated without requiring a
URL change. This dynamically updated content is hidden from general
search engines. In this paper, we present the rst empirical study on measuring and characterizing the hidden-web induced as a result of client-side JavaScript execution. Our study reveals that this type of hidden-web
content is prevalent in online web applications today: from the 500 websites we analyzed, 95% contain client-side hidden-web content; On those websites that contain client-side hidden-web content, (1) on average, 62%
of the web states are hidden, (2) per hidden state, there is an average of
19 kilobytes of data that is hidden from which 0.6 kilobytes contain textual content, (3) the DIV element is the most common clickable element
used (61%) to initiate this type of hidden-web state transition, and (4)
on average 25 minutes is required to dynamically crawl 50 DOM states.
Further, our study indicates that there is a correlation between DOM
tree size and hidden-web content, but no correlation exists between the
amount of JavaScript code and client-side hidden-web.