From Navicrawler to HyBro: a brief history of webcrawlers for social sciences

Benjamin Ooghe-Tabanou (1)

1 médialab - médialab (Sciences Po)

The World Wide Web’s original design as a vast open documentary space built around the concept of hypertext made it a fantastic research field to study networks of actors of a specific field or controversy and analyse their connectivity. Navicrawler, IssueCrawler, Hyphe... Over the past 15 years, a variety of web crawling tools, most often free and open source, have been developped by or for social sciences research labs across the world. They provide means to engage with the web as a research field or to teach students what the WWW is beyond Google or Facebook’s interfaces. We will first present an overview of this history of open source web crawling tools built for research, teaching or data journalism purposes. Then we will propose a short demonstration of the latest version of médialab's HyBro, aka Hyphe-Browser, a tool built to let users benefit from automated web crawling as well as in situ web browsing and categorizing. Its friendly user interface allows a variety of publics to engage with web crawling, including non-experts like students, social science scholars, and activists.

Domaines: Web; Sciences de l'information et de la communication

Benjamin Ooghe-Tabanou. From Navicrawler to HyBro: a brief history of webcrawlers for social sciences. FOSDEM 2021 - Open Research Tools and Technologies devroom, Feb 2021, Brussels, Belgium. ⟨hal-03904098⟩