Web Mining for Innovation Analysis

Web mining for innovation analysis harnesses the abundance of publicly available web data to generate real-time, granular insights into the activities, networks and outputs of firms and innovation ecosystems. By applying computational methods such as natural language processing and network analysis to website text and link structures, researchers can overcome the limitations of traditional innovation indicators—such as patent counts and survey data—in terms of timeliness, coverage and depth. This approach enables the identification of emerging technological topics, the mapping of inter-firm relationships and the prediction of innovative performance at scale. Web mining methods have been employed to detect latent thematic patterns, track diffusion of general-purpose technologies, and construct innovation indicators that capture the strategic behaviour of firms and clusters across regions and sectors. Despite challenges relating to data quality, representativeness and the need for methodological transparency, web mining offers a cost-effective and reproducible framework for informing policy, guiding investment and enhancing our understanding of the dynamics driving technological change globally.

Research from Nature Portfolio

No recent Nature Portfolio content available.

Research from all publishers

Recent studies have demonstrated how web mining can illuminate the diffusion and adoption pathways of digital technologies. One investigation applied a transformer-based language model to textual data from over a million websites and constructed a hyperlink network of hundreds of thousands of firms, revealing clustered adoption patterns of artificial intelligence and highlighting the roles of regional hotspots, direct knowledge transmission and relational embeddedness in technology diffusion. Another line of work developed deep-learning classifiers trained on labelled web texts alongside traditional survey data to predict product innovator status for large samples of firms; this approach yielded reliable, regionally granular innovation indicators that correlate with patent statistics and survey benchmarks. A further contribution proposed a scalable framework for mapping innovation ecosystems by extracting and analysing textual and relational content from firm websites; a large-scale pilot illustrated how this method can visualise cooperation networks, detect sectoral biases and support ecosystem mapping in contexts ranging from national economies to specific technology domains.

Web Mining for Innovation Analysis publication trend

The graph below shows the total number of articles in web mining for innovation analysis across all publications each year (not limited to Nature Index journals).

Technical terms

Web mining: The process of extracting and analysing large-scale data from websites to uncover patterns, trends and relationships relevant to innovation activities.

Natural language processing (NLP): A set of computational techniques for analysing and deriving meaning from human language text.

Transformer model: A deep-learning architecture that uses self-attention mechanisms to process sequential language data efficiently.

Hyperlink network: A graph representation of websites as nodes connected by hyperlinks, used to study inter-firm relationships and knowledge flows.

Deep learning classifier: A neural network-based model trained to categorise data—such as website texts—into predefined innovation-related classes.

Innovation indicator: A quantitative measure designed to capture the presence, intensity or impact of innovation activity within firms or regions.

References

1. Epidemic effects in the diffusion of emerging digital technologies: evidence from artificial intelligence adoption. Research Policy (2024).
2. Predicting innovative firms using web mining and deep learning. PLOS ONE (2021).
3. Web mining for innovation ecosystem mapping: a framework and a large-scale pilot study. Scientometrics (2020).