Design of an algorithm for crawling web articles focusing on the given topic

Abdulbasit Ayinde Aliyu

Bakalářská práce

Abstract:

This thesis aimed to design and implement an algorithm that crawls web articles focusing on the agrarian sector. The theoretical part of the work focused on exploring the Internet, World Wide Web, HTML5, and web crawling algorithms, with a particular emphasis on Focused Crawler and Incremental Crawler. The partial objectives of the thesis were to create a review of similar solutions, select a suitable technology for implementing the algorithm, and validate the algorithm in the context of English-language websites. The algorithm faced some challenges, such as dynamic content, duplicate content, errors and exceptions, and managing large amounts of data, but was addressed through appropriate methods. The findings have the potential to guide future research in this area and improve the quality and accuracy of web crawling algorithms.

Keywords

Web, Webbot, Algorithm, Crawler, Internet, www, Web Crawling, HTML

Jazyk práce: angličtina

Datum vytvoření / odevzdání či podání práce: 14. 3. 2023

Obhajoba závěrečné práce

• Vedoucí: Ing. Jan Masner, Ph.D.

Česká zemědělská univerzita v Praze

Provozně ekonomická fakulta

Bakalářský studijní program:
Informatics