Mining Stack Exchange to Discover Patterns of Global Crowdsourcing

Himesha Prabhakara Wijekoon

Master's thesis

Abstract:

Among many popular crowdsourcing platforms, the Question & Answer website Stack Overflow in Stack Exchange Network is used daily to share knowledge globally by millions of software professionals. Therefore Stack Overflow data can reveal important patterns in global crowdsourcing beneficial for software industry. The aim of this study was to perform data mining on Stack Exchange data, to discover some of these patterns. Main focus of this research was to analyse the global user distribution and contribution, analyse contribution related to user age, classify users with regard to their involvement and identify popular topics with trends. Big data analytic techniques were used for data mining activities using Apache Spark with Python language. Oracle Data Visualization Desktop and scikit-learn python library were used for visualization. The results show that although majority of the users are from USA and India, the average contribution is higher in European countries. Key findings also reveal that younger people ask more questions than older people, while vice versa for answers. Further, users could be classified as one-timers, question askers and answer providers. Popularity and trends of different programming languages, databases and frameworks are also presented.

Keywords

Stack Overflow, Data Mining, Big Data Analytics, Crowdsourcing, Pattern Discovery, Data Visualization, Apache Spark, Oracle Data Visualization Desktop, Software Engineering, Stack Exchange

Language used: English

Date on which the thesis was submitted / produced: 31. 3. 2018

Thesis defence

• Supervisor: doc. Ing. Vojtěch Merunka, Ph.D.

Czech University of Life Sciences Prague

Faculty of Economics and Management

Master programme / field:
Systems Engineering and Informatics / Informatics