T02: Learn about Crowdsourcing by Evaluating Crowdsourcing Platforms

Friday, 26 July 2019, 08:30 – 12:30
Back to Tutorials' Program


Javed Vassilis Khan (short bio)

Eindhoven University of Technology, Netherlands

Konstantinos Papangelis (short bio)

Xi'an Jiaotong-Liverpool University, P.R. China



This tutorial will offer you a meaningful and engaging introduction to what is needed for designing a successful crowdsourcing platform.


Content and benefits:

We have co-developed a deck of cards and the activities that will help you evaluate existing platforms and in that way learn what is needed to design one. The evaluation is based on heuristics drawn from literature on successful online communities. Although the tutorial is geared for evaluation of crowdsourcing platforms practitioners interested in online communities will also benefit. Over the years we have refined the course by offering it in several educational and commercial organizations across the world.
Before the tutorial participants go through: 1) an overview of the heuristics we co-developed in a poster-format for quick review to familiarize themselves with the heuristics 2) a list of platforms and choose one that they would like to work during the course. On the tutorial day we present the rationale and process followed to define those heuristics. Next, we hand over the deck of cards to make the activity easier and engaging. Participants choose one to two cards and perform an evaluation of a platform of their choice. Next, participants share their findings. Finally, we plan a re-design group activity based on the prior findings.


Target Audience:

innovation managers, managers in open innovation, outsourcing managers, community managers, designers of online communities, instructors, practitioners and students of Social Computing.

Bio Sketches of Presenters:

Dr. Javed Vassilis Khan is an assistant professor at the Systemic Change group, Industrial Design Department of Eindhoven University of Technology in the Netherlands. He holds a P.D.Eng. and a PhD from the same department and a MSc in computer Engineering from the University of Patras, in Greece. His current research activities focus on the use of crowdsourcing for design and how to design crowdsourcing platforms. Recent related research projects include CrowdEvaluation.nl, a method to evaluate crowdsourcing platforms and CrowdScouts.com, a platform to index existing platforms. On his PhD he worked on novel mobile and pervasive applications for intra-family communication as well as, on a methodological side, improved the experience sampling method. On his PostDoc he applied some of the methodological findings to support patients suffering from obsessive compulsive disorder. Past research activities included context-ware computing and evaluation of mobile, location-based ads with the help of a mixed-reality setup. He has held in the past research positions at Philips Research, Vodafone R&D and the Computer Technology Institute in Greece.

Dr. Konstantinos Papangelis B.Sc. (Hons.), M.Sc., Ph.D. is an Assistant Professor in Computer Science at Xi’an Jiaotong-Liverpool University (PRC) and an Honorary Lecturer in Computer Science at the University of Liverpool (UK). He is also a fellow of the The Royal Society for the Encouragement of Arts, Manufactures and Commerce (RSA) and is heavily involved the Association for Computing Machinery (ACM). Currently his research focuses on location-based social networks, the physical web, location-based/in-situ crowdsourcing, pervasive games, and novel mobile technologies. He has published his work in multiple journals and conferences including: Interactive Mobile Wearable and Ubiquitous Technologies (IMWUT), Computer-Supported Cooperative Work and Social Computing (CSCW), and Human Factors in Computing Systems (CHI).