Benutzer:IHEARu/iHEARu-PLAY
The research project iHEARu (Intelligent systems' Holistic Evolving Analysis of Real-life Universal speaker characteristics) is a human-based computation game for crowdsourced database collection and data labelling. It is being developed and run as part of a collaboration of the Chair for Complex and Intelligent Systems at the University of Passau and the Institute for Human-Machine Communication at the Technische Universität München. The main purpose of the game is to motivate people to voluntarily annotate data in order to support Supervised learning techniques for computational paralinguistics [1].
It is an alternative for most crowdsourcing services that rely on click-workers which are being paid a rather low compensation for their work efforts and therefor are often unmotivated which leads to poor results for some workers, increasing overall costs and reducing reliability.
Concept
In order to motivate people to annotate data, the system behaves like a game, so that the user spark their interest and not just in playing the game once but rather to play on a regular basis. The players should have fun while playing since this will be an important intrinsic motivator due to the lack of extrinsic monetary compensation. Therefor, iHEARu-PLAY will
- motivate people with the help of gamification,
- reward people with virtual goods, such as scores and badges,
- optionally hold sweepstakes with non-virtual prizes.
Feedback needs to be immediate, otherwise the player does not know what effect his action caused and he will quickly become distracted and lose focus on the task at hand [2]. Instead of just selecting a random answer, users should focus and think before labelling any audio file, because a good answer will yield a better reward. In order for this to work, it is necessary to identity and distinguish between good, neutral, and bad answers: Once the player submits the answer, it will be evaluated and scored. Depending on the type of question, different complex formulas are used to score the answer itself. The submitted answer is added to the list of answers before the evaluation. Thus, players answering a question that no one else has before, are not at a disadvantage as it will result in a good or neutral grading. There are formulas in detail for the three possible questions.
Single-select questions
The user has to select a single answer from a list of choices. After the user submitted his or her answer, the system will calculate the mode (the answer chosen by the most players) and also calculates the distribution over all choices. The evaluation involves three steps, independent from one another: 1) If the answer is equal to the mode, its score will increase by 3. 2) If the answer received more than 50% of the total votes from all choices, its score will be increase by 3. 3) If the answer received more votes (in percent) as expected by a uniform distribution (e.g. for 4 possible choices, the answers total percent have to be greater than 25%) its score will increase by 1. A final score in the range of [1, 3] will result in a neutral grading, lower and higher will be bad and good, respectively.
Multiple-select questions
The user can choose multiple answers from a list of choices, however at least one item has to be selected with a maximum of all available choices. Since the total number of votes over all possible items, can be much higher than the number of players that answered that question, two different percentage values for each item have to be calculated. The percentage of players that selected this item is called the ‘global percentage’ and the number of votes received, relative to the other choices, the ‘local percentage’. Each selected choice is then evaluated in two independent steps: 1) If the local percentage is higher then expected in an uniform distribution its score is increased by 1, if it is lower or equal its score will decrease by 1. 2) If the global percentage is higher than 50% (that means, more than half of the players selected this item as part of their answer), its score will increase by 3, otherwise it will decrease by 3. The same two steps will be applied to all items not selected by the user, however the scoring will be different as before. If the final score is equal to 0 it is considered a neutral answer. A positive value is considered good and a negative one bad, respectively.
Likert-Scala questions
The user has to select an integer value from a 5-point Likert-Scala, although other values (including negative numbers) are supported by the system as well. For each question the median, mean and mode will be calculated from all previously handed in answers. The evaluation involves three steps: 1) If the answer is equal to the median its score will increase by 2. 2) If the answer is equal to the (rounded) average, its score will increase by 1. 3) If the answer is equal to the mode, its score will increase by 3. A score in the range of [1, 3] is regarded neutral, higher or lower values good or bad, respectively. Before a player can receive the points based on his answer, the graded answer will be evaluated against the other modules of the gamification concept.
Gamification
[[Datei:|miniatur|rechts|Examples of current available badges]]
Rank System
Each player starts with the rank ‘Beginner’. Upon reaching a certain amount of points players will advance to the rank ‘Expert’, which will earn them the ability to access other players profiles, view new leaderboards and earn points even while being offline. However the aim for each player should be to obtain the currently highest rank of ‘Master’ which will be awarded upon reaching even more points. At rank Master, players have the same rules and permissions as the rank of Expert, except the Master leaderboards which are of course exclusive to players with the rank of Master.
Leaderboards
iHEARu-PLAY features several leaderboards based on the following criteria: amount of points accumulated, number of questions answered, databases completed and number of badges awarded. Additionally each of these leaderboards will be available for the individual ranks (Beginner, Expert, and Master rank), resulting in twelve leaderboards. Only the leaderboards for the current rank of the player will be visible though.
Points and Multipliers
As described, each answer given by the player is analyzed and rated good, neutral or bad. This will set the base points to 20, 10 or 5. The final points awarded to the player are based on the given formula: points = base(Mr + Mc + Tu/100 + Md), where: base: base points, Mr: rank multiplier, Mc: users’ current multiplier, Tu: users’ trustability score, Md: databases’ multiplier. Points have to be calculated with floating points numbers but will be rounded to an integer value before assigned to the player. The multipliers are based on the current rank and can further be adjusted through bonus items. Additionally, the users current multiplier will reset every day at midnight and will increase by 0.1 for the first ten answers given on the current day to a maximum of 1.0 (excluding modifications from other bonuses).
Points from other Players
Players with the rank of Expert or higher can earn points through the actions of other players. If you are the first Expert to chose a unique answer to a question, the system will make a special note of it. If other Expert players chose the same answer, thus reinforcing your decision, you will receive ten bonus points each time. These points will even accumulate while the original Expert is offline and will then be assigned to his or her user profile on this next login. (S)He will not be able to see from which individual players (s)he received those points though in order to suppress collusion between players.
User Trustability
Each user starts with a trustability of 100%. Single- and multiple-select questions can contain control answers which will result in a 50% decrease the users trustability if (s)he selects any of these. If the user does answer a control question correctly, his or her trustability will increase by 10%. Trustability does affect points awarded to the user. At the same time, the amount of wrong control answers will be tracked and this value could be used to detect players who do not read the question and just select a random answer.
Bonus Items
Every time the player submits an answer, (s)he has a chance to find a bonus item. The probability starts at 1% and increase by another 1% for every answer submitted until a bonus item has been awarded. After this it will reset to the base of 1%. Currently implemented bonus items contain bonus points, modifier modifications and bonus probability modifiers. Upon rewarding a bonus item, one item is randomly selected from a list of all available bonus items, based on its own probability value. Using a weighted random sampling algorithm [3] is helping to give each bonus item its own probability value. Aside from bonus points, other bonus items have a dynamic durability and are active only a certain count of questions. Multiple bonuses can be active at the same time though.
Badges
Badges are awarded to players who fulfill special requirements unique to each badge. Examples include number of questions answered, number of times logged in, using the platform at special times (e.g. during the night) or anything else. Every time the player interacts with iHEARu-PLAY, either by logging in or by submitting an answer, all not yet obtained badges are checked to verify if the requirements have been fulfilled. Players are able to view the profile of other players and see their acquired badges.
Voice Analyse Application
Voice Analyse Application (VoiLA) is a speech classification tool integrated in iHEARu-PLAY, developed for analyzing and categorizing speech data with the help of a so called classifier according to different aspects including the gender, or the emotional state of the speaker. The classifiers are under development and therefore these results may not always be accurate. The success of these techniques depends highly on the quantity and quality of labelled data, since the created models learn to classify based on annotated data. The annotated data will be used to train this classifier via machine learning algorithms to automate a better Speech recognition of certain criteria of new data files without human intervention. This allows a delivery of better results in VoiLA.
Implementation
iHEARu-PLAY uses HTML 5 in order to run on any modern device/browser without utilizing additional plug-ins. For the technical realization, the free and open-source highlevel Python Web framework Django was chosen. It comes with much built in functionality and can further be extended through 3rd party modules, called apps. Since it runs on Python, Django is essentially platform independent and can be deployed to Linux, Windows or OSX servers. Thanks to its template engine, the layout and design of the web application can quickly be replaced and also allows easy localization. The current prototype uses a free HTML 5 theme which is also responsive and thus will display correctly on a mobile device with a reduced screen size as well.
Once the player selects an available database, the browser will start playing back a random audio file from that database. For the playback, the default HTML 5 audio controls are used, however the controls are hidden so that the user can not seek during playback. A toggle button for pausing and playing is available to the user instead. Using jQuery and Javascript supports hiding the list of possible choices until the audio file has played past the half of its total length. This is to thwart bots and randomly clicking players by forcing them to listen to at least the first half of the audio recording. Only after the user selected an answer the submit button will be visible. All forms are automatically protected from Cross-site-request-forgery thanks to Django’s csrf token.
The current implementation allows for nearly unlimited badge conditions to be created, without having to rely on a complex and heavyweight rule engine. As Django uses an object-relational mapping (ORM), all database tables and attributes are defined as Python classes. This allows rapid development, and changes to the current database model, without losing data. When an answer is submitted, not only the answer tuple (consisting of user : userID, question : questionID, audioData : audioDataID, timestamp : DateField, body : string) is saved, but also the total count for this answer in a separate table is updated. This allows quick look-ups on the count of answers for a specific question, which is needed to evaluate the rewarding grade for a new submitted answer. The system expects the audio files to be stored somehow (e.g. FTP) on the local (or net mounted) hard-drive. When the administrator wants to add a new database, (s)he will just create it in the admin interface and point it to the local path where the audio files are being stored. iHEARu-PLAY will automatically search the given path for all files and create audioData entries in the database. In a second step, it will create entries in the AudioQuestPair table based on the questions enabled for this database by the administrator. Thus it will create an entry for every possible combination of audio data and question.
References
External links
- iHEARu-PLAY
- iHEARu-Project
- S. Hantke, A. Batliner, and B. Schuller: "Ethics for Crowdsourced Corpus Collection, Data Annotation and its Application in the Web-based Game iHEARu-PLAY."
- S. Hantke, F. Eyben, T. Appel, and B. Schuller: "iHEARu-PLAY: Introducing a game for crowdsourced data collection for affective computing."
Kategorie:Crowdsourcing Category:Human-based computation Category:Data collection Category:Human-based computation games</nowiki>
