IDENTIFYING USER BIAS AS HIDDEN STATE IN A/B TESTING WITH HIDDEN MARKOV MODEL APPROACH

• Main Author: Hartadi Suliman, Andreana
• Format: Final Project
• Language: Indonesia
• Online Access: https://digilib.itb.ac.id/gdl/view/84526
• Institution: Institut Teknologi Bandung

Along with software development, testing continues to be carried out, including A/B Testing. A/B Testing is a testing method that compares two software designs to determine which one is more effective. A commonly used metric is click rate, which is the number of users who click on the target in the design. The initial hypothesis is that variation A (control) has a higher click rate than variation B (experimental). The drawback of A/B Testing is user bias which can lead to inaccurate results. Hidden Markov Model (HMM) is used to detect hidden states that affect A/B Testing results. Implementation of HMM into A/B Testing is done to minimize user bias. HMM, which is commonly used to detect hidden patterns, is implemented to identify user behavior bias in A/B Testing. The HMM algorithm changes the display frequency of the design unequally to test the initial hypothesis and produce tests that are affected by user bias as little as possible. The results show that HMM successfully detects user bias and improves the validity of A/B Testing results. This research provides a new solution to overcome user bias in A/B Testing by changing the display frequency of each design variation based on the click rate, and opens up further research opportunities on the application of HMM in software testing optimization.