Low Data Volume
Whilst surveys and polls have a sample size that is meaningful, mystery shopping is a one-man job, the smallest of the small samples. Executing a mystery shop takes some time, a shopper needs to be found, assigned, then travel to the shop, execute the shop, and then the mystery shopping company needs to check the quality, follow up with unclarities. This can be costly and results in companies cutting down their mystery visits per store, resulting in fewer data points that can be collected every week or month.
Surveys can be designed to bring results that are as objective as possible, but still, a shopper is not unbiased, has own preferences, habits and shopping behaviour. There are many trained shoppers that understand what customer service should be and how to conduct a mystery shop, however, there are also many shoppers that do not. The mystery shopping industry is fragmented, with not always sufficient quality control. This can skew results and the validity can be questioned as every mystery shopping report becomes questionable. As mystery shopping limitations go, data validity is the least problematic as most companies – AQ included – have strict quality assurance policies in control in order to ensure that clients receive only correct data to work with.
A mystery shopper captures a snapshot of the whole. A moment is captured, and depending on the circumstances, such as staff shortage on an extremely busy day, experiences differ and do not always reflect the regular customer visit. If a mystery shopper had a negative experience in one store, that experience is projected to the whole store and all the employees. Having many data points that would proof so, received from a larger sample set, would make it worth investigation, but with the few data points per shop, this becomes a dangerous assumption.
A mystery shopper has to match a certain profile to provide the best results and to make the interaction credible, depending on criteria, finding shoppers with a specific profile is very difficult. Also then, the mystery shopper is not a real shopper. He or she is trained to look for certain points of attention. Who knows if a real customer notices or is bothered by that dirt pile at the back of that corner the mystery shopper carefully spotted?