CosmicQueso
Member
if we want to evaluate how big of a boost a platform enjoyed from a set of circumstances behind it, the question to be addressed is the following:
'How many new users bought a PS4 beyond the norm due to conditions XYZ being present?'
Sure. Understanding the incremental lift over baseline demand is the best way to do this. Agreed.
To address this question you'd need to have a baseline estimate for comparison, which we do (May 2015). Not all data nor all manipulations of data are relevant for addressing the posed question. I'd imagine everyone agrees with me up to this point and assume we are on the same page.
Well here is a bit of an issue. In order to compare baseline vs incremental demand in the period being examined (May 2016) one would also have to strip the incremental from the baseline in the preceding periods that are being compared. That's what makes this kind of analysis exceptionally tricky/impossible, especially when we're looking at monthly aggregate extrapolated data.
An estimate of how many new users bought PS4 specifically due to those conditions XYZ should focus on the difference between the control (May 2015) and the experimental (May 2016) data.
Yeah, problem is May 2015 isn't really comparable, until you strip out incremental and then adjust for cyclicality. It's a real tough comparison to try and do.
Conclusion: Percentages aren't the best measures of how significant a boost in hardware sales is. They obscure the much more relevant data point (difference between result and baseline).
I think I'd agree with you that the % isn't a great measure in this case, but really, I'd extend to the actual unit difference as well. The variance from the expectation isn't significantly higher, it's not outside what an expected range of error would capture.
Opinions can vary on what constitutes a 'big/medium/small' boost. My opinion is that I would have expected a much larger difference than just 50k given the 3 or 4 very major conditions promoting PS4 sales in May.
Yes, agree with you on the big/medium/small and assigning qualitative measures (impressive/not impressive) without quantitative framing is frustrating/pointless. As for the number itself, coming in over 200k was a good result, but I wasn't thinking that it was exceptionally so. More in the range of, "yeah, okay, that's alright". I don't know. I don't do qualitative assessments very well lol.
If someone wants to argue that the relevant metric for quantifying a boost is not difference in sales compared to baseline but instead is an entirely multiplicative factor, that's fine and would make the pure % difference meaningful if compelling, but I have yet to see any such argument.
I think both % and unit changes are important and meaningful, but I think they're only truly meaningful when placed in context with each other. And I very much prefer the baseline/incremental demand methodology for measuring sales of consumer goods. It's just that the data we have to work with is such absolute shit for doing anything but the most rudimentary analytics that it makes the exercise laughable in practice.