I'm aware of this, but I doubt most people are. That's why I'm suggesting using the test on only new data. Yes there are ways to account for selection bias, but they are complicated and require understanding on the part of the user.
This utility is being designed for people that don't understand how it works completely. To target this demographic we have to keep it simple...but thanks for the suggestions.
~FK
--- In vpFREE@yahoogroups.com, Mitchell Tsai <tsai@...> wrote:
>
> Frank,
>
> This is where Bayesian theory (and more accurate "a priori" beliefs)
> allow more accurate probability calculations.
>
> If you use past data, and assume P(all events) = equal, then you often
> run into pre-selection bias; e.g. I picked a weird set of data.
> So Bayesian analysis will use P(my data set is unusual) = whatever you
> set.
>
> Another example, say I'm considering video poker games in Las Vegas at
> 1) major casino in Las Vegas - P(prior belief in gaffed machine) < 0.01
> 2) non-name casino at Indian reservation where other people are
> reporting suspicious result - P(prior belief in gaffed machines) = 0.25
>
> Then P(belief machine is gaffed after test | prior belief) = function
> of test result and P(prior belief in gaffed machines).
>
> If you use a non-random set of data (e.g. data you have gathered
> before), then
> P(belief machine is gaffed) = function of test result and P(prior
> belief in gaffed machines) and selection-bias-in-original test)
>
> Mitchell
>
> A similar example of selection-bias is one about weather.
> My friend tells me that last week it rained 6 out of 7 days, and they
> ask how unusual that is...
>
> Most people will just calculate how unlikely it is to have rain 6 of 7
> days.
> A better calculation will take into account that my friend is only
> telling me this because it is "somehow weird" (e.g. no royals in
> 120,000 hands)
> and factor in the "selection-bias".
>
> On Apr 16, 2012, at 3:09 PM, Frank wrote:
> > OK. You completely misunderstood what I was saying. It will
> > completely invalidate the testing utility I'm making if people use
> > their currently existing data. Why? Imagine this.
> >
> > You post in the newspaper that you'd like to do a study into how
> > likely it is to be hit by lighting. Not surprisingly, the people
> > that answer your add are those most concerned about this issue (AKA
> > people that have been hit). After looking at all your volunteer test
> > subjects you conclude that the chances of being hit by lighting are
> > 1 in 1.
> >
>
