So how do we deal with it? As Nassim Taleb has mentioned in Models on Models, practically profitable trading strategies often hold robust theory and have little dependence on probability estimate.
Purely statistics based trading strategies are dangerous.
Pure number crunching does not reflect how markets evolve, and that financial distributions have consistently implied a pathological nature. This means we can NOT assume that bigger sample back test results converge to population statistics, i.e. great historical performance are really meaningless without robust principle explaining the inefficiency exploited.
Here's a simple applet from HQuotes showing possible equity curves with customized expected values, for a small number of bars(trades). We can see that unless you have a significant edge, short term performance really can not be trusted!