“I possess a quality unquantifiable by its very nature.”
“And what exactly is that, Peter?”
“Perversity.”
Peter Riviera & Molly, Neuromancer

Prediction is a funny thing.  You’ll encounter lots of traders who swear up and down that what they do is not predicting the market.  This, of course, is nonsense.  When you take a position in the market you’re predicting that price will move in the direction of your position and thus you’ll be able to exit your position at a profit. It’s as simple as that.  Prediction is the heart of trading   So why do many traders, even some successful ones, deny they engage in prediction at all?

The reason, I think, is that predictions are so frequently wrong.  As I’ve previously mentioned you don’t have to be particularly accurate at predicting the market in order to make substantial money assuming you make a large number of predictions and the associated trades.  You just have to be somewhat more accurate than random.  What many people miss, however, is that this also means you be wrong extremely frequently.  If you take the example from that post you’ll be making good money by being right 56% of the time.  That means you’ll be wrong 44% of the time.  Honestly there’s an ego issue here.  No one like to be wrong, and if you’re wrong 44% of the time and trading several times a day, you’re going to be wrong a lot.  It’s just something you have to get used to.

The important thing you have to do is separate the quality of your predictions from the outcome of any single, or even a short string, of predictions.  The obvious tool to accomplish this separation is to look at your predictions in sufficiently large groups via statistics.  But psychologically it’s still unplesant – you’re just wrong a lot.

What’s worse, you are going to be wrong a lot more frequently than you’d like no matter how sophisticated your predictive methods become.  This is because of something I call the ‘perversity of the market’.  Now that should turn up some interesting google traffic 🙂  In all seriousness, the effect is best explained by telling a story, a sort of trading koan if you will:

A grain trader and a technical analyst were watching the wheat market decline in price.  As price stabilized at a lower value, the technical analyst explained that this would be the low for the day due to a combination of strong support from the trend line and the strongly oversold MACD indicator.  The trader examined the chart and remarked “looks like you’re right”.  Then he sold 500 contracts of wheat.

I’ve heard variations of this from several traders – it’s a sort of trading inside joke.  First, you have to know something about the wheat market: it’s pretty thin.  Even doing business in the pit in the old days, selling 500 contracts would move price substantially lower.  Second, you have to realize it probably didn’t actually happen – it’s unlikely anyone with trading authority to take multi-million dollar positions would do so just to make a rhetorical point.  But none the less it captures the essence of something you need to understand about trading: unpredictable shit happens.

The point here is that none of the technical analyst’s tools were able to predict the trader would sell.  The most rigorous numerical methods can’t overcome simple bullheadedness.  Let’s assume for a second the technician’s methods were not only effective but perfectly effective – they captured all the information to be had in previous price, volume and tape data that would shed light on the future price of wheat.  That’s giving the technician a lot of credit – MACDs and trend lines aren’t typically the most effective predictive methods in my experience.  But I’m feeling generous.  Let’s just assume the technician was right, and had the trader not sold that would have been the low price of the day.  But he did sell, and a new low was made.  So clearly the technician’s prediction methods were not up to the task.

It’s easy, but wrong, to view this as an argument for fundamental analysis over technical analysis.  Fundamental analysis would be equally ineffective at predicting the price decline.  No amount of estimating the number of acres under cultivation, surveying wheat yields, or predicting the demand for flour next quarter would predict that 500 contract sale was coming.

Similarly, it would be easy to view this as an argument for the efficient market hypothesis.  This has a bit more merit.  If price moves are based off unpredictable events shielded from public view, then all prediction is hopeless.  But I think the efficient market view is ultimately flawed.  It in effect presumes all traders buy and sell for reason as arbitrary as the guy in the koan.  While it’s possible for price to move for perverse reasons, that’s not the only thing occurring in the market.  Far more traders are making decisions based on some combination of market conditions and their fundamental views.  The behavior of those traders is in fact predictable with analytic tools.

So now we have a new view of the market where price moves as a result of two causes – orderly activity you can predict with the right tools, and perverse activity that can’t be predicted with any tool.  You could, in engineering terms, think of it as a mix of signal and noise.  Unfortunately, unlike most engineering signal processing problems, the noise component is frequently quite large.  And this in turn circles back around to why your predictions will never be exceedingly accurate.  Because the better you analysis is, the more the noise component of price works against you.

Let’s say you have a prediction method that is as good as possible.  Looking at whatever data, you can predict where price will go 100% of the time barring arbitrary new participants like the trader from the koan entering the market.   So you set up your trades such that your profit taking exit order is a limit order located where you predict price will go.  For safety/capital preservation you also place a stop loss order in the opposite direction – for simplicity’s sake let’s say the same distance.  Now we introduce noise into the equation.  If the noisy price movement is in your favor, it does not benefit you.  Price just goes to (and likely past) where you thought it was going to go.  But if the arbitrary movement is against you, price moves to your stop and take a loss.  So you get results that look like this:

• No big random activity: you win 1 unit
• Random activity in your favor: you win 1 unit
• Random activity against you: you lose one unit

So clearly your outcome is dependent on the probability of perverse market behavior.  You go from winning 100% of the time if there is none, to winning only 50% of the time if it occurs every trade (assuming the resulting price movement is random in direction).  And this is with the best possible analysis.  What this means in practice is that it’s very rare for even the best trading systems to have profit factors of greater than about 3 – which corresponds more or less to being right 3 times for every time you’re wrong.  It’s much more frequent to have a profit factor between 1 and 2 – representing an accuracy between 50% and 66% assuming  wins and losses are of the same size.

With practice, you’ll learn to read off the charts when your losses are due to things that were building up for a long time (and thus could have been predicted) vs. when they’re associated with random events that you can’t do anything about.  This makes it a little easier to cope with the perversity of the market since you’ll at least know your analysis wasn’t responsible for your losses.  But it’s still unpleasant.