Backtesting

Backtesting is probably the most dangerous activity you can do, since it is all in interpreting the results, and setting it up right.

Some common questions you will need to answer in backtesting any system are:

1. Over which period am I going to backtest ?
2. What parameters am I backtesting ?
3. How am I going to interpret the results ?
4. Am I curve-fitting ?
5. How often do I need to backtest.

I have a lot of experience on backtesting systems and then running them for real. Over the years I have learnt some valuable lessons.

• It is quite easy to develop a system that returns huge profits, but when running in real mode forward you get lots of losses.
• A system with many parameters will be falling into the curve fitting trap easier. Just think how easy it is for a human to identify lossing and winning trades looking back. The more parameters you add the easier it will be for your computer to backtest a "good" result given enough parameters. In fact, given an unlimited amount of parameters you would probably find a combination that was always winning in the "lookback period".
• Any system should perform both well in a trading and in a trending market. if you backtest in a trending market, but end up running in a trading market or vice versa you will probably end up with a loosing system also.
• Suppose you have data from jan - dec for 1 year. Why not back test on jan- may and apply to the jun-dec period. Is the system still profitable ? (roll forward test)
• Don't take a huge backtesting period assuming the system will find avarage parameters. You probalby end up with bad or mediocre results. I have seen people buying 5 years of data, and then backtest on all years. But is there not a huge difference (points and percentage wise) if the stock or index is 100 in stead of 1000?
• There is a huge difference between calculation with percentages in your trading system or fixed numbers. At one period in time I was using a percentage as a stop loss. According to the backtest it was 2% as an optimum. But 2% of a DAX 3000 is different then 2% of Dax at 6000.
• I am a true believer that you have to adjust and thus backtest your system regurlarly. I know some people say "I backtest once and it should always run ok if the system is good". This is just plain nonsense ! What if you backtesting during the .com bubble crash ? You probably would find only short trades ! Good luck with that in the period just after the crash and the bull market since that lasted for more then 3 years after. From this example you can see that backtesting at least every quarter might be a good idea !