Building a back-testing suite

In it’s simplest form, back-testing is the process of attempting to verify how a given investment strategy would perform in real-world conditions, without risking real $.

Let’s say you have a hypothesis that a perfect time to buy is when the 20 day moving average (DMA) crosses the 50 day moving average to the upside. You could go out and trade this immediately, but a back-test will help you rule out obviously bad strategies.

I highlight those specific words because there isn’t a full-proof manner to ensure your trading strategy will end up in the black. However, a good back-test will give you a good indicator on how your strategy will behave in the tested market conditions. As long as the delta between anticipated performance and actual performance is relatively small, your customer will be happy.

Below are a few considerations for someone on the journey to build out a back-testing suite.

Back-testing basics

The most novice of mistakes are normally the result of bad data management. Ignoring the total return in your back-test model is the most common. For example: you ignore the dividend return of high-dividend stocks; or you ignore defaults or stock-splits.

For those who have built generalized investment systems will start to recognize that understanding all of these and applying adjustments in the appropriate manner, has some overlap with the needs of portfolio construction and accounting systems.

Implementing these separately to your accounting system leads to a couple risks; your logic may differ (e.g. bugs), or the there could be differences in handling of these adjustments between systems (e.g. your backtest and accounting system disagree on the date of a default). The benefit you get for taking on this risk is that you get to go quicker; at least in the short term.

Takeaway: a good back-testing system needs good accounting logic.

Back-testing beyond-the-basics

Unique identifier != trade identifier

A common reason for building out a back-testing suite is the core systems only support held securities, and don’t support the whole universe of securities that a systematic investor may want to back-test. Additionally, the core systems don’t necessarily understand how a systematic strategy will trade. Once you get beyond cash equities and into futures, you have to consider instruments that roll, such as futures.

Let’s say an investor believes they have an insight that will help them trade oil futures. Their core systems will probably understand CLZ1, CLH2, CLM2, etc; which are the codes for futures with different expiries. However, to back-test a strategy of oil futures, you need to understand when each contract was liquidly traded, and test your strategy against that. Simple back-testing suites will simply use reference symbols like CL1 which refers to the next future that will mature.

Implementing a back-test against such a reference symbol risks ignoring complexities when it comes to rolling futures. For small shops, this might be okay, but manual effort needs to go into considering ‘one-off’ events like the oil futures trading in negative dollars.

Takeaway: a good back-testing system needs good reference data.

Beyond reference symbols

Details relating to traded instruments and ‘reference symbols’ span many asset classes (e.g. first future symbols rhyme with on-the-run treasury codes, and so on). Systematic trading with options probably deserves a whole other article, but it should be easy to see that options which have an additional dimension of strike price has yet more complexity.

Additionally, the trading strategy may not simply consider price, it may abstract contracts somewhat, using greeks to allow for ‘late instrument selection’.

Transaction Cost Forecast

One of the biggest challenges with building out good back-testing suites is understanding the cost to run such a strategy. A strategy with a high-frequency of trading may look great ignoring transaction costs, but may be a disaster when fully incorporated. The basic approach is to assume fairly static transaction costs by asset class, such as ‘It will cost me x basis points per trade to execute this treasuries strategy’. This is essentially the same problem as trying to guess-timate rolling costs.

Perversely, the issues with sub-standard transaction cost forecasts tend not to show up until a strategy is starting to make profit and attract money inflows. A simplistic forecast ignores the depth of the order book, so once a trading strategy attracts money, it will potentially start accumulating larger and larger transaction costs as it becomes a bigger share of the market volume. Or put another way, a small strategy may execute close to mid, but the same strategy with large AUM may shoot through the order book.

The workaround for this is to only play in liquid markets, but then you increase the chances that you’re competing with others with similar strategies.

Takeaway: a good back-testing system needs GREAT market data history, including volume data.

Margin / Borrowing Costs & Tax

Another shortcut taken by many systems is to ignore the cost of leverage, or tax impact of different strategies.

Takeaway: a great back-testing system needs good financing/tax logic.

Integration / Process Issues

Once you have a back-testing suite up and running, there are a suite of integration and process issues to worry about.

• My systematic strategy says to trade X, but I have exchange limits of X-Y.
• I want to use my systematic strategy to trade X, but client account Y doesn’t allow it; what do I do? (e.g. explicit guidelines / lack of custodial support / etc)
• My systematic strategy wants to trade a calendar spread but my downstream system doesn’t understand executions that result in 2 positions
• and so on

Building a back-testing suite also means thinking about how to integrate it into existing systems. This links back to business strategy. Once you’ve built the right product, you need to spend time and effort to get it established in a client and start monetizing your work.

Stepping Back

Hopefully the pattern is becoming clear. As you increase the ambition of a back-testing system, the more you start replicating what core systems do; the more expensive your solution and the more similar it looks.

There are multiple back-testing products out there, and more being built. Additionally, there are many internally built systems at different firms that do a really solid job at evaluating niche strategies.

My belief is that when someone can finally build the core systems that can scale beyond ‘held securities and positions’, there will be a shift to simply using those to trade model portfolios for back-testing. If you can believe that, then why not attack the core problems head on, then offer back-testing as an add-on service?