Building Profitable Trading Systems
Building Profitable Trading Systems
Table of Contents 2
Introduction 4
The ELM Strategy Development Protocol Overview 8
Step 1: Your Key Idea 12
Step 2: Define Your Development Environment 19
Step 3: Sanity Check 28
Step 4: Stress Test Key Idea 31
Step 5: Market Environment Filters 40
Step 6: Create The Baseline System 49
Step 7: Independent Testing 52
Step 8: Combining Rules 69
Step 9: Out-Of-Sample Testing 82
Step 10: Live Market Testing 92
Step 11: Position Sizing 94
Step 12: Live Trading 103
The Next Level! 107
Closing 108
This e-book is for general educational purposes only and should not be constructed as personal
financial or investment advice or instruction. The information provided should not serve as a
substitute for professional financial or investment advice. Although our employees may answer
your general customer service questions, they are not licensed under securities laws to address
your particular investment situation. No communication by our employees to you should be
deemed as personalized investment advice.
Building a profitable trading system is very difficult. I'm not going to sugarcoat it. It's a difficult
project where most of your ideas will not work. To make matters worse, you're bombarded with
misleading advertising, confusing claims, and simply bad information. Learning a computer
language to build a system is great, but that's not enough to build a successful system.
Combining a few technical indicators with a buy order and adding trailing stops are not going to
make you a profitable trading system. You are going to need an organized approach and perform
specific steps to create a successful trading system. Often, this information is difficult to locate as
most information on the web is geared towards outrageous get-rich-quick promises, equity
curves that promise a smooth ride to weekly income and high pressure sales. In short, within the
field of system development there exists a lot of noise, hype and misinformation that can take you
astray.
So, how do you create a successful trading system? You start by reading this consumer guide.
In this fact-filled eBook, you’ll discover how to avoid common pitfalls amateurs make all the
time. You'll learn the important system performance metrics and what they mean to your bottom
line. You'll discover how to correctly isolate trading ideas and independently test those ideas to
produce a robust system that will have a better chance of surviving on out-of-sample data.
Finally, you will learn the proper way to use optimization to avoid curve fitting.
We wrote this guide to help you better understand the system development process. Now with
this information, you can begin to build more robust trading systems and begin your path to
system trading success. And if you have any questions about system trading, you’re invited to ask
us at: ebook@[Link]
We’ve dedicated our business to educating consumers. We’ll be happy to help in every way.
Truly,
Jeff Swanson
EasyLanguage Mastery
Website: [Link]
E-Mail: ebook@[Link]
The solution to our haphazard way of developing a trading system is borrowed from the scientific
world. The solution is to adopt a systematic method to build and test trading systems. Within this
eBook you will find the ELM Strategy Development Protocol of automated trading system
development. The ELM Strategy Development Protocol will help reduce the chance of creating a
poor system that is curve-fitted to historical data and increase your chance of success when
testing on out-of-sample data. In short, this method can help you build a robust and profitable
trading system.
Developing trading systems is not a true science like physics. It’s really a mix of both art and
science. Your creativity comes into play when attempting to uncover market edges that may lead
to a great trading system. The science side of the process comes into play with careful
observation, note taking and rigorous testing of your trading model. Unlike science however,
there are no hard set rules that will produce constant results or winning systems. In physics the
law of gravity affects all objects on Earth the same way, all the time, everywhere. It’s a law that is
universally known and understood. The theory of gravity can make excellent predictions about
the future. For example, it can predict when the moon will rise and how to estimate the flight
time of an international flight. The market follows no such law. You can’t predict where the
market will be with 100% accuracy 10 minutes into the future or next year. The market is
dynamic and not governed by predictable laws of nature. In fact, the markets are known to harbor
Black Swans where unpredictable events that have a dramatic impact on your trading suddenly
appear. This is like gravity suddenly changing its strength making all your predictions obsolete.
Can you can imagine the chaos that would cause?
So while we are not scientists studying the laws of nature, we as system developers can borrow
from science some of the tools and methodologies to help bring some rigor to our testing. It is
this very area of system development where a lot of people fail. Over time I’ve borrowed concepts
from other developers, amended their ideas and added a few ideas of my own to come up with a
process to help me develop trading systems. I’ve also borrowed concepts from science to help
solidify my method of system development. Is my method the best there is? Of course not. There
are many ways to successfully create a trading system, but there are even more ways to do it
wrong. Pitfalls abound! This eBook highlights some of the bigger mistakes to avoid as well as
Following the very rules in this eBook allowed me to create a trading system with an equity curve
like this:
$50,000
$43,750
$37,500
$31,250
$25,000
$18,750
$12,500
$6,250
$0
-$6,250
1 13 25 37 49 61 73 85 97 109 121 133 145 157
WHAT’S INCLUDED
This eBook describes an overall process in building and testing automated trading systems.
In this document I’ll walk through the major steps I perform to develop a system. I will show you
the complete path and we'll end up with a fully automated system. I can’t say the final system will
make you rich because it wont! However, going through the steps in this eBook will provide you
with a method for testing and developing a trading system.
SAMPLE CODE
With this eBook you also received the sample code. This sample code is the actual code used in
the system development example throughout this eBook. If you do not have this sample code
please contact ebook@[Link].
The notes created during the development process I call my System Development Notes (SDN).
Each system you create should be accompanied with its own collection of SDN which are nothing
more than organized pages of data and observations. You will find yourself returning to your
SDN during the development process months and years later! Your SDN will be an invaluable
asset to your system development process.
In order to help you with this I’ve included example SDN forms. They are included with the
purchase of this eBook as a separate PDF document. As you go through the development process
described within this eBook you can print the needed SDN sheet f and fill it in. Each sheet
Using SDN to record your progress will save you a ton of time. It can also provide valuable
insight as you may discover patterns between the various systems you develop.
Feel free to use my SDN exactly as they stand, or use my SDN as inspiration for creating your
own custom SDN. I’m sure you’ll end up doing just this and that's great. The SDN I've provided
are rather generic knowing most people will want to create their own forms tailored to their
specific needs and personal style. You can easily create great looking forms with Apple's Pages
program, Microsoft Word, or Google Docs. You can even forgo paper and physical binders if
you wish. Jump on the digital bandwagon by using on-line tools like Evernote, Google Docs or
even a note taking app on your iPad to create and archive your SDN. However for starting out, I
would recommend using a pencil and copies of the SDN provided . Down the road after you have
done this a few times and you feel the need to begin to change them around to suit your needs,
start exploring other tools to create and archive your SDN.
In any event, the SDN found within your download package should provide you with a great
starting point on creating your own SDN. Throughout this eBook I will be demonstrating the use
of these forms as we progress through each step in the development process.
You will notice the second page of the SDN labeled "Check-list" contains a quick summary of all
12 steps to perform. This sheet provides a place to record a date and brief summary of your
progress as you progress through the different steps.
3. Sanity Check
Quickly eliminate unproductive Key Ideas by glancing at equity curve.
7. Independent Testing
A method to avoid curve fitting while improving system performance.
8. Combining Rules
Bringing together the best ideas into a final system.
9. Out-of-Sample Testing
Testing system performance on unseen data.
This key idea should be simple and easy to program into an automated system. For example, a
key idea might be to simply purchase a 20-day high. The hypothesis behind such a key idea is the
belief that all new trends start with a 20-day high. This is an example of a trend following trading
concept. However, not all markets have trending characteristics. For example, there are many
examples on EasyLanguage Mastery that show the S&P E-mini futures market exhibiting strong
mean reversion behavior. In this case fading or going short upon 20-day highs may be a better
idea. The point here is you need to know your market. You need to know its characteristics and
come up with a key idea.
Once you have a key idea it can easily be programmed into an automated system and tested to see
if it has any validity. If it does, you just might be able to develop the key idea into a complete
trading system.
In the real world, steps 1 and 2 can can switch positions. That is, there is nothing stopping you
from first defining your testing environment first then attempting to discover and build a system
around your testing environment. However, it seems to me most of the time I will run across a
potential trading concept first then begin the process of nailing down the testing environment.
Where can you come up with key ideas? Of course as you begin to develop your own system, read
about the market and gain experience with the market, you will begin to notice key ideas you can
test. But there are other ways to discover key ideas and those can include articles found on blogs,
magazines, forums, books and just about everywhere else. Keep your eyes open to any potential
Where do I get some of my ideas? My ideas can come from just about anywhere. Some ideas I'll
stumble upon when performing my own research but there are so many great resources for
trading ideas you can really leverage your time by seeing what others are doing. Some of my
favorite ideas have come from various blogs which focus on systematic trading such as
EasyLanguage Mastery. Other ideas come from magazines like Technical Analysis of Stocks and
Commodities and Futures. Finally, books can also be another great source of ideas. The problem
is not finding ideas, but capturing those ideas so you can test them later before you forget about
them. I solve this problem by adding every idea I run across to a folder in Evernote. Opening that
folder now I'll have easily over 100 ideas to test.
It’s important to capture every idea you think might be useful either now or later when
developing systems. This could be a broad market study that demonstrates stocks above their
200-day simple moving average tending to perform better than those below it. Or it could be a
unique and specific way to use an indicator as a dynamic profit target. Put these into Evernote
(or something equivalent). Why do I like Evernote? It’s cross platform thus, I have access to my
notes on a Windows machine, Mac, or even my iPhone. I never know when I’ll run across an idea
but with Evernote on my team, I’ll be 99% likely to have a way to capture it. A smart phone
provides the best ability to do so. I can use the camera to capture an image found within a
magazine and email it to Evernote. I can create a copy of an entire online article within Evernote
with a couple of clicks. If I receive an interesting email, I forward it into Evernote. It has become
my central and ubiquitous storage location for all ideas worth capturing.
When I create a system I often like to have two components to the key idea in regards to opening
a new trade. The first is the setup. They setup is nothing more than an indication that the market
environment is ripe for a possible new trade. This might happen when an RSI level reaches below
10 or when a fast moving average crosses above a slower moving average. It could be when price
The second part is the entry technique. This is a method to pinpoint an exact entry. Often the
point of an entry technique is to 1) find an optimal point of entry as to reduce risk and/or
maximize profits and/or 2) provide confirmation of the trade. A entry technique can be simple as
well.
For example, let's say we have a moving average crossover system that wants to go long when a
10-day (fast) moving average crosses a 50-day moving average (slow) on a daily chart. When the
fast moving average crosses above the slower moving average this signals a long trade set-up is
occurring. At this point we now turn to our entry technique which is to delay buying the market
until three days later. After three days we look to see if price is above the 50-day simple moving
average. If it is, we go long. If it's not, we don't [Link] entry technique acts as a filter to
prevent whipsaws by helping to confirm price is still acting bullish. You can see this in action in
the article, Improving The Moving Average Crossover System.
Other entry techniques could include buying at the high of the highest past 3-bars or setting a
limit order at the low of the last three bars. Ideally an entry technique would be based upon
information different from your setup condition. What I mean by that is most setups are price
based. For example, a moving average crossover or an RSI below 10 are all based upon price. If
these were your setups then finding a technique that uses something other than price could be
very beneficial. Instead of price using volume, the put/call ratio or even the $TRIN could make
great confirmation techniques. But, this is not always possible.
Another excellent way to create an effective entry technique is to use another timeframe. Drilling
down to a 5-minute chart from a daily chart in order to pinpoint an optimal entry point can be
fantastic entry technique. I've successfully used this to generate signals on daily chart but then
pinpoint an entry on a 5-minute chart. However, this is an advanced technique which will not be
covered here. But you should keep it in mind as a developer.
With the idea of setup and entry technique explained you will notice our example key idea does
not have a entry technique. You'll run across that as well as you find key ideas in magazines or
website. This provides a great opportunity for you to test entry techniques. However, for this
eBook we will not be testing a entry techniques for our example system. This is just simply for the
For this eBook we are going to use the daily reversal bar as our key idea. Our hypothesis is that a
reversal bar often signals a trend reversal in the market. This key idea is going to be the core of
our trading system.
What is a key reversal bar? Well, there are many definitions but for our purposes we will use the
following simple definition:
• For Shorts: A bar that produces a new 10-day high and then closes below the close of the
previous day. Enter at open of next bar.
• For Longs: A bar that produces a new 10-day low and then closes above the close of the
previous day. Enter at open of next bar.
The image below is a daily chart with arrows showing key reversal bars. Sell short signal is red
arrows while the go-long signal is the blue arrow. The key reversal bar appear before the trade
opens because all trades are entered at the open of the next bar.
Here is what the EasyLanguage code looks like for defining if a Bullish or Bearish key reversal
has taken place.
KR_Bullish = ( Low < Lowest( Low, LookBack )[1] ) And ( Close > close[1] );
KR_Bearish = ( High > highest( High, LookBack )[1] ) And ( Close < close[1] );
The LookBack variable is 10 thus, we are looking for the Lowest or Highest bar over the past 10
days. The variables KR_Bullish and KR_Brearish are boolean variables which will be set to true
if a key reversal bar is detected or it will be set to false.
Now we will need an exit. A simple exit is to exit when price closes above a simple moving
average.
• Exit: When price closes above (below) the 10-day simple move average, exit at the close
of this bar.
Here is what the EasyLanguage code looks like for the exits.
Variables:{---------------------------------------------------------------------------------}
KR_Bullish = ( Low < Lowest( Low, LookBack )[1] ) And ( Close > close[1] );
KR_Bearish = ( High > highest( High, LookBack )[1] ) And ( Close < close[1] );
vShares = Intportion( AccountSize$/Close );
If ( Marketposition = 0 ) Then
Begin
If ( KR_Bullish ) Then
Buy ("KR Bull") vShares contracts next bar at market
Else If ( KR_Bearish ) Then
Sellshort ("KR Short") vShares contracts next bar at market;
End;
On the next page you will find the entire code for our key idea.
Action Items
• Don’t let a good idea escape you! Organize your favorite blogs and blog aggregators into a
single folder (or RSS reader) and review them regularly so you can capture any good idea.
• Download Evernote and start using it. When you find an interesting idea, article, image, book
recommendation, website, or anything else save it to Evernote. These will provide inspiration
on key ideas to test that just might turn into profitable ideas. Of course, you don’t have to use
Evernote. Use whatever works for you but I found with Evernote’s web clipping feature is very
easy and convenient to use.
• Use SDN Sheet labeled "Step 1: Describe Key Idea” to begin listing other key ideas you would
like to test.
If you only work with one platform it may seem a bit silly to record this, but the version number
may be important down the road if suddenly your strategy no longer functions correctly. This
might give you a hint that an upgrade may be conflicting with your strategy. Your SDN will tell
you what version it was working under correctly.
WorkSpace Name
For TradeStation users this is the name of the WorkSpace you will be creating to hold your chart
and strategy. I like to record this because down the road you may not remember what you called
it. You will then spend a lot of wasted time looking for it when you want to view it. If you don’t
use TradeStation this will be different for you.
This is the actual name of the strategy code file. In TradeStation this is the name of the
EasyLanguage file which contains the actual strategy code. Again, I like to record this so I can
easily find the file when looking for it many months or years from now.
200
This may not apply to you if you’re not using TradeStation. However, TradeStation users must
provide the maximum bar look-back value within when backtesting their strategy. This value is
the number of bars required for your various calculations used within your strategy code. For
example, if you have a moving average crossover system you will need the longest look-back value
used. Changing this value can have an impact on the performance report! Thus, it should be
recorded to avoid confusion down the road.
I will be developing a system to trade S&P ETF (SPY). The trading system should also work on
other stock index futures markets.
This is entirely up to you. Each market has its own characteristics and based upon your account
size, goals and personal preferences you will need to narrow down your idea to a specific market.
In the case of our example system, a stock index market.
For newer system developers I would first suggest you get accustom to building a system for a
single market. That is, build a system to trade APPL or GOOG but don't build a system for both.
However, for advanced people there are other options which go beyond this eBook. See the
Advanced Tip below. Given you have developed a system on a single market I will point out that
it's an added bonus if your finished system will work well on other related markets. In our
example case, if our finished system also works well on the S&P this boosts our confidence in the
soundness of our system since it performs well across a different instrument that was not used
during development.
Advanced Tip: A more advanced method would be to develop a trading system over a
portfolio of different instruments. In our example, instead of simply building the system
specifically on ES contract we could also include other major future indexes such as the mini-
DOW and Russel. Notice these different instruments are logically similar. The example system
we are building is for a major U.S. stock index market, thus it would not make sense to add
the Euro currency futures or Apple stock into our portfolio. But adding the mini-DOW and/or
My system will generate its trading signals on the close of the daily bar and enter on the open of
the next day.
Here is a big hint: the smaller the timeframe the more difficult it is to develop a successful
system. In other words, developing a system to trade on a 5-minute chart is more difficult than
developing a system that trades on a daily bar. There is a lot more noise on the smaller
timeframes and given the massive amount of data you would need to test the system over years
and years of different data, it can be a very difficult task.
Smaller timeframes often mean less profit per trade and less risk per trade. It's a good idea to
strike a balance between your trading account size and the risk you are willing to take. While this
topic can get more complicated, let's just highlight two important aspects. First, the smaller your
trading account the smaller the timeframe you should trade. Second, the smaller the timeframe
the more difficult it is to trade.
Do you see the trap here? New traders often come into the trading world with very little cash
hoping to make a quick buck. They trade small timeframes because they think day trading is the
way to make money, so they start trading on S&P or Gold futures on a 5-minute chart. By doing
so they suddenly put themselves in a very difficult spot because they are trading a very difficult
timeframe when they are least experienced!
It's much easier to develop profitable trading systems on daily timeframes than a 5-minute chart.
Because of this, I urge new system traders to build trading systems on daily or weekly bars charts.
I don't care if you ever plan on trading those timeframes; you do it to build confidence and skills
I want to say a few words about a trading style often called "scalping." Scalping is something that
intrigues many system traders. The challenge at taking small, consistent trades from the market
daily while risking very little is appealing. With scalping it’s generally expected you are trading
from a small timeframe, probably 5-minutes or less. The idea is to open a position and capture
only a few ticks of profit. The appeal is since we are trading from such a small timeframe, your
risk is small thus, you can trade with a small account. Often you will have setups that produce
high win rates and occur more frequently than setups on a higher timeframe such as hourly or
daily. There tends to be a higher frequency of trading opportunities with scalping which can
potentially lead to large accumulated profits versus your starting equity.
In my opinion, scalping for the retail trader is very difficult to do. If you're new to system trading
I would say, don't even think about attempting to do it. Why? One of the biggest barriers is the
transaction cost in both commissions and slippages. Scalping means smaller profit per trade yet,
as you drill down to smaller and smaller timeframes your costs remain fixed. Thus, the negative
impact of commissions and slippages grows taking a bigger percentage of your profits. A single
tick of slippage is hardly noticed when you are holding a trade for several days with an average
profit of $600 per trade. However, on a scalping system that single tick is the difference between
life and death. Then throw in latency, computer issues, internet issues and your margin for error
is tiny. Again, on larger timeframes you can exit a trade now or in a few seconds and it won’t
matter that much. Not so in the scalping world where everything is hypersensitive and your
margin for error is tiny. Unless you are skilled at system development, stick to higher timeframes.
What timeframes do I trade on? I've developed profitable systems down to the 5-minute time-
frame but there is a big point to make here. All of those systems utilize multiple timeframes. For
example, a given system may open and manage trades on a 5-minute chart, but the the software
looks at the hourly or daily time frame to help determine when a trade setup is taking place. In
general, the idea is to use a larger time frame to determine when potential trading is setting up
and use the smaller timeframe to pinpoint your entry. Often I will also manage the trade at the 5-
In this light, I'm going to be developing a system that trades based on the close of a daily bar.
A total of $20 will be deducted for commissions and $.04 per share per round trip.
Be sure to always include an accurate value of both commissions and slippage when developing a
system. Both eat into profits and can turn a seemingly winning system into a loser if you forget to
add them. This can often be the source of problems with new traders. They ignore the negative
impact of slippage and commissions and develop what they believe is a profitable system.
However, in the real world both slippage and commissions can steal vast sums of money from
your account.
To demonstrate the insidious effect that commissions and slippage have on a scalping trading
system, for my first backtest I deducted nothing for slippage and commissions. I then added
commissions and slippage and ran the backtest again so you can see how it impacts the equity
curve. You just might be surprised at how damaging commissions and slippage can be.
The above equity graph shows just under a $7,000 net profit. So we have gone from $22,000 of
profit to only $7,000 of profit with a $5.00 commission charge and 1-tick of slippage! The visual
difference between the graphs is even more dramatic if they were all plotted with the same scale.
Nonetheless, the final equity graph is radically different from the first equity graph. Looking at
Trading Equity
$20,000
Take a reasonable stab at what the trading equity will be for trading. This does not have to be
actual value of your trading account. It should be realistic. That is, if you’re trading futures on a
daily chart don’t put $5,000 as trading capital. That’s far too little. We need to enter this into
our testing platform so it can correctly compute the returns on your trading system. Later in this
eBook we will actually calculate the minimum size trading account to trade the system we are
creating. For now, just enter a realistic value.
This will give us about 14 years of historical data to develop our system. During this period we
also had several major bull markets and several major bear markets.
One has to use as much backtesting as required. There is not a set answer that will work for all
systems. In general, the more the better. The real point is not how much data do you need in your
historical test but, how many trades are generated during the timeframe and do you cover both
bull and bear market regimes? We don't want to simply develop our system under one market
condition. Developing a system over a wide range of market conditions helps us ensure our
system can withstand these different market conditions. It makes our system more robust. It’s
easy to make a trend following system on a market that is in a powerful bull market, but what
happens when that bull market ends? Thus, be sure to include enough data to cover a wide range
of market conditions. Your system should be developed over bull markets, choppy markets and
bear markets. A good example is the U.S. markets, such as the S&P, during the year 1998 - 2012.
The out-of-sample period will be from January 1, 2012 to December 31, 2019.
Here it will be important to have enough data to generate a reasonable number of trades so we
can get a look at how the system performs on “unseen” data.
Another number to keep in mind is that the size of the OOS can be around 20% or more of the
historical data. But what’s more important is the out of sample period should contain a
reasonable number of trades. What is reasonable? I would say an absolute minimum would be 30
but 50 is much better. Having more than 100 trades is even better. The more the better. I would
also recommend the OOS period should also contain both bull and bear regimes. Just like the in-
sample period where we developed the system over various market regimes, we would also like to
see how the system performs on the OOS given different market regimes. This might not always
be possible, as is the case here. In order to create enough historical trades on the in-sample
segment I needed to extend the range to include 2011. Thus, the out-of-sample data, from 2011
through 213, comprises of a bull market. This is not a deal breaker but it's not optimal either.
When you have your completed trading system you will want to compare the performance versus
a benchmark. This will provide you with a way to gauge how effective you system performed.
When comparing our system to the benchmark we will not only be looking at annual returns. We
will wish to compare other values such as drawdown and time between new equity highs. All of
these values, and more, will help paint a picture of the behavior of our system and determine if
your system is truly tradable on the live market. In the end, we want to see how our trading
system's performance compares to our benchmark over the same horizon and if those metrics are
more aligned with our personal tolerances in trading.
At this is point I perform a sanity check on my Key Idea to compare it with my other Key Ideas I
have yet to test. With so many ideas, I'll often have a dozen Key Ideas awaiting for me to test. I
can't always get to them right away so I set them aside for latter.
With so many key ideas we need a way to quickly determine which are the best. We need a way to
rank our Key Ideas and eliminate the unproductive ideas. By doing this, we can focus our time
and energy on the most promising prospects. This will help us get the results we want, more
quickly and effectively.
A simple sanity check is to rank your key idea based upon an objective score. This score can be
something as simple as the net profit, the profit factor or the annual rate of return. I like to use
Expectancy Score as defined by Van Tharp.
Expectancy is a concept that was described in Van Tharp’s book, “Trade Your Way To Financial
Freedom”. Expectancy tells you on average how much you expect to make per dollar at risk. For
example, if you have a trading system that has a .50 Expectancy that means for every dollar you
risk the trading system returns $.50.
Now that we have our Key Ideas Expectancy value are we ready to use this value to compare to
other Key Ideas? Not yet. There is another step we must perform first. Our Expectancy value
simply tells us our historical profit per dollar risked for each trade. But we are missing
something. Let’s imagine we have two trading systems that have two different Expectancy values:
Based on what we know it appears Trading System #2 produces more profit per dollar placed at
risk. In fact, it produces twice as much profit per dollar risked. Thus, if we risked $500 on each
trade, Trading System #1 would generate $125 dollars while Trading System #2 would generate
$250. But this not the complete picture.
We are missing the frequency at which each trading system operates. For example, maybe
Trading System #1 trades once per day while Trading System #2 trades once per week. We need
to take into account the number of times the trading system trades. In Van Tharp’s book he
described this as Expectancy multiplied by opportunity. Opportunity is nothing more than how
often does a given trading system trade. Opportunity times Expectancy leads us to our final
calculation for Expectancy Score.
This value is an annualized Expectancy value which produces an objective number that can be
used in comparing various Key Ideas or even complete trading systems. In essence the
Expectancy Score factors in a trading system’s trade frequency. The higher the Expectancy Score
the more profitable the system. The calculation is below.
Once we have expectancy score we can now rank each of our Key Ideas by this value. I personally
keep a spreadsheet containing a list of the Key Ideas that need review. One of the columns on
this spreadsheet contains the expectancy score. I then can easily sort my Key Ideas based upon
this score thus, revealing which are the top performing Key Ideas. I can then start testing my best
ideas first.
A word of warning, the expectancy score is not perfect. Like any performance parameters it has
it's advantages and disadvantages. A particular system that ranks well really may not be very
practical for you because it just trades too often or the drawdowns are just too large. So, this
method of ranking a Key Ideas is not perfect but, the purpose here is to help you cut through the
clutter.
Note, this score is generated over the in-sample data segment. Ideally, you will want to compare
your other Key Ideas vs the same time period. This will help ensure an accurate reading as the
systems are being tested over the same data. However, this is not alway practical. Don't worry
about this too much, as generating the score is much better than not having one. You could also
perform the sanity check over the entire historical data set (in-sample and out-of-sample). You
would never use the out-of-sample segment this during our development, but for the sanity
check this is fine. In this method you capture the most current trades which get baked into the
score. This can help rank your Key Ideas based on some recent trades. Based on our score with
our Key Idea, we are fine to proceed.
Action Items
• Take SDN Sheet “Step 3: Sanity Check” and perform the Sanity Check on your trading
platform. If you’re a TradeStation user, use the “Key Reversal S&P ETF” workspace provided
with this download.
• Perform the sanity check on other key ideas you have and note which ones are worth pursuing
and others which are not. Enter this information in to a spreadsheet which contains a list of
your Key Ideas to test. Once they have been all scored, you can sort you Key Ideas based upon
the best scores. This will be a great way to start narrowing down all the worthwhile ideas you
have.
Your key idea has now passed the most basic of tests, the sanity check. Now we will stress test
(sometimes called a robustness check) the key idea to see if it easily falls apart.
A stress test is subjective but plays an important role in helping ensure a given input parameter is
not curve fitted to the historical data. We are going to use the stress test on our key idea. Stress
testing involves making changes to the input values to see how the system performs. In order to
do this we will need the ability to vary an input value. Given our key idea, we have two input value
to stress test:
Sometimes a key idea will only have one or two variables. Others will be more complex. What we
want to do is vary the input parameter to see how our key idea holds up. For example, if we
change the look-back period from our default value of 10 to 9, I don’t want to see the system’s
equity curve suddenly change dramatically. Neighboring values should still show positive results.
Likewise, if we increase the look-back period from 10 to 11, I don’t want to see a drastic change in
results. They key point is, the neighboring values around the default value of 10 should still
produce positive results. In fact, it would be great to see the system remain profitable over a wide
range of values. Often what you will see is as you change a parameter in on direction the net
profit may slowly climb then fall over a series of many values. It’s also most like a wave as the
value reaches an optimal peek and then collapses.
A system that holds up over a wide range of parameters demonstrates robustness and passes our
stress test. A strong market edge (one least likely to be curve fitted to historical data) will allow
By default the exit condition is when price closes above a 10-period simple moving average. We
are going to vary this default value of 10 to see how the neighboring values behave. To quickly
test a range of input values I will use TradeStation’s optimization feature. To do this, the value
will have to be made into a strategy input value. In our case it already is. If a value is not an input
parameter you can manually change it in the code and test each value one by one. This would
involve replacing the default value, recompiling the code and executing the system over this in-
sample data, then repeating this process until you recorded the results for all variations. This
could be rather tedious to do. A better way would be to convert the value into an input parameter
so you can quickly change the value. Any input value allows you to utilize the parameters within
TradeStation’s optimization feature. I will test values between 2 and 20 in increments of 1.
Below is a graph depicting the results. The x-axis depicts the look-back period and the y-axis
depicts the profit generated for that particular run.
What does this tell me? The default value appears to be far from optimal which is a good sign.
You can easily see the default value is not an outlier. Furthermore, we can clearly see most values
show profit over a wide range of values, another point in our favor. In the end, I feel confident
A really poor system will have some values producing negative profit and other nearby values
producing positive profit. Stable regions will be more difficult to observe or simply nonexistent.
You don’t always have to use net profit. Sometimes it’s helpful to look at another metric. My
second choice is Profit Factor or Average Net Profit Per Trade. Below we’ll take a look at the
same look-back value versus profit factor.
Remember, with profit factor a value above 1.0 is profitable. We also see our similar picture when
compared to the net profit chart. Neighboring values around the 10-period look-back value do
not vary considerably. This looks OK as well.
During this portion of the stress test I noticed an interesting observation that was not first
apparent. When using a moving average as an exit point this technique works great when a trade
is opened below it. With our bullish key reversal pattern many setups appear after a pullback
Let’s move on to the second parameter, the key reversal look-back period.
Using TradeStation’s optimization feature I’m going to look at look-back values from 1 to 20. On
the next page is a graph depicting the results. The x-axis depicts the look-back period and the y-
axis depicts the profit generated for that particular run.
The first observation I noticed is there is a lot of stability with the loopback. The next thing to
notice is our default value. First, it’s clearly not an optimal value. Far from it! So what am I going
to do about this? Right now, nothing. Why? This situation is opposite of our theoretical overly
optimized example above where the default value of 4 gave us an overly optimistic net profit
To be clear you may still be asking, why did you change the value of the overly optimized
example? During stress testing I will only modify the inputs in one direction - making the system
look worse. Remember, in the overly optimized example it was clear the default value of 4 was an
outlier thus this was giving us an inflated view of performance. Therefore, I moved the value to a
more realistic - reduced level of performance. Again, we are not optimizing here. If in doubt,
don’t change any values. But if a default value clearly looks optimized, select a more “reasonable”
value and start over with the Sanity Check. We are poking a few holes in our system attempting
I hope I’m being clear. In our case, I’m not going to change the look-back period of the key
reversal value because it’s clearly not an optimized value. I’ll note in my SND the bar graph above
and the value should likely be reduced in the future to produce more profit. But this will happen
at a later stage of development.
Obviously when you are developing different systems on your own, your key idea will be
something different and it's likely the input parameter will have a different meaning than our
existing look-back parameters. For example, testing thresholds for an oscillator will only make
sense in a defined range, -100 to 100. On the other hand, when testing a look-back period only
positive integers will make sense. In short, you will have to test values that "make sense".
Reject or Accept
It is at this point you determine if you should reject or accept your key idea based upon the stress
test. What you determine as a "passing" result for the stress test is subjective that will become
more clearer with experience. Follow the guidelines provided in this chapter and you should be
OK. Looking at the results of our stress test I have to say our system passed the stress test. To me
this indicates our testing of the key reversal idea can continue to the next step. We have the
green light to dig deeper. Since we are going to dig deeper, it is at this time I like to record some
performance metrics.
Record Performance
It is at this time I like to record the performance of the key idea over the in-sample period. Below
is the performance summary along with the equity graph.
Max Intraday Drawdown $5,617 Sanity Check. This will eliminate even more
Often when designing a system it’s important to keep the big picture in mind. What is the overall
market doing? Market environmental filters is a term I use to describe the overall market
conditions for the instrument you’re trading. These are the large macro level pictures that appear
on the daily or weekly charts. This large macro picture can be applied to any market. For
example, when looking at a daily chart, the market can be either in a bull market, bear market or
perhaps in a range bound market. To keep things simple we can use the binary technique to
divide a market: bullish or bearish.
We are all aware that price action is a mirror of human psychology of market participants,
therefore price action can often be very different between these two regimes. Sudden market
plunges that we see within strong bear markets, such as in 2008, behave much differently when
contrasted to the continual grinding, upward market we saw after 2009. People behave
differently under fear (how low can it go) and pain (look at all the money I lost) versus. doubt (this
market can’t go higher) and greed (I’ll just make a bit more before I exit). Since people behave
differently under these two market regimes it makes sense that we should design trading systems
that take advantage of the different market characteristics for each regime. We want to build a
system that dynamically adjusts its trading parameters based upon which market mode we’re
experiencing. To do this we can use any number of indicators and we’ll explore several of them in
this chapter.
A bull or bear market regime filter simply divides the market into modes: bullish or bearish.
Often this is done so you can trade with the major market trend. For example, only take long
Once again, a market environment filter is nothing more than a way to divide the market into two
distinct modes. In this case we are looking to divide the market into bullish or bearish. A very
simple way to accomplish this is to use a 200-period simple moving average on a daily chart
(yellow line on charts below). A market that is trading above its 200-day SMA is bullish while if
it's trading under it, the market is bearish. Introducing this simple concept along can improve
the performance of many trading systems. I’ve personally used this technique with success many
times.
The most obvious way to accomplish this is to only take long trades during a bull market and only
sell short during a bear market. The concept is to only take the type of trades (long or short) that
are favorable to the overall market environment. By doing this we hope to eliminate some losing
trades, increase our profit factor and increase the average profit per trade. This simple concept
may not work for every trading system, but it's a great place to start. Below is an image where you
can see the trading system only takes long trades during a bullish phase (above the 200-SMA)
and only takes short trades during a bearish phase (below the 200-SMA).
So how does this affect our trading system. Below are the results compared to our baseline
system?
Notice we make more net profit with 60% less trades. Our trading has become more efficient as
we eliminate unproductive trades. We do this by generating more money per trade. We
increased the number of winning trades. We reduced the largest losing trade and significantly
reduced the drawdown. We increased the consecutive winning trades and reduced consecutive
losing trades. Everything really looks much better. How does the equity curve look?
I think our results demonstrate market participants are behaving differently between a bullish
and bearish market. We can use this knowledge to improve our system by adapting to the market
- taking trades only in the direction of the prevailing market trend. Overall, I would say this
market environment filter has improved the system a lot. However, I have to wonder, is the value
we used for our SMA filter an outlier? Is it robust?
Once again we are going use TradeStation’s optimization feature to stress test our look-back
period used by our bull/bear market regime filter. This is something I like to do even if I don’t
see much improvement in our regime filter. In this case the results look better after adding out
200-day SMA filter, but even if they didn’t maybe a different value than our default 200 would
work better. Furthermore, I would also like to see if our current default value is an outlier. If it is,
we can’t rely on the improved results we just saw.
The procedure for stress testing is just as we did earlier within Step 4 in this eBook. You can
refer to that section for reference. For now I’m simply going to show you the results of the simple
We can clearly see the values neighboring 200 still produce solid results and all values produce
positive results. We can feel good that our 200 default value for the simple moving average is not
an outlier as well. Notice our 200 value appears to be well within a stable range between 180 and
250. This is great. Over all I feel really good that our 200 value is robust and will work well into
the future.
Our simple moving average filter works well, but we don’t have to stop testing bull/bear regime
filters just yet. There are other techniques to divide a market and some of these might produce
better results than our old reliable 200-day simple moving average. Just about any other
indicators could be used. Some may make more sense than others while others may require a
little creative thinking to make it work. I encourage using indicators differently than their
1. Rate of Change (ROC). Traditionally the ROC is used to show momentum but we can
use it’s to define a bull market when it’s above zero.
2. RSI on a weekly chart. RSI is often used to locate price extremes. But here we can apply
it to a higher timeframe to determine if we are in a bull market when the value is above 50.
3. Smoothed Adaptive Momentum was created by John Ehlers. It’s a complex indicator and
more than I want to get into at this time. Google it if you wish and you can also find the
EasyLanguage code here. But again, this momentum indicator can be used for a bull market filter
when the value is above zero.
4. Relative Strength Ranking (rsRank) is an obscure indicator and you can find a little more
about it on the EasyLanguage Mastery website. I will say this about the rsRank for now,
traditionally it’s used as a ranking tool to compare a group of stocks or ETFs to determine which
specific instrument is performing best. As its name implies, it ranks each instrument based on
how well the instrument has been performing. You can then compare this score to the other
stocks or ETFs in your basket of trading instruments. Thus, you can simply pick the instrument
with the highest rsRank score when creating a momentum based trading system. However, I’m
using it for a completely different purpose. Again, when the value is above zero we are in a bull
market.
All of these indicators are also smoothed to help reduce whipsaws. The Smoothed Adaptive
Momentum indicator by John Ehlers already contains a smoothing element applied to the
indicator. However, to smooth the ROC, Weekly RSI and rsRank indicators I used the same
technique found within the Ehlers indicator. The formula is this:
This same smoothing technique is applied to ROC, weekly RSI and rsRank as well.
The code used for this testing is available in your download package when you purchased the
eBook. Where appropriate we are going to use a 200-day period for all our examples. 200 days
represent about ten months of trading if you figure there are about 20 trading days per month.
It’s also a common longer term period applied to moving averages as well. This is not a magic
For more ideas on using different indicators to act as a market regime filter read this article,
Testing Market Regime Indicators.
Beyond Bull/Bear
Of course you do not have to stop with simply dividing the market into two regimes (bull or bear
market). You could divide the market into three regimes. Bull, bear and range bound. This
begins to complicate your system more, but might be worth looking into.
Another market characteristic which is common for all markets is the strength of a trend. A
market may be in a bull regime, but how strong is the trend? Is the market rising fast or is it a
weak trend. By measuring the strength of the trend we can further divide the market to make our
trading system more adaptive. By introducing trend strength we now create four distinct market
environments that can be used to determine how an automated system trades.
You might be wondering is it always appropriate to break the market into more and more
regimes. While it’s true it begins to complicate your development and I often like to keep things
simple, sometimes it can have great impact on your system so it’s always worth testing. I
currently don’t go beyond four regimes. However, there is one issue that may put the breaks on
dividing the market into three or more regimes. The biggest concern is the number of trades in
your backtest. Each time you break the market into a different regime you reduce the total trades
and the number of trades per regime. Think of each regime as a separate trading system. For
example, our current trading system with our bull/bear market regime filter is really two
systems. One goes long during a bull market. The other goes short during a bear market. Thus, I
would like to see many trades for each “system”. If you divide a market into four regimes and
each regime has several hundred trades, you’re in much better condition than having only 50
trades in each regime. Our example really does not have an over abundance of trade examples as
Action Items
• Add the regime bull/bear market filter to your code and test it on your trading platform. Fill
out SDN sheet “Step 5: Market Environmental Filter “. If you’re a TradeStation user, use the
“Key Reversal S&P ETF” workspace provided with this download.
• Be sure to also test your systems by dividing them by market volatility. This can be an
important factor when building a trading system. But be careful when you start dividing the
market by different regimes that you have enough trade samples.
To properly perform our testing we first need a baseline system. This baseline system will be
used as a benchmark to determine if we have improved our trading system substantially enough
to justify the given modification. What is our baseline system? It is simply the system we created
during Step 5. That is, our key idea with our environmental filters applied. In our case, this is our
key idea with a bull/bear market regime filter added.
As you may have noticed, the key idea uses no hard stops. There is simply a dynamic exit when
price closes above a moving average. Should our baseline system have stops? Maybe.
I never trade without a hard stop when it comes to trading futures. However, are there times
when a hard stop hurts performance? Furthermore, do we always need a hard stop? It’s true that
many buy-and-hold people are trading without stops at all when they purchase mutual funds,
ETFs and stocks within their retirement account or personal brokerage accounts. Think of all the
people you know who are holding securities in their 401(k) without stops. You probably did it
yourself and you may be doing it now. I know I am. It’s true. I’m trading my 401(k) without stops.
Am I crazy? Maybe not.
Keep in mind because there are no hard stops in my 401(k) strategy that does not mean I will ride
the next bear market down with it. Far from it. I trade my 401(k) with a modified version of the Ivy
NOTE: If you would like to learn more about the Ivy Portfolio I do recommend the book,
“Ivy Portfolio”.
As our key idea is currently programmed, it has something similar to my Ivy Portfolio example. It
has a dynamic stop already built into it. So, the question arises, do we really need a stop in this
case? We can better answer this question by performing a study which we will do later in this
eBook. Before we do, let me be clear. When I’m trading futures contracts all my trades have a
hard stop. Based upon the leverage of the futures markets and the types of strategies that are
traded, a hard stop is required. It actually improves performance. So in some cases I use a hard
stop and other times I don’t. Is this not contrary to the popularly held belief that you always need
a hard stop? Are you confused yet?
My point in bringing this up is that hard stops may not be necessary for your system and other
times they are mandatory. In general, leveraged instruments like futures should have a stop or
your position should be hedged. Yet, an ETF strategy like the Ivy Portfolio does not require
stops.
I think it’s important to realize that some systems do not require a hard stop. In fact, some
systems will produce some of the poorest results when a hard stop is introduced. Yet, other
systems require it and actually can benefit from it performance wise. In the end, deciding if a
system should have a hard stop is based upon two criteria:
All the systems have a stop or exit point of some kind. Some are dynamic while others are a
physical hard stop placed at the time of opening the trade. Again, our key idea has a dynamic exit
point for all trades while other key ideas that you run across will have a hard stop. This does not
Baseline Performance
Included with this download is the TradeStation performance report of our baseline system (see
Key_Reversal_Baseline.xls). Below is a performance of our baseline system.
Baseline
Action Items
• Record the baseline performance on the SDN sheet “Step 6: Baseline System”.
Following this important step can go a long way in preventing you from developing a system that
is overly optimized to the historical data. In general, the more trading rules you optimize
together, the more likely your system will suffer from being overly optimized. It’s important to
make an attempt to keep your trading system as simple as possible. If you only take away one
concept from this eBook, make it this one because over optimizing is an area where many people
stumble. To help avoid curve fitting the key is independent testing of each rule.
For example, let's say you have a simple moving average crossover system. You then wish to test
a trailing stop and a trend-strength indicator to filter trades. So we have the following items to
test:
How do you pick values for each of these indicators? How do you optimize your system? We
already know not to pick the best optimization but when you have multiple items to optimize,
how do you go about it? In my opinion you should attempt to perform independent testing
whenever possible. What does that mean? Each new rule that we test must be done independent
of the other rules. While it’s not always possible to completely do, it should be done whenever
possible.
Brick By Brick
Does each rule we test have to dramatically improve our system’s performance? Not necessarily.
Since we are testing rules independently, it’s possible the improvement may not be dramatic.
Yet, combining two mediocre rules together may produce outstanding results. But we won’t
know this during this phase of development because of the independent nature of our testing.
What we are trying to do in this step is to build a collection of rules that show robustness which
we will use later during the development process. We are assembling a collection of building
blocks consisting of various filters, entry methods and exit methods which will be later assembled
into a single system. Think of each independently tested rule as a single brick. Each brick is an
independent unit separated from all the other bricks we are testing. Each brick is closely
examined to see if it’s robust or not. We eliminate obviously weak bricks and keep the more
promising ones. Then in the next step, Step 8, we assemble these bricks into different
combinations producing a final trading system.
Again, each brick is independently tested on the baseline system. We mark which bricks we are
keeping and which ones we are tossing out. Bricks we are keeping have robust values. For
example, if what we are testing is a moving average, we will test the look-back period for this
moving average over many input values. Ideally we will find a stable region and pick a reasonable
Before we start out independently testing different rules to create our collection of bricks, let me
explain a bit more on why we are taking so much time to test our rules in this manner. As you
probably know, this is done to avoid curve fitting or over-optimization.
Curve fitting may not be the best term to use here. Curve fitting from a mathematical standpoint
is attempting to fit a line or curve through a series of data points. That’s not what we are doing
with a trading system. When it comes to trading, we are attempting to optimize a trading
system’s performance. Often this is a key performance metric, such as net profit. We do this by
attempting to optimize various parameters in an attempt to maximize profits. Of course, one
single metric is not all we look at when attempting to optimize a system. We take into account
many different performance metrics such as profit factor, drawdown, and average profit per
trade, to name a few, to help us gauge if a particular system is tradable or not.
When it comes to making a profitable trading system we depend upon finding market edges we
can exploit. All successful trading relies on recurring patterns found in the historical market data
and exploiting those patterns to make a profit. This is true for discretionary traders as well as
system traders. However, much of the historical market movement is noise or random movement.
So, the problem arises when we over-optimize our system to the historical data and unknown to
us, our system begins to key-into patterns within the noise instead of exploiting a true market
edge. Finding a pattern in the noise can produce great looking equity curves on historical data,
but on out-of-sample data the equity curve can break down rather quickly.
In summary, we want a trading system that exploits a true market edge and avoids keying-off
patterns in the market noise. The idea of separating a true market edge from the noise is what
this game is all about.
So how does over-optimizing occur? It’s often accomplished unknowingly as a system developer
continues to tweak the trading system to obtain better results over the historical market data.
Over-optimizing can often occur when:
As you know, the first step when developing a trading system is to divide your historical data over
two segments.
1. In-sample segment
2. Out-of-sample segment
This basic step is required for all system development. The in-sample segment should consist of
the bulk of the historical data. Let's say about 80% of your historical data should be dedicated to
this segment. The in-sample segment is the historical data used when developing your system.
It’s where all your development decisions are made. During your development process you never
apply your work in progress trading system to the out-of-sample segment. This segment is
reserved for testing your final system to see if it can hold up on historical data it has never
“seen".
Out-of-
In-Sample Data Sample
Data
Today
Once your development is done one of the first things you will do is now test the performance on
the out-of-sample segment to see how it performs. Ideally, the system should perform well as the
equity curve advances.
When your system is applied to the out-of-sample data it either passes or fails. While it’s true
that mediocre results on the out-of-sample segment may be OK, don’t start tweaking the
parameters.
Proper Optimization
Optimizing particular strategy inputs, such as a look-back period of a moving average crossover
system, can be a strong source of over-optimizing your system. However, you can significantly
reduce the risk by following a few key points.
1. Don’t pick the best. When optimizing a parameter you don’t want to pick the best value.
3. Ideally, you want to find a cluster or range of values where your system performs well.
5. If a default value looks optimized or not robust, then you may be forced to abandon the
system.
6. The key to optimizing parameters is something I call robustness in the given parameter
you’re optimizing.
Let’s say we are attempting to optimize a look-back period for a moving average. Let's also say
the default value is currently 50. When optimizing this value I want to look for two things. First,
if the default value is optimized. Second, if other nearby values produce similar results. In short, I
would like to test the robustness of the look-back period. For example, if we change the 50-day
default value to 49 or 51, I don’t want to see the system’s equity curve drastically change.
Furthermore, the system should not radically change if I use 47 or 52. The neighboring values
Below is a bar graph of a look-back period optimization covering the range between 20 - 200.
The x-axis contains the look-back period while the y-axis contains the trading system's total
P&L. Remember, our default value was 50 and we want to optimize it.
First, the default value of 50 is far from an optimal value. It appears a wide variety of look-back
values produce positive results. Since we don’t want to pick an optimal value, how do we pick a
value? Notice the values starting from 20 climb in net profit as we reach 80 then level off until
they start to climb around 150. It’s reasonable to pick a value within this range of 80 - 150. Often
I will pick the midpoint. You can calculate this by adding the values (80+150) and dividing by
two. In this case, the value of 115 would be our new look-back period.
You could also make the case there is a stable region between 150 - 200. It might be worth
extending the look-back study to see values beyond 200. Picking a midpoint value within this
region is most likely fine as well. It would be worth testing and comparing the various system
performance metrics to see if there is a reason to pick one stable region versus the other. This
When optimizing a value, you don’t simply want to pick the best looking value. Find a region
where the values appear stable. That is, an area with little variation. Once you find that region,
pick a value within the center of that region. Doing this will help reduce over optimizing your
trading system.
It’s now time to start finding bricks to build our system. To do this we are going to test various
profit targets and stops to see how they behave with our baseline system. Some examples of rules
to test include:
• Hard stops
• Trailing stops
• Profit targets
The Exit Methods Mind Map found within your download package provides some exit ideas.
For our example system lets start with testing a hard stop value. Why a hard stop value? Well, the
hard stop value is not only used by most people, but it also helps to determine some very
important aspects of our trading system including what size of trading account we will need to
trade.
Remember at the very beginning we talked about the concept of having both a set-up as well as
an entry technique? Recall that our key idea had no entry technique. Instead, it simply required
us to open at the open of the next bar. Well, this would be a good place to test various entry
[Link] could try out entering on limits or stop orders. You could also try using a
different time-frame to pinpoint an [Link] topics will be beyond this eBook but during this
phase of development it would be appropriate to explore these ideas.
My favored method to determine a hard stop is to use TradeStation’s optimize feature just like we
did with our stress testing of the input values. Here we can test a variety of stop loss values and
see how they affect the performance of our key idea. I’m going to test a hard stop value from zero
to $5,000 in increments of $200 to see how it affects our system’s performance. Why am I
starting from zero? Well, the input value of zero is going to tell the code not to use a stop. In
other words, it’s going to only exit based on the exit rules provided in the key idea. Below is the
bar graph of the results in net profit.
Looking at the Net Profit bar graph below, you can see the large bar on the far left side. This is
our system with no hard stop. Immediately to the right of this bar we have a very small net profit
with stops values of $250 through $400. As we increase the size of our stop value, our net profit
What is this telling us? The built-in exit rule is our primary method of exiting both winning and
losing trades not a hard stop value. A hard stop value in this system is best set a great distance
away as a catastrophic stop loss. A value of $2,000 should be about right. Remember, as we are
building this system we are dedicating $20,000 to our trade. Thus, we are setting aside a
$2,000 stop for our $20,000 in trading capital. This means we are risking 10% of our trading
capital per trade. This may be rather high and brings up a good point that many new traders fail to
realize. Determining account size is a function of the stop loss applied to each trade. Unknown to
many newbie traders, the stop loss is determined by the system not by the trader! Furthermore,
this stop loss value determines the size of your trading account. It has been written that account
undercapitalization is a leading cause for trader failure.
Why do so many people fail at trading? It is a reasonable estimate that many people don’t follow
proper risk management for the simple fact they have an undercapitalized trading account for the
type of trading they are doing. Sure, some are unaware of risk management and others just simply
ignore it. But often the root cause is you simply don’t have enough money to be trading. As
stated above, your trading system determines your stop loss and thus, the size of your account.
You just can’t trade any trading system. And you can’t just simply pick a stop which you think is
right. Again, both your account size and stop loss are linked to the performance of your system.
Let’s use our current system as an example.
We just determined that it appears our current system may require a stop loss value of $2,000.
We can now use this value to determine the account size required to trade such a system. It’s a
common understanding that any one particular trade should risk no more than 2% of your
account. Our stop loss value is the dollar amount we are willing to risk on each trade. With some
basic algebra we can take this information and solve for our required account balance.
So we come to the value of $100,00 for an account size. The general formula for determining
account size is:
Of course this is slightly generalized. Some people will only risk 1% per trade while others are
OK with 5%. Again, this comes down to personal tolerance and characteristics of the particular
system and what you're comfortable with losing. But a 2% value is a good general value.
As an exercise in building a system, distance yourself if you believe this exercise is not
worthwhile. What’s probably going through your head is, I would never trade this system which
requires a minimum account size of $100,000. True, you may never trade a system like this but
the lessons are still valuable. Furthermore, as you become more experienced in developing
systems you have other options which you can apply. First, you can move to lower timeframes
which reduce your stop loss thus, reduces the required account size. I personally have a trading
system on the futures market where signals are generated on a daily chart but I use a 5-minute
chart to pinpoint entries and place my stop. In this case, I’m able to reduce the stop to only
$500. This also significantly reduces the size of your trading account needed.
Another idea is to switch instruments, if possible. If you’re building a system for the futures
market, maybe the system can be modified to trade the ETF equivalent. This would be another
way to reduce the stop loss value.
As you can see, there are options in reducing the risk per trade when initially looking at a trading
system on a daily chart. However, these techniques of trading using multiple timeframes and/or
switching to a correlated instrument will not be explored in this eBook. The big takeaway for this
chapter is when you are determining your system's stop loss value, you are also determining how
big of an account you will need to trade the system successfully.
Remember back when we were stress testing the input values? I made a note about how we may
not want to use a simple moving average for an exit because there are times when trades are taken
above the moving average. This makes me wonder if a simple time based exit would work just as
well or even better. To test this I removed our current exit from the code and replaced it with the
following.
If MP = 1 and BarsSinceEntry = NumBars Then Sell("X Bar Exit") next bar at market;
If MP = -1 and BarsSinceEntry = NumBars Then Buytocover("X Bar Exit ") next bar at
This code will exit a trade after NumBars (an input value) days have gone by. Remember, we exit
a trade at the open of the next bar. So a value of 1 will mean we hold the trade for 1 day and close it
at the open of bar 2. A value of zero means we close the trade the day after we open it. To the left
is the bar graph of the hold
period from 0 to 20 versus
net profit. Also keep in
mind, we reverted back to
our baseline to make these
changes! That means the
hard stop is not used in this
test. We are isolating our
change to the exit method.
We are independently
testing the exit method from
our hard stop.
Remember how market participants will behave differently between a bull and a bear market? It’s
the very reason we used a 200-SMA to divide the market. Well, another way to look at our exit
method is to separate our bull market from our bear market and see how it affects our holding
period. Perhaps during a bear market it’s much better to have a holding period that is shorter
while during a bull market holding longer is better. We can easily test this by testing the bull and
bear markets individually.
Below are two bar graphs that depict the net profit versus the holding days. The first graph shows
trades taken during a bull market and the second graph are trades taken during a bear market.
The bull market graph looks rather flat once you get beyond 6 days.
The bear market graph (below) has a much more defined optimal region. Values between 8-10
clearly produce the best results as profit drops off sharply outside of these values.
It’s interesting that our median value of 10 falls within the stable region for both bull and bear
trades. However, we can clearly see there is a difference between bull and bear trades. This is
unlike the bull market as it appears to be much more forgiving. That is, you can hold trades at
different time intervals and still get similar results. This tells us once again that there is a
difference between trading on the long side and short side! But can we use this to our advantage?
Maybe. We will note this information in our SDN for future use. At this stage of the game we are
analyzing potential exit candidates and noting anything interesting. In this case we explored
three possible exit methods
1. A single N-bar exit method for both long and short trades
In the end we will record these results in our SDN and make a note that this exit method may
have some value.
RSI Exit
Let's look at using RSI to exit. A two period RSI is often used to enter/exit trades by
highlighting short-term extreme moves. Let's use a two-period RSI calculation and exit trades
when the values are 90 or above for long trades and 10 or below for short trades. These two
values are our trigger thresholds. I will use TradeStation’s optimization feature to test the
robustness of these thresholds to exit a trade. I will test values from the range of 1-20. What does
this mean? This will be the extreme range where a trade will be closed. For example, a threshold
value of 5 means a long trade must be closed when the 2-period RSI calculation is above 95 (100-
threshold) or a short trade closed when the 2-period RSI calculation is below 5 (threshold).
Below is a bar graph depicting net profit versus exit threshold.
Looking at the bar graph below we can see nearly all explored values produce positive returns.
There is somewhat of a stable range between 2 and 11 which produces a median value of 6.5. We
can round this up to 7 . This value is far from an optimal value and appears to be a reasonable
pick.
When comparing the bull market and bear market graphs we can see they behave strikingly
different. The bull graph (top) produces the best profits with a tight threshold value of 2 and
tappers off. On the other hand the bear graph (bottom) needs a very wide threshold value. It’s not
until you get beyond 15 does net profit jump well above $2,000.
What does this tell us? Based on the two graphs I would have to say that a wider threshold on the
shorter side is needed as opposed to the bullish side. In other words, if we were using a two-
period RSI as our exit method we may want to consider having our trading system use two
different RSI thresholds based upon the market regime. A tight threshold value of 2 on the bull
side means we are looking for a strong move in our favor. A wider threshold value of 18 on the
bear side means we are looking for a less dramatic move in our favor. I’m guessing we exit
quicker on the short side and are willing to hold longer on the long side. Looking at the
performance report generated by TradeStation I can confirm this. Long trades are held on
You can repeat this process for all types of exit methods: trailing stops, ATR stops, profit targets,
and a whole host of indicators. Likewise, you can then use the same procedure to conduct a study
of entry methods which can include limit orders, stop orders, and delayed entry methods.
Summary
As you can see, you can spend a lot of time in this area examining the characteristics of different
exit methods and how they behave. In this eBook I’ve only demonstrated a couple of examples to
explain the process. I hope it’s clear to you. I would urge you to explore as many exit methods,
entry techniques and filters as you see fit. Again, this may take a lot of time but it can really help
you understand how your system behaves. Keep in mind, each new rule you explore should be 1)
independently tested and 2) results recorded in its own SDN sheet.
Action Items
• Make as many copies as needed of SDN Sheet "Step 7: Independent Testing" to record your
results for all the rules you're testing.
• If you like, you could spend some time testing various entry techniques to complement the
basic setup.
Sometimes when adding a rule you see solid improvement in the performance results. In this case
you can keep it. Other times adding a rule does little to help the system performance. In this case
it’s best to keep things simple and eliminate the least productive rule. Be selective and only
include a rule if it clearly helps. If it does not significantly improve the system's performance, it’s
probably best to leave it out.
You can start with testing different combinations of exit methods with each entry method. In our
Key Reversal system I might simply try the 2-period RSI exit using the two different thresholds
for the bear and bull markets. This would be one test case. With each test case the results should
be recorded under SDN sheet "Step 8: Combine Rules".
Go through all the trading rules from your independent testing and start combining them.
Another question you might be asking, what’s considered a “significant” change to the
performance? For example, let’s say a single rule introduced to the baseline system reduced the
number of trades by 1/3 or even 1/2. At the same time the profit factor increases from 1.07 to
1.50 and the average dollars per winning trade increases from $20 to $60. To me this is
significant because we reduced the number of trades by removing unprofitable trades. This may
be an exaggerated example, but it could happen. Generating fewer trades and creating more
dollars per trade is usually always a great idea even if the system is making less net profit. Keep an
eye on the drawdown as well. Maybe a rule does not significantly improve the net profit but does
Here is a list of system performance statistics that I keep an eye on when making the
determination to keep a rule or not. These are also great at evaluating your final system.
• Number of Trades – Lower is often better because it keeps transaction costs down. Yet
too few trades are not good either. From the point of view from statistics, you probably
should have at least 100 trades. If you have too few trades you’ll have too many rules or
you’ll need more backtesting data.
• Steady Equity Curve - I like to look at the equity curve to see if there are long periods
between new equity highs. An equity curve that slowly climbs and has short durations
between new equity highs is preferable. Likewise, shallow pullbacks are better than
deeper ones. Always ask yourself if you could trade this system without much emotional
trouble if it has long periods where the equity does not grow or even declines.
• Average Profit Per Trade - Like profit factor, the average profit per trade tells me if a
system is making enough money on each trade. When designing a trading system I like
to see an average profitable trade above $50, before commissions and slippage are
deducted, at an absolute minimum. If the average net profit is above $50 with
commissions and slippage deducted, that’s even better. The higher the average profit per
trade, the better.
• Profit Factor – While net profit can be a factor in your decision about a particular trading
system, profit factor is often even more important in my opinion. Profit factor measures
the efficiency of your trading system. Profit factor is calculated by dividing the generated
profit by the generated losses. A profit factor of 1.5 indicates for every two dollars lost,
three dollars are gained ($3 win / $2 lost = 1.5). Obviously a number above 1.0 means
you are making money. I like to see a profit factor of 1.5 or higher.
Now that I’ve explained the numbers behind a significant change there is something else very
important you must consider. Is the system tradable? What does that mean? <explain looking at
EQ curve>
Keep It Simple
Keep in mind that simple is often better. When possible, keep the number of parameters and
trading rules to as few as possible to accomplish the task. Introducing more trading rules
increases the likelihood of over-optimizing the system to the historical data. If you introduce a
trading rule to a system, it must have a significant positive impact on the trading rules in order to
justify keeping it. If the results are deemed mediocre, then throw out the rule.
Let's first take our baseline system and review what we did in Step 4 when we performed a stress
test on our default input values. Reviewing our SDN for the exit period we see:
• The look-back period for the simple moving average exit is 10. This values is not an outlier
or outside of a stable range.
• The look-back period for the key reversal calculation is 10. This values is not an outlier or
outside of a stable range.
Our job is easy here. We don’t have to modify our default values. I would leave them just as-is. If
we discovered the default values were overly optimized (outliers) or did not exist within a stable
range, we would change those values now. However, our default values look fine so we can move
on.
Reverting back to the baseline system I’m now going to use our N-Bar exit. Based upon our
independent study we determined an N-Bar exit had a stable range between 8-12. Picking the
midpoint value of 10, the results with this exit are below.
Looking at the results of our 10-bar exit we do generate a tine more net profit and we have a tiny
increase in annual return. But does that alone make this a better system? We have fewer winning
trades and a larger average losing trade. Let's look at the equity graphs to see which system
appears to be more tradable.
Baseline
Which equity graph looks like it would be easier to trade? I think the baseline looks better. The
equity graph is smoother, has less delay between new equity highs and rarely dips blow the zero
line. This is a great example of why looking at the equity graph can be a big help and also why you
can’t just judge a system based upon its net profit or annual return. In this example we can see
Reverting back to the baseline system I’m now going to use our RSI exit. Based upon our
independent study we determined an RSI exit had a stable range between 2-11. Picking the
midpoint value of 7, the results with this exit are below. Remember, this means we are existing
when the 2-period RSI crosses above 93 for longs and below 7 for shorts.
The RSI exit does not help much. We make more profit but lower profit factor. Reviewing the
performance it appears this works well for long trades, but not as well for short trades. This is a
bit disheartening as we have spent a lot of time studying the robustness of the input values and
performing independent testing on two exit methods. What do we have to show for it?
At this point we have discovered that a bull/bear market regime filter really improved our system.
Looking at our market environment filter section of the SDN, by just adding the environment
filter we have a dramatically better system. As a reminder, below is the performance difference
between our key idea and the key idea with the regime filter applied.
Next, during our stress testing of the default input values (the exit look-back period and the
reversal look-back period) and environmental filter, we found all parameters were both robust
and non-optimized. This was a great vote of confidence that our system is not overly curve fitted
to the historical data. A big plus!
So, we are really far from completely testing our system. However, at this time I would like to
show you another technique that may help us out.
One thing that became apparent to me while testing this system was the difference on how long
trades and short trades behave. This is not too uncommon and makes sense as we’ve talked
about before. Bull market participants behave differently than bear market participants. Now, if
market participants act differently during these two regimes, should we have a system that adapts
to this difference? Why should we trade the same way during a bull market versus a bear market?
You can bet discretionary traders note a difference between the two regimes and adjust their
trading. Why shouldn’t we?
So, we want to make our system more adaptive to the environment. We want our trading rules to
be tailored to which market we happen to be trading in. In short, we want to develop two trading
systems! Think of it this way. Once we apply our market environmental filter we essentially broke
our historical data into two segments. The first segment is an environment favorable to long
trades (above the 200-SMA) and the second segment, an environment favorable to short trades
(below the 200-SMA). During our first pass through the development process we work on the
long trades only. Once this is complete then work on the short side only. We may use the same
key idea, but our exits and entry trigger may be different.
By doing this you are doubling your work as you are running through the entire process
described in this book twice! Once for long trades and once for short trades. In essence, you're
creating two separate trading systems. This can work wonders for your system because as we
talked about before, market participants behave differently during bull and bear markets. It
First, it would be a lot more work for you and me. At this point I’m just trying to explain my
process and that would just complicate things. I’m revealing it now for completeness.
Second, you are taking on some risk when you break a system into two different systems. Why?
As stated above you are basically building two different trading systems. One for the long side
and one for the short side. What does this do to your number of trades? It breaks them apart
between the two systems. Thus, you significantly reduce the number of historical trades for each
system you’re developing. For some systems you will still retain a good number of trades. But
with some, including this one, you may begin to feel you don’t have enough historical data. Let's
use the SMA exit as an example.
Based upon our current testing it seems the original exit method of using a simple moving
average produces the best results. This exit has two important factors. First, it's a dynamic exit
expanding and contracting with market volatility. Second, it’s simple. However, can we better
adapt this exit method based upon the market regime? Using our baseline system and
TradeStation’s optimization feature to test the look-back period on our simple moving average
during both a bull regime and bear regime, let's see what we discover. I’ll use a new “Stress Test
Parameter” SDN sheet to record our parameter study. Below are the results.
Our discovery is not totally uncommon. Some setups just perform poorly in particular market
regimes. The obvious issue with our current system, if we decide to make it a shorting-only
system, is the low number of trades. This is often the price when working from daily price data.
With these two values I’m now going to create a version of the key reversal system that changes
the exit period based upon the market regime. This will make our system more dynamic to the
given market conditions. Below is the performance followed by the equity graph on the next
page. This also includes our stop.
Baseline Combined
System SMA
Again, there are still many things we could test. However, I think you probably have a very good
understanding of the process so far. So now, it’s time for you to take some action and then we’ll
move on to the next step.
Action Items
• Test the key reversal exits and test the difference between bull and bear market exits.
• With other systems you have worked on, you may find that a particular key idea will shine
under a particular market condition. Make a plan to test other systems you’re working on!
• Do we make new equity highs or does the equity curve remain flat?
What we want to see is an equity curve that has characteristics similar to what we see during the
testing period. In other words, drawdowns and new equity highs should look similar. If they do,
you have created a system that just might be a winner. However, if the equity curve deviates from
what you see from our historical data, such as dramatically breaking, it’s time to get back to the
drawing board.
Sometimes you will have a system that goes flat on the OOS data. Does this mean it’s a failure?
Well, not always. If your historical test period, the in-sample data, had similar flat periods then
the current flat period may very well be normal. I would be more suspicious if you had a steady
rising equity curve on the in-sample data and then saw a flat period on the OOS data. A flat
period or even a drawdown on the OOS data does not automatically disqualify the system as long
as it’s within the parameters of what was observed during the in-sample period.
So how did we do? On the next page is the equity curve of our Key Reversal system over the
entire date range including our out-of-sample period. If you will remember, the testing period
generated 84 trades. Looking at the new equity curve all trades after the 84th trade are occurring
on the out-of-sample data.
Over all, OOS data is making new equity sights in 2019. I like to see a minimum of 30 trades on
our out-of-sample and we have that. However, there is one issue. Our OOS data is only during a
bull market. We really would like to see how our system performs on the out-of-sample data. But
that’s not always possible as in this case. The market has continued to march higher. Nothing in
trading is perfect!
Over all this looks very good and we did our best to avoid curve fitting. Nicely done!
What happens if you are not happy and wish to modify the system in an attempt to improve it?
This is certainly a possibility. Keep in mind that once you have applied your trading system to the
OOS data, that data is no longer “unseen”. Like opening that can of soda, it can never go back
to its pristine or fresh state. The OOS data is now tainted. How can this be? Well, once you make
a decision to adjust the trading rules because you are not happy with how it performs on the OOS
data, you are introducing a bias into your development. What this means is you are using this
“unseen” data found in the OOS test period to make decisions on building the system during our
historical testing period. You are allowing information from the OOS data segment to seep into
the in-sample data. Think of it as peaking into the future to make trading decisions today!
Clearly, this is not right and can lead to the dreaded over fitting your system to the historical data.
So how do you avoid this? The first and most obvious way is don’t do it. Once you have tested
your system and it fails on the OOS testing, it’s time to start over with a new key idea. This is the
most extreme fix but it helps ensure you will avoid historical bias.
A second way is to take a gamble by going back and recombine your different independent rules.
Maybe a rule should be eliminated. Again, don’t start introducing dramatic changes. Keep in
mind the act of going back and tweaking the system can increase the chances of you over fitting
your system to the historical data. An option to help avoid this issue is to apply a third set of data
to the final system. This third set of data could be live market data (covered in the next chapter)
or simply waiting for the required number of weeks or months to pass and using that data as the
new OOS. Obviously, this is much more practical on trading systems that operate on smaller
timeframes. For daily charts you might be waiting for a few years to collect enough data!
The third way is one that I use if I’m in this situation. It’s also something I do even if my system
performs well on the OOS data. That is, use a Walk Forward Optimizer (WFO) to test the system
on the OOS data. Many trading systems need to have their parameters adjusted over time as the
market dynamics change. WFO analysis optimizes a trading system on a segment of historical
data, selects the best parameters and tests those parameters on an out-of-sample segment. There
are times when a newly created trading system does not do so well on the OOS data, but when
WFO analysis is done, the performance on the OOS segments do very well. Some systems don’t
The entire topic of WFO is far too complex to get into here, but I wanted to make you aware of it.
If you would like to learn more you can start by reading an article on the subject called, " What Is
Walk Forward Optimization", on the STS website. I hope to create future articles in the near
future demonstrating how to use it. It’s a great tool and I encourage you to read up about it and
experiment with it.
It’s time to take a look at our trading system’s performance vs our benchmark. Our benchmark is
simply purchasing $20,000 worth of SPY on October 17, 1997 and selling it on March 31, 2014.
This resulted in a nice $19,290 in net profit vs our $10,344 in net profit. Looking at annual rate
of return we see a 4.1% annual rate of return (without dividends) for buy and hold vs our 2.5%
annual return. However, if we take our profits and add them to our trading capital we can push
our net profit to $13,443 with an annual return to 3.12%. Remember, during development we are
not adding our net profits to our trading capital. But as a trading system, it’s likely we will do this
to compound our winnings.
Buy and hold clearly beats our system when compared on total returns and annual rate of return.
But is that all there is to consider? Holding SPY through the dot com bust and through the
financial criss of 2008-2009 was very scary. You would have been on a powerful and steep roller
coster. Could you have held on or would you have bailed? Holding positions during these
difficult times can be extremely stressful.
As you can see our system is far from a money machine, but it certainly has some appeal when
looking at time exposure in the market. The Key Reversal system is only in the market 8.2% of
the time. The vast bulk of the time it’s sitting in cash or could be moved to another instrument. It
almost seems to me a system like this is best for those who can’t or don’t want to be exposed to
the market very much. This might include people who need access to their cash. Buy and hold
can be great if your investing time horizon is long enough to hold through several years of steep
drawdown. Having your cash locked up in a buy and hold strategy may not always be practical.
In the end, comparing our system gives us another way to look at our trading system’s
performance.
If you feel your system looks solid on the out of sample data, what else can you do to help build
confidence that it will work on the live market with real money? How confident are you that it will
continue to profit in the future? At this point I would like to highlight a statistical based metric
that can be used to help indicate the likelihood a given system will continue to generate profits in
the future.
A complicated approach would be to use the Monte Carlo method, but not everyone has access to
this yet, we all have access to a simple calculator. By visiting a topic in statistics called Confidence
Intervals (CI) we can obtain a hint at what’s possible and perhaps find weaknesses in our
seemingly profitable trading system.
The average net profit of a trading system is simply the historical P&L for each trade over a given
time period. Using our OOS performance report for our Key Reversal trading system we have 91
trades. Some of the trades are winners and some are losers. We add together the total P&L for
each individual trade and divide it by the number of trades – 91 in this case – and we get $113 per
trade. Clearly this is well above zero so in the long run, this system appears profitable.
However, we also know that individual trades can be very different from our average profit per
trade. Some trades produce much larger winning trades while others produce smaller winning
trades. Still, other trades produce a range of losing trades. If we graph each trade’s P&L and then
draw a line representing our average P&L we would see each individual trade falls around our
mean (average) value of $113. In other words, the P&L for any given trade will fall around this
Statistically speaking, a trading system that exhibits a large standard deviation of profit per trade
will have an increased chance of failing in the future. This is true even if the average mean is
currently profitable. But what makes a standard deviation too large? This is explained below
when we attempt to use our confidence interval to estimate a likely range of average P&L values
into the future.
What we wish to do with our confidence interval is estimate, with 95% confidence, if our system
will likely produce a negative average P&L into the future. In other words, is it likely our
seemingly profitable trading system is based upon chance? We can estimate this with our CI
formula.
CI = t * SD / squareroot( N )
CI = Confidence Interval
t = T-score (we estimate value to be 2 and the reasoning behind this is beyond this article)
With our imaginary trading system we have a $100 average net profit and 60 trades in our
sample. Please note that in order for this method to work, you must have a minimum of 60 trades
in your sample. Let’s also state the standard deviation for all trades is $400. With this
information we can compute our 95% confidence interval.
CI = 2 * $306 / squareroot( 91 )
CI = $612 / 9.539
CI = $64.16
For simplicity let’s round the confidence interval to the nearest dollar which is $64. What do we
do with this value? We create a range or band around our average net profit value of $117 by both
adding and subtracting the CI value.
We have now created a range of $49 - $117 for our average net profit. What does this mean?
Based on our calculation we have estimated with 95% confidence that our trading system’s
average net profit could be as low as $49 or has high as $117. The important number is the lower
band because this represents a worse case situation. In our example, we have a positive value
which indicates our system is likely to remain positive in the [Link] this lower value was a
negative value or near zero, we have a system where the average net profit could be based upon
chance and in the future could produce a negative P&L.
In the case of the confidence interval there are two factors at play which are critical. Those values
are the number of trades (N) and the value of the standard deviation between trades. Modification
of the standard deviation can be achieved by altering the trading system logic. Modifying stops,
targets and other trading rules will change the standard deviation value. The goal would be to
tighten the variation of each trade to reduce the standard deviation. This in turn, would create a
smaller CI. However, if you don’t want to modify the system or if you are unable to modify the
system there is another way: find more trades! This might mean waiting for enough time to pass
to collect more market [Link] more data points (trades) can have a significant impact on the
CI calculation. For system trading there are many reasons for having a large number of trades. Of
course continuing to add more and more trades is not going to turn a losing system into a good
system. The point here is sometimes you need to have more data before making an informed
decision. If you have what you believe is a good system, yet you only have a few data points, the
CI calculation may be warning you to get more trades in the test sample.
CI does not indicate if this system has been curve fitted. If we have a killer system with 1000
trades with a CI range of $100 – $200 that’s great. However, it’s pointless if the system is curve
fitted to the historical data and there is no way our CI calculation can tell us. That's why it's so
critical you follow the independent testing steps and proper optimization steps outlined in this
eBook. But even if we have a solid system that is not curve fitted to the historical data our CI
calculations are no guarantee of success in the future. The markets are dynamic and changing
and it’s possible the distribution of trades will change thus altering our average trade and
In trading there is no guarantee for future results – ever. The point of testing a system is not to
prove how much money it will make in the future. The goal is to find reasons why not to trade it.
The goal is to find weaknesses so we can address those weaknesses now before we have money on
the line. Our job as professional system traders is to manage risk which means eliminating risky
actions. That’s all.
By using CI we have another tool to find weaknesses and ultimately give us more confidence that
a particular system will likely continue to success into the future.
The subject of testing trading systems is a vast and growing topic that can get very complex.
From hypothesis testing, data-mining bias, cross validation and bootstrapping, you can get deep
into this subject very quickly. I encourage you to explore these subjects down the road. For now,
I would focus on build your systems following the guidelines within to avoid over-fitting your
system to the historical data. Below are two other methods for testing your system which are very
common. To expand your testing knowledge beyond this eBook I would recommend you study
these topics next.
Action Items
• Perform your out of sample testing, review the results including the equity curve. Note your
results and observations on SDN sheet “Step 9: Out of Sample Testing”.
• Review your results on the OOS and live testing segments to ensure you hand handle the
drawdowns, time between equity highs and other aspects of the system. Match your
expectations with reality.
This is a rather simple process of loading your strategy into your trading platform and allowing it
to execute trades on live market data in simulator mode. This is the closest testing you’re going
to get without using your own money.
What you wish to see is your trading system performing as expected. That is, identical to its
historical performance. I would examine the trading performance of the live data and compare it
to the performance statistics of the backtested data (this can include the OOS data as well). It
should be similar. You should see both winning trades and losing trades that all fit within the
parameters of the backtested results. The equity curve should behave as it does in the historical
data; slight pullbacks and pushes to new equity highs. Remember the maximum drawdown while
reviewing the historical results? Within this test period I would be very suspicious of a system
that breached this maximum drawdown value.
It is during this time you may see coding logic problems appear. One that got me a few years ago
was attempting to close a trade on Friday afternoon with a newly created futures trading system.
The code worked well during the historical data but on the live market there was a little glitch.
The system was designed to close any open trade at the end of the trading day on Friday. It used a
market order to exit. Unfortunately the system needed to close the trade at the open of the next
bar. The key phrase here is “next bar”. Well, the market was now closed for the weekend, and the
next bar would not appear until Sunday afternoon. So a trade which was supposed to close
Friday, did not close until Sunday afternoon. Not exactly what I wanted. These types of coding
The next question is, how many trades do you want to see on the live market before you consider
trading with real money? There is no hard set answer. For a guideline, a minimum of about thirty
trades should be collected in the live market. For active systems this will not be much of a
problem since such a system may generate dozens of trades in a couple of weeks while more
longer term trading systems may take months or years to generate thirty trades. In our example
system, we may have to wait year! In reality, when it comes to systems that trade on daily charts,
like our Key Reversal system, it’s often not as critical to collect as much data as you should on an
intraday system. Software glitches tend to have much less impact on the daily chart. In the end,
any software error can often be corrected with manual intervention with little harm. I would
recommend you trade the first couple of trades with a minimum position size, just in case.
The next step, Position Sizing, can actually be performed in parallel with your live market
testing.
Action Items
• Record each live trade within a spreadsheet. Make observation and notes on anything that
might be an issue. Use SDN Sheet “Step 10: Live Market Testing”.
I’ve written about risked based position sizing in an article on my website called, “Risked Based
Position Sizing”. However, for your convenience I’m going to reproduce it below.
Let's start with a thought experiment. Let's say we have two traders who are each given identical
trading systems to execute on the SPDR S&P 500 ETF (SPY). Because both systems are
identical copies of an automated trading system, both trading systems generate the same buy/sell
signals, use identical stop loss and trailing stop parameters. These two trades are also going to
start trading on the same day, with the same starting account size. In essence, let's pretend we
have two traders that have identical trading circumstances. At the end of the trading period we
would expect them to have the same account balance, right? But at the end of the trading
period, reality has made fools of us. One trader generated nearly twice as much net profit. How
can this be? Reviewing the trading results for both traders you can see both traders took the same
signals. Both traders have the same win/loss ratio and even the same number of stop-out trades.
But there is one difference. One trader used a simple mathematical formula to determine how
many shares to buy. This formula utilized a risk-based metric to calculate the number of shares
thus, help to protect the trader from risk. His account grew at a much faster rate.
Position sizing is an integral part of both system trading and discretionary trading alike. It
answers the question, how many shares or contracts should I buy? Too often this question is
answered by nothing more than an educated guess. Too many traders position size their trades
For most people they simply use a fixed number of shares or dedicate a fixed dollar amount for a
trade. Let's think about this for a moment. When a trading system generates a new buy signal
you could risk all of your cash on that one signal. If you win you might double your money very
quickly. Yet if you lose, your account it at zero. It's probably not a good idea to bet all of your
account on a single trade, right? Most people understand this extreme example and thus, will not
bet their entire trading equity on a single trade. It's way too foolish. So, how much do you bet?
Maybe 90% of your account? Or maybe you decide 100 shares is about right. What many
system developers fail to see is how their trade size affects the risk and reward of the given trade.
As you risk more, the payoff can be greater. However, risk too much and you can be quickly
wiped out by a string of losing trades. There exists a theoretical optimal balance between the risk
and reward based upon the given trading system. Your job is to find a near optimal level that is
comfortable for you as a trader.
While talking about the nuances on finding an optimal position sizing method for a given trading
system is well beyond this article, I'm going to demonstrate the difference between using a
risked-based position sizing algorithm versus simply buying a fixed number of shares. Buying the
fixed number of shares will represent what most people do (not taking into account a risked
based metric to properly size their trades for the given market conditions). Does using a risk
based position sizing method really help? Let's find out.
We will be using a simple strategy model for our demonstration on position sizing algorithms. I
chose a simple RSI based system since most strategy traders have experience with this type of
setup. I based our testing on daily SPDR S&P 500 ETF (SPY) data going back to 2/1/1993.
Trades are executed on a daily bar and all trades are long only. Calculations are performed at the
end of the trading day and orders are placed at the open of the next trading day. In all cases, we
assumed a starting capital of $50,000. $20 for commission and slippage was accounted for each
round trip.
Exit When:
2-Period RSI > 70
The fixed-share method is a non-risk based method. In this case we simply buy the same number
of shares (200) for each signal.
The fixed-dollar method is a non-risked based method. In this case we simply dedicate the same
dollar amount ($10,000) for each signal.
The percent risk method is a risked based method. In this case we determine a fixed percentage
of our equity (2%) to risk on each signal. This dollar amount is then divided by the dollar amount
you are willing to risk on an individual trade. If our trading system had a known stop value we
would use this value in our calculation. However, since our demo trading system has no stop
value we are simply going to estimate a dollar number based upon the security's price. Let's use
5% of ETF price as the amount to risk. For example, let's say SPY is trading at $100 and we are
willing to risk 5%. Thus, we are risking $5 ($100 * .05 = $5 ). As the amount we risk climbs, we
reduce the number of shares to purchase.
The percent-volatility method is a risked based method. In this case we determine a fixed
percentage of our equity to risk on each signal. For our example we are going to risk 2%. We
then take this dollar amount and divide it by a multiple of the security's 10-day average true
Results
In this chapter I wanted to demonstrate the difference between non-risked based position sizing
methods (Fixed Shares and Fixed Dollar) and risk-based methods (Percent Risk and Percent
Volatility). I also wanted to highlight the importance of understating risk management and
implore you to use it!
As you can see our demo trading system generates improved trading performance with the risk-
based positions sizing methods. You can see this with the slight increase the Profit Factor scores
as well as the increased profit per trade. Most traders simply use the non-risk based methods or
guess the number of shares. Such a strategy may not be optimal for your trading system. It will be
important to test different position sizing methods to find out what works best. In our simple
example system, it appears the Percent Volatility model holds a better edge.
While this is far from an in-depth look at position sizing and how it can be used to improve your
trading, this should highlight the impact a position sizing model can have on your system.
Looking at the table above you can see the Net Profit can vary from $11,034 to as high as $25,300
depending upon which model you use.
When it comes to adding a position sizing model to our trading system I want to remind you this
is a last step! Don't ever use position sizing during your development to skew your results. I was
hesitant to include this because position sizing is really removed from the development process.
It's something we add later after we have built a successful system. I guess what I'm saying is,
don't add a position sizing model to a poor system. That's not what it's for. A position sizing
model is designed to make a good system even better.
The first thing to remember when it comes to our example system is we are not applying our net
profit for each trade to our account equity. Every trade that is made is based upon our initial
starting equity of $20,000. If we simply add our trade profits to our account and use this total to
compute the number of shares to buy, increase our annual return to 2.53% from 3.18%. The
results are below.
No With
Position Position
Sizing Sizing
But for now, position sizing is well beyond this eBook and could easily become its own topic. But
you should be aware of it. Again, it’s a very final step of system development and if you’re not
Along with position sizing as the holy grail you'll also find the concept of preserving your capital.
While all systems will be expected to have losing trades and periods of drawdown, if your system
should fail on the live market you will need to have a plan in place to minimize the damage to your
account. Not losing too much of your trading capital is critical to long-term success. We all know
that losing 50 percent of your trading capital means you have to make 100 percent just to break
even!
How do we preserve our trading capital if our trading system begins to fail? In essence, you will
want to quantify your system's results. Then monitor your live system's performance looking for
deviations from the norm.
Let's first determine what failure is for our Key Reversal system. Early warnings might include
exceeding the maximum consecutive losing trade or an equity curve that enters a flat period
which exceeds historical extremes. But we can get more specific. In the case of the Key Reversal
system we have the following expectations:
All of these values listed above can be found on the "Performance Summary" tab of any
TradeStation strategy report. Since our trading system dedicates 100% capital to each trade and
we never accumulate profit, we know each trade represents $20,000 at risk. From this we can
turn all of our dollar based metrics into a percentage. How? By taking the above values and
dividing them by $20,000.
How to Build Profitable Trading Systems Using EasyLanguage v1.05 Page 100
Why did we turn these into percentages? So you can have the flexibility to estimate these critical
values with different trading account sizes. For example, if you start trading this system with a
$100,000 account and it's currently accumulated $50,000 in net profit before experiencing a
$5,000 losing streak, you might ask, is that drawdown out of the ordinary? Well, a $5,000
losing streak of trades on a $150,000 account is a 3.3% drawdown. Clearly this value is within
our maximum drawdown tolerance, which is 5%. So, a $5,000 drawdown is expected and
nothing to be concerned about.
Using the standard deviation of losing trades we can begin to see when our trading system might
start to fail and turn south. Looking at the TradeStation's performance report, second tab
labeled "Trade Analysis", you will find the average drawdown and the standard deviation of
drawdowns. In our example system those numbers are the following:
Using the standard deviation of $382 for drawdown and knowing our average drawdown is $331
we can estimate that 68% of our trades should have drawdowns within $714 ($331 + $382).
Likewise, 95% of our trades should have drawdowns within $1,095 ($331 + 2x$382). When we
go out to three standard deviations we are at $1,477 ($331 + 3x$382) and would expect 99.7% of
our trades to fall within this range. Of course, trading has fat tails. Extreme events happen more
often than expected but this gives us some reasonable guidelines. Looking at the TradeStation
performance report we can see our largest historical drawdown was $1,901 - 3.87 standard
deviations. Let's now convert our standard deviations into percentages.
When a drawdown begins to surpass our first standard deviation we may take note, but it's not an
extreme event. Based on the math 32% of the trades will exceed 1 standard deviation so I would
not change any of my trading rules. At two standard deviations away from our average drawdown,
How to Build Profitable Trading Systems Using EasyLanguage v1.05 Page 101
you may want to consider reducing your position size by 1/2. Example, if your normal size is 500
shares, you would not be trading 250 shares. Surpassing our third standard deviation reducing
your position size would be wise to reduce your position size by 1/2 again (125 shares). Reaching
beyond our historical maximum value you should either be trading an absolute minimum amount
or switch to paper trading. A trading system that goes beyond our maximum historical drawdown
will most likely result in you pulling the plug on the live system. This may not be a permanent
phenomena. It's very possible that the given system is experiencing new maximum drawdown
before shooting to new highs. I've seen this before. But during this time you should be trading
on paper.
Once your system turns the corner on paper and rises above that, you can begin to increase your
position size as the equity curve rises from the new drawdown peak. You can increase the size in
the same manner as you reduced your size.
Following these guidelines will ensure that as your system begins to fail, you begin to minimize
the damage to your trading capital. In fact, if all goes as planned you should never lose much
beyond 10% of your trading account trading this plan! Of course this does not protect you from a
catastrophic loss. A sudden plunging market or technical glitch that causes you to lose far beyond
your normal drawdown are unpredictable dangers that we all must face.
Action Items
• At this point I would read about position sizing, download the spreadsheet and play with it.
However, your main goal is to build profitable systems first! This topic does not need to be
implemented in your system at this time. Of course ambitious programmers should feel free to
explore this more.
• Remember, never use a position sizing method to improve a poorly performing system.
• Capital preservation is important! Compute the expected drawdown levels of any system you
might trade live to know when the system might be broken. Follow the scaling out procedure to
protect your capital. Managing risk is number one priority for all traders.
How to Build Profitable Trading Systems Using EasyLanguage v1.05 Page 102
Step 12: Live Trading
Notice the name of this chapter is “Live Trading” and the previous chapter was “Live Testing”.
There is a single word difference between these two chapter titles but it’s a big difference when it
comes to real-word execution. In this chapter we are going to take your trading system live. That
is, you will make the decision (or not) to allow your system to execute on the live market with
your real money.
Once you have tested your system on the live market in simulation mode and you feel confident
in it, you can transition your trading to the live market. There are many things to be aware of
when trading on the live market including how to handle power outages, dropped internet
connections, and your trade becoming out-of-sync with your brokerage account. All of these
topics are beyond the scope of this eBook which is focused entirely on the development process,
not on platform specific issues or live trade execution.
Here are some recommendations on transitioning your trading to the live market. First and
foremost is risk management! This comes in many forms including having a stop on all trades
that’s within your risk tolerance. In this case since we are starting to trade a new system with real
money it is recommended you start trading with a significantly reduced position. If you normally
risk about 2% of your account equity, cut that down to 1%. The idea is to reduce your risk until
you have demonstrated that your trading system is working properly with live executions.
Trading on the simulator is different than on the live market. Bugs that were not apparent in
simulator mode can show up during the live market. This is, of course, on top of the other issues
I discussed above that will occasionally happen on the live market. In short, we want to reduce
this risk during the transition period.
What about our system we have developed during this eBook? The system does appear to be
tradable. If you load SPY the system does produce a positive return. We can feel much more
confident when our system performs OK on similar markets without any modifications. What
does this look like on DOW or the S&P futures contract? If it looks OK on these similar markets,
How to Build Profitable Trading Systems Using EasyLanguage v1.05 Page 103
this tells me our key idea probably has an edge and we are not curve fitting our system. This is all
great news for our system.
However, there are a few issues to keep in mind. First, the system did not do well during the
market from 1999-2003. If those market conditions return, we will likely see demolished
performance. This is minor in my opinion as it’s done very well on the out-of-sample. But
remember, we also did not have bear market out-of-sample data. So, how will it perform during
the next bear market? We don’t know for sure so, that’s something to keep an eye on.
Other items of importance are how well funded are you and how psychologically fit are you to
trade this system. However, please note this is a swing system that trades off daily bars with a
$2,000 hard stop. What does this mean? You need to be properly capitalized in order to trade
this system. For example, if you only wish to risk about 5% of your account on any given trade,
you would need $40,000 to trade this system. Also, take a look at the times when the system
takes heat, possibly for days! Could you handle that? These are all questions you need to ask
yourself before putting real money on the line.
Let's say your are committed to putting some real money on the line. Let's also say you decide to
only trade one contract until you gain confidence your system is working properly - always a good
idea. Do you just jump in and start auto trading? You can, but I often like to time the start of live
trading during a drawdown period. This is probably just personal preference, but I hate the idea
of jumping into a system when it's making new equity highs. I wait until a small pullback then
start. This is just an idea.
This document is not designed to give you a killer system where you can quit your day job and
become independently wealthy. It is a guide to a process that can help you produce a profitable
trading systems. Our Key Reversal system we produced as an example does appear workable. It
has potential! But, it has its warts as well. Of course, nothing in this business is perfect. All
trading systems have warts and given the nature of the changing market, unknown risks are
always lurking. That's why it's important to trade a portfolio of automated systems, each working
on different timeframes and markets. It's a fantastic way to smooth out that equity curve. No one
system dominates! But if you're new to this, you need to start out with just one system.
Below is a quick review of some things to keep in mind when reviewing the performance report of
any trading system.
How to Build Profitable Trading Systems Using EasyLanguage v1.05 Page 104
Review
The following are important questions you will want to ask yourself when evaluating the trading
system.
As we talked about at the beginning of the eBook, both slippage and commissions should be
deducted for each transaction. The effect of both slippage and commissions must never be
overlooked or underestimated. Be sure you included proper slippage and commissions for each
trade.
Obviously the more profit per trade, the better. In general, the average profit per trade should be
above $50 per trade for intraday trading systems and above $100 per trade for higher timeframes
to make the system viable for real-world trading.
Look at the equity curve and imagine you were trading the system. Look at the dollar value and
time between the equity peeks and valleys. This represents drawdown in dollars and time
between new equity highs. Some systems will be in drawdown for months or years! Other
systems may have “quiet” periods where the equity curve simply meanders around without doing
much at all. Can you handle the drawdowns and the quiet periods between new equity highs?
Look at the maximum drawdown from the highest equity point. Count the number of days the
system was in drawdown. Note the amount of dollars the system was in drawdown. Then ask
yourself if you can handle it. Can you continue to trade a system that will have this type of losing
streak? This is vital to your personal trading psychology.
Many people don't even look at this number. This value is another way to look at those losing
periods where your personal strength is going to be tested. Again, imagine you were trading this
How to Build Profitable Trading Systems Using EasyLanguage v1.05 Page 105
system and had to deal with this many consecutive losing trades. Can you handle the number of
consecutive losing trades?
It's important that a trading system functions well over different market conditions. Make sure
the system performs well during bull and bear markets.
All trading systems must perform well on out-of-sample data. Does the equity curve during this
period create new equity highs? Or does the equity curve languish and begin to decline?
Action Items
• Always keep a spreadsheet of each live trade. It’s important to document live trades so they can
be properly analyzed.
• Follow you scale-out plan if you start to encounter deep drawdowns. Capital preservation is
vital!
How to Build Profitable Trading Systems Using EasyLanguage v1.05 Page 106
The Next Level!
If you enjoyed this eBook, you’re going to love my amazing video course on building trading
systems. It’s called System Development Master Class and you’re going to love it! I go into a ton
of detail and cover topics not discussed in this eBook.
How to Build Profitable Trading Systems Using EasyLanguage v1.05 Page 107
Closing
Thanks for purchasing this eBook on how to build profitable trading systems using
EasyLanguage. There is a lot of information within these pages, particularly if you are new to
system development. The process within the eBook is the very crux of the process that I use to
develop a trading system. I hope this process gave you useful insights into my process and
provided you with enough ammunition that you can start taking real action to improve your
system development.
Your best bet is to start building systems. It does not matter if they do not work well or if you
build systems on markets and timeframes you don't trade. Just start practicing building systems.
Get your hands dirty and put to use what you have learned. You will soon find what works for you
in the system development process and what will need to be changed. Take the SDN template
and use it and change it based on your needs. You will begin to refine the development process
described within this eBook into a process that is your own. One skill that you should learn as a
developer is accessing multiple timeframes. I think this could help you out a lot in many different
situations. TradeStation makes it relatively easy but complications are abound. This topic alone
would make a great eBook or video series. Maybe someday…
Until then you can continue your education by joining me in my video training course on
building profitable trading systems. It’s called, System Development Master Class. You’re going
to love it!
Truly,
Jeff Swanson
EasyLanguage Mastery
How to Build Profitable Trading Systems Using EasyLanguage v1.05 Page 108