Backtesting results not repeatable

General discussion about Haasbot's Scalper Bot. (Beginner)
merkle-mayhem
Registered User
Posts: 17
Joined: Fri Nov 24, 2017 8:07 am

Backtesting results not repeatable

Postby merkle-mayhem » Wed Jan 03, 2018 2:22 am

Attached are three screenshots of scalperbot backtests. The second test was done 15 minutes after the first, and the third test was done 45 minutes after the 2nd.

All backtests had identical settings and were cleared prior to testing.

The second backtest achieved roughly DOUBLE the trades of the first backtest, and was almost 3 TIMES as profitable.

The third test only achieved a SINGLE trade.

The backtesting feature is really valuable, and I primarily want to use it to find the best variable settings for the Haasbot.

HOWEVER, if the backtests can't produce reasonably repeatable results, then it makes such variable optimisation POINTLESS, because one cannot have confidence in comparisons between different variable values.

My impression is that tradebot backtests are also not very repeatable if done at different times.

Why are the results varying so much, and can they be made more reliable?

Setup:
Backtesting period was 2 weeks.
Note that all the simulated trades in both tests occurred at least 1 week ago.
BETA Tradeserver
Running on a fast windows VPS (quad-core)
Accessed via firefox browser (within the VPS)
Simulated GDAX account
LTCBTC
Attachments
Screen Shot 2018-01-03 at 1.15.34 pm.png
test 3
Screen Shot 2018-01-03 at 12.29.16 pm.png
test 2
Screen Shot 2018-01-03 at 12.14.24 pm.png
test 1

Synthetic
Registered User
Posts: 16
Joined: Wed Nov 08, 2017 2:30 pm

Re: Backtesting results not repeatable

Postby Synthetic » Wed Jan 03, 2018 5:01 am

You are correct in that the results of the scraper are dependent on when you started them. This is because the scalper waits until the price difference is 1.25% (in your case) then sells. Then waits for the price to move 1.25% in order to buy.

Using the new bot performance tool, you can see that the scraper does most of its buying and selling while the price is in a horizontal movement (price line is mostly flat, with small changes).

Most people will run multiple scrapers, with slightly different settings.

merkle-mayhem
Registered User
Posts: 17
Joined: Fri Nov 24, 2017 8:07 am

Re: Backtesting results not repeatable

Postby merkle-mayhem » Wed Jan 03, 2018 6:35 am

Thanks very much for your reply.

I think I see the problem:

The 'missing' trades (achieved in the second test, but not in the first test) are well within the 2 week window. HOWEVER, the first trade is right at the start of the 2 week window.

Buying at a slightly different start price each time could result in the the scalperbot being in a buy/sold state or buy price state that cannot take advantage of the subsequent 'missing' trades.

This reveals a flaw in using 'window' backtesting periods (1 day, 1 week) with any bot that starts with a blind buy/sell; backtests run minutes apart can produce cascade effects (based on starting price differences), making variable comparisons very unreliable.

A simple fix for this would be to have the window period start at a specific time. For example, for a 2 week testing window the backtest could start at midnight on the day two weeks prior. Then at least comparative backtests run on the same day would be usable for tuning variables.

Obviously the tradebot interface has a superior backtesting setup in this regard, where the beginning and end of the backtesting period can be specified.

Does the above make sense? I think we are describing the same cause, but describing the consequences differently.

Synthetic wrote:You are correct in that the results of the scraper are dependent on when you started them. This is because the scalper waits until the price difference is 1.25% (in your case) then sells. Then waits for the price to move 1.25% in order to buy.

Using the new bot performance tool, you can see that the scraper does most of its buying and selling while the price is in a horizontal movement (price line is mostly flat, with small changes).

Most people will run multiple scrapers, with slightly different settings.
Last edited by merkle-mayhem on Thu Jan 04, 2018 6:53 am, edited 2 times in total.

Synthetic
Registered User
Posts: 16
Joined: Wed Nov 08, 2017 2:30 pm

Re: Backtesting results not repeatable

Postby Synthetic » Wed Jan 03, 2018 8:09 am

Yeah you got it. So if you wanted the backtests to be reproducible, you will need to select the start time. When you run a backtest from the bot's page, you can select the starting time.

PS: Also heads up in that your "Safety Threshold" value is 99. That means a 99% loss.

merkle-mayhem
Registered User
Posts: 17
Joined: Fri Nov 24, 2017 8:07 am

Re: Backtesting results not repeatable

Postby merkle-mayhem » Thu Jan 04, 2018 8:23 am

Thanks for confirming that.

Also, good to know re the Safety Threshold; was not sure if they meant 1% loss or a 99% loss(!)

As far as I can tell, only the tradebots allow you to set (and save) a specific start time for backtesting, while the custom bots (like the scalperbot) and the indicator scanner only allow a time window to be selected. I have resorted to setting up a tradebot to emulate the scalperbot to get around this.

Synthetic wrote:Yeah you got it. So if you wanted the backtests to be reproducible, you will need to select the start time. When you run a backtest from the bot's page, you can select the starting time.

PS: Also heads up in that your "Safety Threshold" value is 99. That means a 99% loss.

bpztrader
Registered User
Posts: 1
Joined: Sun Dec 17, 2017 9:30 pm

Re: Backtesting results not repeatable

Postby bpztrader » Sat Jan 20, 2018 7:49 am

I have noticed this too, custom bots cannot be custom back tested producing non-intuitive results. It would be great if we could set the back test dates with custom bots.


Return to “Scalper Bot Discussion”

Who is online

Users browsing this forum: No registered users and 1 guest