New Ensign: Playback


The old Ensign used specially built playback files to play back a prior date's ticks. The new Ensign has the perfect resource already available in the form of the tick database files.   So the new Ensign will use the tick database files now instead of the special playback files.  That is the essence of the change.  The new Ensign has eliminated that step of building special playback files, and plays the tick information directly back.   So what you collect or refresh is the source file for the playback.   There is no need to do anything more, and there is no longer a Build Playback Files process.

@Buffy2: so no more playing back anything we don't collect?

Buffy,you can playback what you have, and we can still post playback files for download like we have done in the past.  The current implementation still plays back a single symbol into the DEMO symbol, but another enhancement I want to work on is an ability to playback multiple symbols simultaneously.  The playback files will now be the tick database files for a symbol for a day.  So the playback download could be an alternate way to get the database files from our web site instead of by refreshing from your data vendor.  Our files would continue to be built from the eSignal feed/refresh.

So first of all, lets go through the step to see what you have for a tick database. The tick files must exist in order to use them in a playback exercise.   Lets use for the exercise the ES #F symbol which is valid for IB, eSignal and Open Tick feeds.   Those on the IQFeed would use @ES#, and those on TransAct would use =ESZ7.

Click menu File | File Manager | Database Manager and on the form have the Ticks bullet selected.   On the left side list select the ES #F or equivalent folder.

The left side is a directory list with a companion column of file names in the selected folder.  My example shows I have tick files for ES #F symbol for every day for the past couple of weeks.   

I have ES #F and NQ #F tick database files on the Ensign web site that can be downloaded using the Internet Services form.  The download process if you elect to get my web files will put the files in the right tick folders.   So, IQFeed and TransAct users can go get the ES #F files for 071108 and 071109 which we will use in today's exercise.   The files I have for download on my web site were built with refresh data from eSignal.  If you build files from DTN the tick counts will be slightly different.

Another step you can do if you not have a file or your file is incomplete, is to select the date using the calendar tool on this form, select a quantity of Today or 1 Day, select your source and click the Refresh button.   If your vendor does not have the data, you can change the source to the DTN Refresh selection and get the data from DTN.   So the first step I am showing is seeing what data is available and helping you get data if you need to.

Go ahead and close the database manager form, and click menu Setup | Playback to show the playback form.

 

Enter in the Symbol field the symbol you will do a playback for.   Today we will use ES #F as the symbol.   You can either select the date using the calendar tool on the form, OR you can browse to the tick database folder by clicking the browse button.

If you browse to the folder, select a file on the list and click the open button.  We will use Friday's date of 11-09 for the playback.  The Filenames are in the format of yymmdd which is 071109 for Nov 9th, 2007.

You would select a speed, such as the 4x which is 4 times normal speed.   A 1 minute period would play in 15 seconds using 4x.   

The Start Time is the point at which you want to start the playback in the day.  Often this is the open time such as 9:30 Eastern time.   For the example, lets use the 1035 entry which then is an hour into the day session.  The time can be entered as either 10:35 or as 1035.  Either format is tolerated.

The Sync time would be blank if you are doing the playback by yourself.   If you are trying to be coordinated and in sync with others in a chat room doing a playback exercise, then a mentor needs to start a playback, and the Sync time will fill in automatically.  This Sync time would then be told to the others in the room, and they too will then be in sync with the others in the room by entering the sync time provided by the mentor.   Let me start a playback and be the one to provide the rest of you the Sync time to enter.

Shar: Do we use EST if our computer is different zone? Like PST?

All times on the playback form are to be entered using Eastern zone time.   After entering the sync time, such as the 16:56:42 to be in sync with me, then you click the Open Demo Chart button.

tim: What is the "Play Back Enabled" box for and do we need to have it checked ?
@Buffy2: I use reset and then just open demo workspace

Now I typically click the Open Demo Chart button and that will open a chart for the time frame listed by the button, which in the example is 1 for a 1-min chart.  You can do the following instead.   You can check the Play Back Enabled box, which starts the playback, and click OK to close the form.   Then you might open a workspace containing DEMO symbols, or use the charts panel to open a DEMO chart.   So either way works, and I personally use the Open Demo Chart button because it does all of the following: checks Enable, closes form, opens chart.   So that was a 3 in 1 click.

 

This chart shows where the playback has progressed so since I started a few minutes ago.

@Buffy2: I have been asked why 4x and 6x is tradeable with new version and sure wasn't on old Ensign as it went too fast -- is it due to the new steady speed in new version?

That is correct.

Data ahead of the start time will be showing already on the chart, and this can include multiple prior days if the tick files for those prior days exist in the database folder.    That is why I suggested that you also have 11-08 tick file for ES #F in the folder.   The charts build the bars from the tick database very fast, much faster than the old Ensign.   You can put the same templates, draw tools, and studies on the playback charts as you use on your real charts.  The ONLY difference about the playback is the faster speed such as 4x, and the fact that only a single symbol feed is being played.

tim: The template name in my chart properties window shows a template name from today but the template on the Demo chart is actually from about 2 months ago so the template name is incorrect in the chart properties window.

Tim, it shows what was on the chart when you last closed it.  So go put on the new template.   That is not a playback question but would apply to any chart you open.... it show how it was last dressed, which can be from 2 months ago too

Saroj: is there any way to download a group of tick data or is it only one day / one instrument at a time? (for playback)

Saroj, one file at a time, or refresh one file at a time.  The files to download are around 1.5 Meg per file, so it is 1 at a time.

@Buffy2: love the fact we seem to be staying in scyc with new way.

Max: What is the purpose of the playback feature? Is it for paper trading? If for back testing, can you show an example pls?

Playback is for training...... some will paper trade, some will practice recognition of the signals and setups they are looking for.   Some will will research and verify their systems and signals trigger in real-time.

JohnA: how do you shut it off?

To stop the playback return to the Playback form and uncheck the Enable Playback checkbox.

Saroj: ahh... so no more Pause at the top?
@Buffy2: you can use the pause and resume button if you want a break also.  But you will not stay in sync with anyone using this.

Good point Buffy, if you pause the others doing the same playback went on.  If the mentor pauses, he should reset the Start Time, clear the Sync box, start himself, and post a new playback form for others to get the new start and sync times.

Saroj: I have found that if I didn't blank out the sync time in the earlier version, it would start at an odd time... not the start time... is that still the case?

Yes.  Still the same.

@Buffy2: if you are starting one alone than yes the sync time should be blank.  @Buffy2: gee ezgoing if I didn't remember to blank it mine wouldn't run in old version -- you are lucky.

ezgoing: This is doing great Howard, I had had problems using my range charts with the old playback but no prob now

tim: Sarjo, I have noticed if I load a large amount of data on a chart i.e. 20,000 bars, it will cause Ensign to lock up during real trading ( your Q about prior tick files )

Large amount of data is not a problem.

daveM: I had what I thought was a 'lock-up', but it was just the machine processing a huge new file. took a couple extra secs to get happening

You do need the tick file in the database for the date you want to playback.   If you have files for earlier dates, then the chart will be loaded with prebuilt bars for the days ahead of the play date.   You will notice a huge speed improvement in how long it takes to open the chart with the prebuilt bars. It is super fast now.   And the DEMO symbol chart will look identical to the actual chart for the bars.

More Tick Data:

Saroj: Howard where can I find out how to get tick data for ER2 prior to 11/08? 

Do not have a source for more than a week back for IB users.   The DTN Market Access source goes back 1 week.

Saroj: oh... so no way to convert previous *.txt data?

1 moment Saroj, let me see if I can help.  You want ER2 tick files older than 11-08.  I can get data for you from eSignal farther back under symbol AB #F.  Let me get several days, and upload them.   You can download them using the Internet Services form and copy them to the ER2 folder.  They are the same data.

Saroj: ok... so it seems that I'm using the Database Manager correctly... its just that these historical files aren't available... right?  But they will build up over time as long as I have my computer on and those symbols are in the Manager (Setup)

Correct, DTN tick refresh is back 1 week.  For IB users the program will automatically relocate the AB download to the ER2 folder.  

Import Playback Files:

jg: will it be possible to write an ESPL script to poke old playback files into the tick database in order for them to be available for playback ?

Good question.  There is great value in the libraries of playback files you have, and the Database Manager can be used to convert a playback file into a tick file.   In the past there has been an occasion inquiry if a playback file would build some bars to heal a chart or provide more back data.  Old Playback files can be converted to get more back data.  You convert the data into the tick and minute files, and then that resource can be used to build bars for more back history.

 

On the database manager is a bullet selection labeled Playback.   When you select the bullet, the Playback folder will show which contains your old Ensign playback files.  Select a file to convert, and click the Refresh button, and Ensign will import the ticks from this file and make a tick database file.   This process happens quickly, so you can walk down a list of files, select each one and click Refresh button for each one.  Then you will have useful tick database files going way back that you might not be able to refresh now, and you will be able to play those old files in a playback.

Saroj: and that data will be available for filling in regular charts too? not just playback?
@Buffy2: yahoo :-)   love it.  Thanks for a great solution.  Wow that is years.

And if you have interest or need the tick file created from the playback can also be used to build a 1-min database file by selecting the Source 1-Min from Ticks.  There you click on the tick file to convert and click the Refresh button with 1-Min from Ticks and it builds that 1-min database file from the Tick file.

@Buffy2: bet many are happy with this solution

When tick and/or minute database files exist, they can be used to build bars on the charts by using the chart's pop-up menu for Rebuild Visible with the area showing on the chart view where you want data inserted.   In my testing, I converted all of these YM files from Playback files, and then build this tick chart from the data.

 

Any questions about playback that are still in your mind?

sportsrep3: will the playback on the old version continue to work?

Yes playback in old Ensign will continue to work and use playback files.  However, there is an issue of mixing old and new Ensign doing same playback in a chat room setting.  The two do not stay in time sync.  The new Ensign is superb at playing back at the intended pace, and old Ensign slips in keeping on pace in fast markets.  Old Ensign experiences time creep until old and new are not in sync with each other.   Sorry about that, but over the next month or so most users will migrate to new Ensign and that issue will diminish.

In the new Ensign the equivalent archive of files for playback is now the tick database.  There is a tick file for each day, organized in folders for each symbol.   The new version's only use of the old C:\Ensign\Playback folder is to convert playback files into tick files.

The revisions made since mid July have been the biggest undertaking in the history of Ensign Windows.  We have more work yet to do over the coming weeks and months.

ticks: Howard what can we say but this is really great and Viva Ensign, Viva Howard !  Bravo !
@Buffy2: very nice for sure Howard
Charles3: thanks Howard - excellent work

Now that the beta test phase has been completed, I can return to working on implementing other good ideas, of which there are many.  Those who participated in the beta testing of the database have been pleased and pretty trouble free. 

ticks: Why isn't every trader using Ensign ??


Last modified 8/9/08 1:48 PM