Ensign: Database Design


Ensign has been designed around a foundation database.   Advantages gained by using a database are:

  1. With Tick and Minute databases there is a reduced need to refresh to have complete databases.
  2. Capacity:   Database can go back many months.  Initial refresh from DTN Market Access will be 8 months.
  3. Speed:   Building chart bars from the databases is considerably faster than extracting from a tick pool.
  4. Consistency:   All time frames build bars from the same database.  Chart bars can be deleted and rebuilt.
  5. Efficiency:   A refresh updates the database.   Do not need to refresh multiple time frames.  The refresh benefits closed charts too, which is a huge benefit.
  6. Preserve:  Ability to lock individual days in the database so subsequent refresh does not overwrite data known to be good.  This will be particularly useful in maintaining continuation files for futures.
  7. Extent:  Intra-day and tick charts can auto refresh upon opening, if a refresh is needed.  Mode to auto refresh database on program run.
  8. Playback:  Reads the tick database.  No need to build or download separate playback files.  This is the real-thing.
  9. Time & Sales:  Quickly jump to any date, any time of day, and view the time and sales ticks.    Example

The database design which embraces all these advantages was accomplished in the following manner.

The Database Manager form is accessed from menu File | File Manager | Database Manager, or from a chart's pop-up menu.    On the left side you will have a folder for Ticks\ and a folder for Minutes\.  In each folder are sub-folders containing the database for each symbol that you follow.   So there is a tick database and a separate 1-minute database.    Each day is a separate file providing the following advantages.

If you need to replace or repair a day in the database, do so by working on that day's file.  The whole database is never at risk of being corrupted or lost.    When you select a date in the database, a row of boxes show that days' contents.  There is a box for each half hour of the day and the colors indicate something about the content in that half hour period.  The 24 hour day will thus have 48 boxes, beginning with the evening session open and going through the day session close.   

A box that is dark blue has data obtained by refreshing.  A box that is green has data obtained from the live data feed...... and a refresh of the same period will change green to blue.  A box in the aqua color has no data, but that was the way it was in the refresh, so the omission is probably correct and doing another refresh is not going to add anything, because you already had a refresh covering that period.   Most likely, the market is just idle or closed during that time period.

A box that is white indicates that no data has been seen (yet) in real-time or in a refresh, such as the remainder of the current day.

@Buffy2: half of it 4:15 to 4:30?

In your example Buffy, the box would get colored because of data from 4:00-4:15.  A red box, indicates that data is missing in the past, and may just needs to be refreshed.  Now, in the example, on the right there are boxes that have a center hole.... this indicates those time periods have not been locked.

If the data is correct, and you do not want it to be changed, click the Lock File button and the boxes with holes become solid boxes.  A refresh will make the box solid to begin with, meaning it is already as good at it can get and refreshing again will not benefit.   The green box could be made locked by locking it, in which case the refresh will not replace it.   To unlock a file that was locked, there is an Unlock File button, in which case a refresh could then replace the bars.   This lock/unlock system will replace the Preserve Existing bars which was a private chart property.

@Buffy2: but work on tick and volume now?

Yes, since this lock/unlock feature works on the tick database files, it is another advantage over the old tick pool system which did not offer any want to preserve bars.

Now the database is a blend as follows.   When a chart is opened, it will load the existing chart file, ie your ES #F.V1000 chart which might go way back.  Then the database for ticks is used to bring it current.  The database can have many months of files, and these do not self delete.   Thus, in the earlier example where a chart had not been opened for 2 weeks, now is not a problem.  These 2 weeks of tick files exist so the chart can be updated.   Then the new updated chart file is resaved so the extraction is not needed the next time it is opened.

The building of bars from the new database is super fast, much faster than extracting from the tick pool.  And any updating done to the database by either the live feed or the refresh benefits all charts that are closed.  A single refresh that adds to the database provides content for any chart that uses the database.  The same principles apply to the minute database.

All intra-day charts read from the same 1-min bar files to be brought current, and the process is super fast.   So you will notice a speed increase as you open workspaces, etc.  Also, you will have much less need to do any refreshing.   In the past, if you opened a 5-min chart, 30-min, 60-min, 13-min, etc, each had its own refresh request.  Now, at most, would be a single refresh request for 1-min data that then benefits all these open charts and all close charts yet to be opened.

Tick, range, seconds, and volume charts all build from the tick database.   Now, this leads us to the next cool feature, and that is to have the program be smarter about what needs to be refreshed.

Refresh on Program Run
Because Ensign can be configured to automatically refresh, the user can give less thought to the chart content.  One of the things implemented to that end was to have Ensign do refreshing before you need the chart.  Ensign does it in advance in anticipation of your wanting it.

We have two new check box options on the Setup | Charts form for starting an Auto Refresh of the minute and/or the tick databases when the program connects.

On the Setup | Charts property form are 2 check boxes to turn on what I call an automatic refresh feature that kicks in when the program runs.  When these boxes are checked and Ensign runs, then the program will go through the symbols on the Setup | Manager list and determine how much data is needed to bring the databases current, and make refresh requests automatically.

Say a user with an eSignal connection is on-line until 9 pm at night, and then exits Ensign and turns it back on at 7 am the next day.  Ensign's databases know they have data through 9 pm, so upon connecting if the auto refresh options are enabled, the program will start going through the symbol list on the Setup | Manager form and request minute data from 9 pm through 7 am, and then when that process is finished, it will request tick data for the same time period.

The minute data base is refreshed first because that completes most quickly.   This user might start to open some charts, and they may or may not be refreshed, yet, but an unrefreshed chart is detected, and its refresh request is then moved to the top of the queue.   Eventually all pending requests on the queue are serviced, and hopefully this user never sees the rest of the day a single refresh request because the databases are current already.

@Buffy2: I see this all the time Howard as I go back to old version to get a template or something I forgot to bring into the clean beta install I opted to do.   It is a great feature for sure.   I just wait until top left says done after opening workspace.  At some point I go to dbm and refresh days not locked which locks them making them read only and pick up the ticks I may of missed due to whatever.

Now the auto refresh is a good intention, but unfortunately it is tainted by which data vendor you use.   And the taint is for the IB users, whose refresh from IB is plagued by 3 issues.

1st issue is that any refresh request from IB is really slow to get a reply.

2nd issue is that the have a cap of 2000 record returned on any reply so we have to break up a refresh for a given period into multiple requests.  All that takes longer to get the total reply.

3rd hindrance for IB refresh is they have imposed a pacing restriction of 60- requests in a 10-min period. and since a period refresh might have been broken into multiple requests, the 60- limit gets used up pretty fast.  So I do NOT encourage the auto refresh options for program run to be checked for IB users.  Otherwise the refresh resource can be bottlenecked for a couple hours waiting for the 10 min limit period to expire.

DTN does not have any such limits.   So if you can, IB users should do their refreshing from the DTN source.

bld: A wonderful program for 25 years gets better and better! Refreshing data was always somewhat difficult.  This seems to answer that problem.

Thus, if the feature is enabled, you might have an updated database within a couple minutes of running Ensign and thus not see another refresh request all day long.  Or, you can have the auto refresh on program run turned off, and then when the chart is opened, we see it might not have a current database, and go do a refresh and show the data when it is received.   Either way works well.

Abinadi: If we leave Ensign running all the time, should we restart it each day now?

No, Abinadi.  If on-line, you have all the data.  You can still right mouse click and ask for a refresh, or you can go to the database manager, and click the refresh button on that form to refresh that symbol.   Both places allow you to pick the amount to download.

Now this is really cool.... and exciting.

If you select the DTN Refresh as the source, you can refresh 1 year of 1-min files in a single request.   The refresh of 12 months of 1-min files takes less than 2 minutes, and would only be needed to be done once for a symbol.   This gives you lots of back data for all the intra-day charts, and from then on you are just keeping the database updated each day via the live feed or an occasional refresh.

@Buffy2: Howard I still don't understand why dbm would think there was a gap when that workspace was up when I closed and got first one and up and closed again when I got second one.  It wasn't down long enough to need over 200 thousand ticks.

Show dbm for the day for ticks for symbol.

The program refreshes because of the single white square it sees.   Ensign wondered if it could fill it by refreshing.   The hole might be real, and will color it aqua if not seen in the refresh.   The refresh that will fill the hole and color the unlocked squares with the locked blue is a refresh for the 13th.   The file is showing the evening of the 13th and the day of the 14th.   Though it is the file for the 14th, the evening is on the 13th.   The refresh for 1 day does midnight to midnight.   So it is a refresh of the 13th that should fill it.

Auto Refresh Disable
One of the additions to the latest version is a new set of check boxes on the Setup | Charts form.

The auto refresh to keep the databases current was doing refresh too often for the liking of some users.  So, in response to requests and suggestions, I have added back a global check box to enable or disable auto refreshing.   In the old Ensign the Auto Refresh check box was on a chart's property form on the Data tab.  In the new Ensign, this global option to enable refreshing is on the Setup | Charts form in the lower right side.   This feature is available for all feeds.

If you want to disable auto refreshing so your charts open faster, then uncheck the On Chart Open for: box.  And I made it a bit more useful if you want to have auto refresh enabled, for one type of chart and disabled for another type of chart.   In the example, I have refresh enabled for intra-day (minute based) charts, and Daily (and weekly and monthly) charts, but disabled for tick based charts (tick, range, volume, seconds).   That is the purpose of the 3 additional check boxes in the Auto Refresh frame.

If you uncheck the first box for On Chart Open for, then the other three boxes are basically unchecked, there will not be any auto refresh.   You can still manually override this using the pop-up menu to refresh a chart.

Q:  Does this over ride the bottom two on left?

A:   The 2 refresh boxes on the left are independent of the Auto Refresh boxes that deal with opening a chart.   The 2 boxes on the left happen on program run.   Opening a workspace is opening charts and affected by the new Auto Refresh options.

The 2 boxes on left are working with updating database when program run.  Nothing needs to ever be opened for this to happen.  Opening a workspace, or opening a chart now can be controlled by the new Auto Refresh being unchecked.   It would have happened at the same time independent of the workspace opening.

If you refresh start by opening a workspace, the workspace opening beat the mole to the task of seeing data was needed.   There is a couple minute delay after program run to give time for the feeds to run their vendor software and connect up.  IB in particular is the slow one to get running and connected to.  The eSignal Data Manager can run and connect much quicker than the TWS can get connected.  So the refresh of the tick and min databases would be superseded by your opening a workspace before they started.

There is no problem there.   It is a queue of refresh requests that are pending, and if the auto database refresh kicks in, it might see pending requests already in the queue for the symbols involved, and would not put in a redundant request.

Continuation Files

Nick: Is your data base going to correct for DTN's early roll date compared to other providers?

No Nick, it does not change DTN's roll date.   If you are using eSignal, then refresh from eSignal to have the roll date for continuation files they way they do it.  That would be the Data Vendor bullet selection.  Also, on the DTN roll date topic, you can do this... collect the data live, such as March data, through the roll day.  Lock the file so you do not accidentally replace any of the march data by refreshing with June data.   That way the file integrity should be preserved that it is march data on a particular day.

Also, you can select a day, and change the cross reference symbol to be a particular symbol, such as march, and refresh for 1 or more days to put march data in the database.  Then change the calendar date, and the cross reference, and refresh to put June data in the database on other days.  So you will have more control than before, particularly by locking data once it is the way you want it.

Nick: but if I want a chart on a new future symbol I am stuck with bad data in the days before the roll date

Nick, I assume you are wanting the eSignal roll schedule data.... so another solution is to have your eSignal friend e-mail you particular date files for your database.  You would replace individual file dates, not the whole database.  Then you will match your eSignal friend though you may have initially built your continuation files using DTN refresh. 

At the top of the Database Manager form are some statistics for the day, such as the open, high, low and close for each session and for the day.   We also show the Up Volume and Down Volume and number of ticks, etc.

@Buffy2: Howard does it matter if I refresh with right click or on the dbm window?
terry: nice. find the data and throw the files in.

Buffy, either way is the same.

Existing Files

Kongfuzi: is it possible to use/plot OLD historic files on S&P and ES years old?

Yes, kongfuzi, old files load and display same as before.   They probably will not find data in the database to be brought current, so they will show as opened/loaded from the old historic file.  Any refresh will update both the database and any open chart.  Any closed chart will benefit from the previous refresh because it is brought current from the database. The former tick pool has been obsoleted and removed.

Time & Sales
Time and Sales now is a view into the tick database, so you can quickly jump to any date and any time in that date, and scroll up or down.  And you can view the tick database as a tick chart by clicking the Chart button on the time and sales form.  T&S can be shown from the chart, from a quote page, or by clicking the Time & Sales button on the new Database Manager form.

Here is an earlier example, but the principles shown still apply.

Lots of Data
Now let me give some training on the topic of of using the database manager to have lots of back data.  I will use ES #F 15-min as an example.   This is training about using the database manager.

Here I have the property from for the 15-min chart with the Max Bars set to a big size of 62000 bars.  On the chart I will right mouse click and select pop-up menu Database Manager.  I changed the Source to DTN Refresh, the quantity to 1 year, the Select box to the Continuation bullet so we will use the DTN continuation symbol instead of the Dec 2007 symbol.

When I click the Refresh button it will go get 1 year of 1 minute data from DTN.  It can take a few seconds before the refresh starts to respond, and several seconds to receive.  The counter on the top of the form will count up to 197000 bars and then populate the database with a year's worth of files.  Now you can change the calendar date to the first file on the top of the list and refresh again another year.  There is data for ES for this far back.  In doing the refresh twice I have populated database files all the way back to Oct 2005. 

Then on the 15-min chart, which initially did not have data that far back, I did the following.  I hold down the shift key while pressing = key, and this jumps the chart to the first bar in the existing chart.  Then right mouse click and use pop-up menu Rebuild Visible.  The chart sees the 1st bar is visible, and realizes you want to try to get data ahead of that bar, and it will populate the chart will all the data downloaded, and the chart then goes back to Oct 2005.  The build of all this data is very fast, and the chart repaints then with the 13661 bars I showed on the chart properties data tab.  This is how I got such a big file for ES #F 15-min.

The chart was day session only, and all I have to do is change the state of that to include all sessions.   There is a keyboard hot key to toggle day session only on or off and that is the @ character, (Shift 2 key).

This is the chart with days session only, all downloaded from DTN using the database manager as discussed above.  I pressed shift 2 (@ key) and in less than 2 seconds the chart with 47320 bars showed, all newly built from the database.   The chart did not exist before I pressed the key to go to all session.

Now let me jump to beginning of chart with shift = keys and repost.

Here is the chart back in Nov 2005.   As I scroll through the chart for the past 2 years, I AM VERY PLEASED and IMPRESSED with how clean the data is for this download from the DTN database.  I do not see spikes.

Now, consider what we have just done.   For a symbol of interest, I have just downloaded a database for 2 years (ES had that much).  It is continuation data, and now that it is downloaded on the computer I will not need to do that ever again.  All intra-day charts will benefit from this nice database.  If you want a new 119 minute chart for what ever reason.... presto you have it.   It did not require any new download or refresh.   It builds the bars from the existing database, and the tiny refreshes of polish off a recent day's collection is about all the refresh you might be doing.   So that is all exciting.

@Buffy2: it is exciting for sure Howard once all understand it.

Q:   I can Refresh the maximum 78,000 bars for the symbol QQQQ (I saw this number when the software refreshed the data).   If each day has 390 minutes for trading time (it corresponds to 390 bars for 1 minute chart), I should open 202 day of bars displayed on 1 minute chart for back testing.  But in fact, I can only go back around 60 day's data displayed.  Is there another way we can dispay the total 78 000 bars for back testing?

A:   Press CTRL+P for chart properties and on the Data tab set the Max Bars to 65530.  Change the refresh source to the DTN Market Access selection.   On the Symbol tab have the Day Session Only box checked.   On the chart right mouse click and use pop-up menu Refresh | Maximum.  The reply takes quite a while but the display on the top caption line counts to over 118,000.   After the refresh the Database Manager shows files back to 12-28-2006.   The chart shows it has 65519 bars now and the first bar on the chart is 04-30-2007.

When the Day Session Only box is unchecked, the program uses 1440 bars per day, divided into 65530 and estimated it only needed to go extract from 45 market days back until now to get the needed bars. When checked it estimates using 390 bars per day and goes back over 150 market days to extract data, and now it gets the wanted 65500 bars.

IB Refresh

@Buffy2: OK so an IB user turns on machine in morning - charts need updating -- they just right click and refresh? or go to dbm and refresh?

For IB user, I would recommend the open a chart, and right mouse click and refresh it as needed.

@Buffy2: so they are back on one refresh for time charts and one for R V T right?

Too many IB are having challenges with the auto refresh on program run so I discourage IB users from using that. Also, IB users are encouraged to refresh from DTN when possible because that source is both fast and does not count against the pacing violation imposed by IB.   But Buffy, a single refresh received by anyone will benefit all time frames for that symbol.

@Buffy2: if IB users refresh a Tick chart that is good for range and volume also

Yes, and that is a huge benefit.

Saroj: so Howard - IB users should always have "DTN Market Access" checked in the Data UI of Chart Properties?

Not always saroj.   The following needs to be considered by IB users.   I used the word encouraged to use DTN because DTN does not have the same symbol universe as IB.  IB might be the only source for refreshing the symbol.   Also, DTN is through a delay requirement, and IB can be through current tick.  Also, DTN will have more ticks in the refresh of tick databases then IB will have sent in real time, and that bothers some to have tick or volume charts changed by refreshing.   So, there is more to consider than just the fact that DTN is fast, has more days back, etc.

On the tick based chart, when an IB user right mouse clicks they have on the pop-up menu refresh selections that would get data from IB and one for getting data from DTN.   So they can choose which to use on the pop-up menu regardless of the setting on the data tab of chart properties.   And on the database form there is the source bullet selection which would control the source for the refresh.

Saroj: I'm still not clear on how to decide when to use which...   and the impact (if any) on how the chart properties "refresh source" is set up on intra-day refresh.  I just downloaded the new version an hour ago... but I'm not clear about when to use IB, when to use DTN, and the interrelationship of the chart properties to the data manager UI

We prepare a refresh request structure and one of the properties is the source, since there might be more than one source with the data, such as from the data vendor or from DTN market access.   That selection is a bullet choice on the database manager form.   The same selection is also on the chart property form on the data tab if that is your starting point.   The selection is also on the pop-up menu for IB tick chart refreshers. 

So where ever you are working from, you make the choice, the request is made and data flows into files in the database folders.  If you click Refresh on the database form, then that source selection is used independent of any chart that is open.  Likewise, if you are refreshing form a chart, then that chart's properties are used for the source.   It is all the same, makes no difference.   The database form is just a way to see what files exist and make a refresh request.   The chart is just a 2nd way to make a refresh request.  They both call the same procedure to do the refresh, and the reply is identical regardless of whether a chart is open or the database manager is open.

Saroj: so to sum up... if both IB and DTN have the same symbol set, use DTN for all but very current data (like the last hour); otherwise use IB?

That is a good rule of thumb

Saroj: ok... that helps a lot... a nice, simple "rule of thumb"... thanks.  And it seems to me that I should leave the Data tab set to IB so when I do a quick refresh intra-day (in case I had to reboot or lost power for instance) it will get the current data from IB?

Saroj: Well, Howard, this is a very welcome change... having this database of data right on ones machine... solves all kinds of problems... thank you for implementing this.

@Buffy2: agree saroj just need to get used to it

Welcome, and I agree it has lots of benefits.

@Buffy2: works well.   I didn't know it had the problems which you are having. But then I use eSignal feed.
Shar: Thanks for you time in coming in and helping.   Sure is making things clearer.
@Buffy2: yes thank you it is getting clearer and clearer :-))

Abort Refresh

al_gorithm: if we inadvertently do a refresh from IB and then forget to change the option back to DTN, is there a way to cancel a refresh.   Opening a workspace with lots of charts trying to refresh from IB clobbers CPU.

A refresh in progress shows on the top caption line a number in brackets which is how many other refresh requests are pending or in a queue.  You can do the following trick to throw the queue away, and the trick is to go to Setup | Connection and change the data source selection to Ensign Internet and then back to IB.  This will clear the pending refresh queue, so that they do not execute.

Database

dougfir: I have ASCII tick data going back 4 years, can I import/convert that to a database readable form?
topset: can the tickfile be exported as well as imported?
zbullzbear: who has 'database' we or Ensign?

A tick file can be opened as a chart and the chart can export.   I do not yet have features to export or import directly to the database files, but intend for such features to be added.   It just takes time to continue to build on the foundation that is in place at the present time.   Also, I want to add tools for editing directly in the database files, possibly use an edit on the chart to make the same edit in the database.   Right now the solution is to try to refresh the database file and clean up a spike by getting a better bar from the vendor refresh.

taku: did I read that it is wise to set # of bars to 65000

No taku, you did not see any suggestion like that.   In fact, I would suggest the opposite.   The database files are independent of the chart files.  Both exist in parallel.   A chart has a Max Bars setting and that is used to retain X number of recent bars.   As new bars are added to the chart the oldest are dropped.   The database, is always accumulating and never dropping.  So data you collect in August can still be there come next July.   2000 is a better number to use for chart Max Bars.

If and when you happen to need more bars in the chart for some purpose, you can at that time change Max Bars to be a higher value, position chart so first bar in file is in the chart view which is easily done with Shift = keys, and then use the pop-up menu for Rebuild Visible, and the chart will then go add more bars to the left of the chart that it extracts from the database files.  This can happen very quickly.  So the database file is a great resource for having back data, and not depend on a refresh to try to replace it.

Now on a related thread of conversation, lets say you missed some tick data a few days back.  In the database you go to the day in question and refresh that one day with ticks or minute bars.  And the refresh of the specific day then rebuilds the open chart to include the new data.  I helped a user today with this very need, and made a web page out of the process so it is now training material for the next person.   http://ensign.editme.com/q1774

See this example where today, the 12th, the users wanted to get ticks for the 8th.  This little example gives some familiarity of using the database manager form, which I most often get to from the chart's pop-up menu but it is also available from menu File | File Manager | Database Manager.

zbullzbear: who maintain the folders?

Setup | Manager List
You can do nothing and let Ensign make the folders and add files from the ticks for the symbols you have on your Setup | Manager list.   Or you can spend some time seeing what the database manager is doing.   Files and folders can be deleted.  Files can be refreshed if they lack completeness.   Refresh can be requested for dates not yet in the database.

Please be aware that custom symbols also need to be on the Setup | Manager list in order to build a chart database.

The symbol universe depends on your data provider, and if you have no provider use the Ensign Internet selection as your source and then all data comes from the DTN Market Access source.  The DTN source has all US futures, stocks and indexes.  Foreign stocks for daily is available from other Internet sources.

Let me post a link showing the refresh that is available.   read more » Data Downloader    You can download ticks for any of the exchanges listed in the table in the Tick column that show a yes, which is US stocks, Indexes, futures, Canadian stocks and futures, IPE, LIFFE, Eurex, etc.  And we can do the same with the Forex currencies for our FXCM feed which is included.

The intra-day charts all read or are brought current from a minute database.  Lets use a 60-m chart as an example.   You open the chart, it reads the 60-min chart file, then sees where it was last updated, and consults the minute database to bring the chart current.  Perhaps the chart is through 2 days ago, so the the minute files for the last 3 days would be opened, 60-min bars built from the 1-min records, and then the updated chart file is resaved so we can eliminate the rebuild of these bars the next time the chart is opened.  This process happen faster than ever before, and the speed improvement is noticeable and appreciated.

Shar:  Howard, do we get fills for Daily, Weekly, Monthly from Internet Service?  And what larger minute charts? We don't have to do those day by day like the tick charts, do we?

The daily, weekly and monthly charts operate the same as before in that the chart file for these IS the database.  What you have in the daily, weekly and monthly folder is what you have.

Refresh Speed
This brings us to another topic of a common question.  Some say the refresh is slower on the new Ensign.   Let me discuss so you see what is happening, and I will again use the 60-min chart as an example.  In the old Ensign many of you would open a 60-min chart, see it needed data a couple days back and you would use Alt-7 to do a refresh Maximum request.

About 4 seconds later you would have the chart repaint with the data, and you gave it no more thought.  Well, doing the same thing now seems slower for these reasons.  First, because we have a 1-min database we want to benefit.   Any intra-day chart refresh is going to ask for 1-min bars though 60-min bars are wanted.  Thus the reply will be 60 times bigger to ask for X days of 1-min bars than X days of 60-min bars.

When the refresh is received, we update the 1-min database and then update the chart with bars build from the 1-min refresh. Also, Alt-7 for Max refresh gets more from DTN Market Access than ever before.  Before max would be 4 months and now it is 8 months.

But consider this, ONCE you have done a max refresh, you should not do it again. You already have the data in the database and continuing to refresh maximum adds no benefit to you.  That is like ripping out the driveway and repouring concrete each time you want to take the car out for a spin.   Be wise.... only refresh data you don't have. 

Look at the database content.  We show a list of dates in the database, and for any given date, you can click on the file and see a row of boxes indicating the presence of data in each half hour period.  It gives a good idea if the file content is complete.  If it is complete leave it alone. 

So today, being your first day, a max refresh or a new chart might have taken longer to refresh because of the detail being obtained.  From now on, 1 day refreshes should suffice, or 2 day refreshes will suffice.  Use the bigger refresh requests as needed.

al_gorithm: Very nice work Howard.
Disciplined: The new Ensign version is really very fast and very smooth Howard. Thanks for this big improvement
claude: I agree. With IB it might take a little more work but it worked great for me today
drh: take a break ...great job ... I am wore out trying to keep current....:)....i started out with Howard at 6:00 am this morning...great job ...my friend 
wout: well Howard, I am off, has been very nice to have your class here and hope to meet you again. You have done a wonderful work!!!! THANKS
@Buffy2: thank you so much Howard for the extra support that was needed also

FAQ:

Q:  I do not yet understand about how past data will be handled when moving to the new system.  It seems from what you say that the old data be "sucked" into the new database (a big plus IMO).   Correct?  Additionally, will any updating of the data be made? For instance, if I have some range files with 65k bars and some volume / tick bars with 65k bars but many with far lesser bars, will the new database be able to fill in the range files but not the tick / volume files?   Or, will the range depend on the granularity of the files that I now have?

A:  Old files continue in their current location..... database replaces tick pool and not the data files.  So old files are brought current from database instead of tick pool, and then resave as updated file.  So no change really, in your legacy files.   You can have some big files and some smaller files.   Those file sizes are still controlled by the Maximum Bars chart property.   This is independent of the size of the foundation tick database.   You will be able to increase Maximum Bars to a larger value, and rebuild older bars from the tick database so you have more bars in the chart.

Q:   One of the challenges which I have with the current data structure in Ensign Software is I must test and research with one time frame (eg ER2 2800) but I can not trade with that time frame as with the many, many bars of the data files (that one of others which I save 65000 bars), I must trade with the 2801V or the 2799V; otherwise, I will run out of memory and also system will be VERY slow (and I know you are working on speed).

Would there be a way in the new system to keep the 65000 bars on the 2800V that I trade with and have a limited number of bars in the chart while trading without truncating the file. (I suppose since you are building a DB, that the answer would be Yes.)

A:  Answer will be Yes. The database will keep the data for the ticks so you can have a chart with a small size without erasing the foundation database.   Then if you what a test file, make the Max Bar count big again and rebuild the chart to restore the bars way back.   Database will have the ticks regardless of what you do with the Max Bars setting for the chart, which is just a view of a portion of the master database.

Q:   First thank you for undertaking a rewrite of Ensign. The speed improvements during fast markets will certainly be a welcome improvement for me as will the ability to display cleaner,easier on the eyes charts using the background gradient feature.  Does the # of displayed bars on a chart affect it's file resize speed?   I mean if a chart has max bars set to 2000 and it is displaying 50 bars will a wider chart displaying 200 bars redraw less often?   I guess what I'm asking is what determines when old bars are "cut" from the size of the file for a given symbol and timeframe such as ES V4500 bars.

A:   The dropping of old bars is when the Maximum Bars parameter is reached by the chart bar count. So the visual display of 50 or 200 is unrelated to when it resizes to drop old bars. The display of 50 bars can be drawn quicker than the display of 200 bars but that is trivial.   On the chart property form is a parameter for the number of bars to retain when the file is resized.  This value can be much loweer than the Maximum Bars so the file resizes very infrequently.   The chart loads in a resized position each morning, so you can probably avoid a trigger a resize all day long for most futures markets.

With the new Ensign you will notice that in the resize, you benefit from snappier chart repainting so it will all see better though you did nothing different.   Ensign will be speedier in the chart repaint.  If you have the new Ensign, try resizing the chart by dragging its border around and you will see the increased fluidity of the drawing.    I have resorted to implementing extremely clever graphics controls by doing more in BitMap graphics in memory for increased speed.   To the end user, you just see the end result and don't quite appreciate the complexity of the overhaul that has been accomplished.

Q:   Also, I have found that the DTN refresh rounded off minutes (on tick / volume data) so the charts were slightly (although significant to my research) different from live data which did not round the minutes.  Will this change to non-rounding on refresh?

A:   I am unaware of any rounding..... the DTN refresh can be different than the live collection, and that fact may have nothing to do with rounding.   I have never seen agreement with live feeds from DTN, eSignal or IB.   Each is different, and since worrying does not change that fact, I ignore the issue and just move on.  It is only you folks comparing yourself to someone else that you notice there is a difference.  Difference exists, and bringing it to my attention repeatedly is not going to resolve anything.

The difference exists at the source.  I just gather data from the feed and show the content.   We are the bucket at the end of the pipeline.  We can catch and show, but we cannot invent or control what is put into the pipeline.

Q:    What about standard symbols?   For instance, will there be any integration between the DTN symbols and the IB symbols? (Like replacing / eliminating the silly @ with a # or coming to some standardization for the most commonly used symbols.)? It seems to me that Ensign Software is in control of the symbol assignment and is the one place were a standardization could best work for Ensign Software users; we have several data feeds and it would be good for us if there was standardization.

A:  Standardization has been implemented more than you realize.... for example we have ES #F as a symbol for you IB users to use.  That is not an IB symbol, and for those on something like a DTN feed or you want to use the DTN refresh, we do have automatic cross reference symbols in the program.  So you can use your IB symbol and get DTN refresh for it.  We have to use cross reference symbols for each source.   For example, take the Dow Jones index.   On eSignal it is $INDU, and we use the same symbol and our behind the scene magic to make it work as the symbol for the IB feed.  The DTN symbol would be INDU.X, and the Yahoo symbol would be ^DJI 

Q:   Is it possible to limit the number of days in the data manager or to just disable it?

A:   I gather you are talking about the Database Manager demonstrated today.  It is needed to bring charts current, so it cannot be disabled.   However, you do not have to build lots of back files via a refresh as illustrated today.   And if you abandon a symbol, there is a Delete Folder button on the form to remove all files and the folder that is selected.   Each day is a file, so you can delete individual back files if you want to remove them.  But hard disks are getting cheap.  Each file is around 56K in size.

Possibly we can add to the Setup | Chart | Maintenance form a task to remove database files that are older than x days, so the program could be used to delete a group of older files.  Now you can use Window Explorer to navigate to the folders, list the files by date, and delete a group of files.  This illustrates another advantage of having individual files by date, instead of all data buried in one big file or database.

Q:   If I refresh/backfill or download the intraday data (such as 5 min data) from DTN Marketaccess, will I get the real (actual, not simulated) Bid/ask volume?   If it's real, how many days real(actual) bid/ask info we can get from DTN Market Access?   Is DTN Market Access data delayed?

A:   The tick refresh from DTN will have the tick volume and we can build 1-min bar database from this tick refresh and thus have actual volume at the bid and volume at the ask. The 5-min bars built from the 1-min database can show actual bid/ask volume, instead of estimated.   Database can refresh for 1 year back from DTN.   Data is still through the delay requirement.

Q:   Should I choose "1-min from Ticks" and then click "Refresh" button to get the REAL Volume at bid and Volume at ask (1min bar) from DTN.   (I'm assuming the raw tick data is refreshed from DTN and then Ensign uses raw tick data to build 1 min database to improve performance)?   It will be great for Swing traders like me to get 1 year's REAL Volume at bid and Volume at ask (based on 1-min bars).

A:   You would work with the Tick database to refresh multiple days to build up the tick database. Then on the Minute database you would use the 3rd bullet selection of '1-min from Ticks' and click the Refresh button. This will build the 1-min bars from the Tick files already downloaded into the tick database. The 1-min bars built will have the true bid and ask volumes this way.   Also, the 1-min database built from the live data feed will have the actual bid/ask volume information.

ticks: Howard what can we say but this is really great and Viva Ensign, Viva Howard !

Q:   This new version seems great but I am having data problems/challenges especially with refresh from IB on the daily charts.   For instance if you go to ZN #F and erase all the data and then try to refresh from IB it is a nightmare.   I am experiencing this with a handful of symbols.

A:   Change the refresh source for daily refresh to use DTN Market Access. Its refresh is fast.   The IB refresh is 1 week at a time, and must be spaced 6 seconds between requests, and no more than 60 requests per 10 minute period.   So use DTN source when possible.   You should not have erased the data already collected.

read more » Database File Format
read more » Tune-up for Peak Performance


Last modified 12/22/08 4:21 PM