Most of your data will be stored in channels.  Channels have a history which is the data acquired with a time associated with each data point.  The amount of data available from a particular channel for graphing and calculations is determined by the channels History Length and its Persistence Length.  This is independent of data logging and has no bearing on logging or export sets.  In older versions of DAQFactory, a channel only maintained its history in memory.  This was its History Length, or the number of points kept in memory for graphs and other calculations.  As described earlier, you can retrieve data from a channel by entering its name in an expression, often using subsetting to specify which part of the history you wish to use.  The data available was limited to the channels History Length which, while user definable, is somewhat limited by your system's memory.  Now channels can also save historic data to disk in an optimized binary format for quick and easy access by graphs and other calculations.  The amount of disk space used for a particular channel is its Persistence Length, which is only limited by your available disk space.  The file is preallocated and optimized so that the Persistence Length is completely scalable without a performance hit.  This means you could keep 10 million data points on hand for your graphs and calculations without having to worry about loading your data in to DAQFactory from a logging file.  

Using channel persistence is as easy as setting the Persistence Length for each channel.  This can be any size up to the limit of your hard-drive, though above 100 million requires subsetting by time as described below.  Each data point takes up 16 bytes, so a Persistence Length of 100,000 will take 1.6 meg of disk space.  The files are, by default, created in a subdirectory called Persist under your DAQFactory installation directory.  The file is preallocated, meaning a Persistence Length of 100,000 will create a 1.6 meg file immediately, even though you haven't accumulated 100,000 data points yet.

Accessing persistence data is done completely transparently, though you must subset to retrieve data past the end of the history.  When you request some channel data, for example by doing MyChannel[0,4000], DAQFactory will first check the channel's history in memory as determined by its History Length and retrieve it from there if it is all available.  If not, it will automatically go to the disk and retrieve it from the disk.  So, if your History Length is set to the default 3600, MyChannel[0,4000] will retrieve the data from the persistence file, but MyChannel[0,3000] will retrieve it from memory.  The only exception is when you do not subset at all, i.e. MyChannel.  This will retrieve the history as determined by the History Length.  This is largely to keep you from accidentally retrieving a huge number of data points from the persist file inadvertently, which would occur because of DAQFactory's expression colorization as you typed your expressions.

This last paragraph is an important point when determining what to set History and Persistence length to.  If you don't need to keep a lot of historical data for a particular channel and you don't care if the historical data is reset the next time you start DAQFactory (separate to what is logged of course), then just use History Length and set Persistence Length to 0.  If, however, you do want to keep a lot of historical data, or you want your data to automatically be available if you restart DAQFactory, then set your Persistence Length to the desired amount.  Even if you use channel persistence, you will want to set your History Length to a non-zero value as well, otherwise DAQFactory will be forced to go to the disk every time it needs data from the channel.  At a minimum, your History Length should be 1, so that all [0] requests for the most recent value pull from memory, but remember it only takes 16 bytes per data point, so if you have a display of the mean of the last 100 points, you probably should set your History Length to at least 100.  What exactly to set it to depends on the situation.  Just remember that data retrieved from memory is much faster than data retrieved from the persistence file.  At the same time, we recommend keeping History Length below 50,000 to 100,000.

Some important points and tips on channel persistence:

· Channel persistence is NOT data logging.  Data logging is completely separate within DAQFactory.  Channel persistence should NOT be used for long term archiving of data.  You should always use a logging or export set to log your data into a ASCII file or to an ODBC database so you can retrieve it from other programs.  This can easily be done simultaneously to channel persistence.  The persistence files are in an optimized binary format and are not readable from other programs.  They are stored in a rolling buffer, meaning that once the Persistence Length is reached, the oldest data is overwritten.  Channel persistence is simply a way to allow quick access to long term data from within DAQFactory.

· Channel persistence does not work with string, spectral or image data.  It only works with normal scalar values.

· Changing the Persistence Length for a channel will clear any data in the persistence file.

· Clearing the history of a channel, for example with the ClearHistory() function will clear the persistence file as well (though it won't change its apparent file size).

· All subsetting functions work with persisted data transparently.  This includes subsetting by time, which makes it really easy to query for data from a particular time frame.  As stated above, the exception is when you specify a channel name without any subsetting.  In this case, only the history determined by the History Length is retrieved.  If no data has been acquired, this means it could be from the persistence file.  

· The Persistence Length is limited to less than 100 million data points.  This is because when subsetting, a number larger than 100 million tells DAQFactory that you are subsetting by time, not data point.  Presumably, you could set the Persistence Length larger than 100 million, but in order to access any points past 100 million you'd have subset by time and not point number.

· If you find DAQFactory is bogging down as it runs, check your expressions.  You may be inadvertently retrieving a large amount of data from the persistence file.  Data retrieval from the persistence file is very fast, but remember that a page component will repetitively retrieve data with each page refresh, so also watch your page refresh rates.  

· Because specifying just a channel name with no subsetting only returns a maximum of History Length points, doing NumRows(MyChannel) will not return the full number of data points available from the persistence file, but instead the number of points available up to the History Length.  To retrieve the full number of data points in the persistence file, use the GetHistoryCount() function of your channel: Len = MyChannel.GetHistoryCount()

· If you want to analyze a very large subset of your data, say over a million data points (or perhaps less), you should avoid doing it directly in a page component.  For example, if you want to display the maximum value read over the last year at 1 hz, you should not do: Max(MyChannel[0,31536000]) in a Display Value Component as the component will force those 31 million data points to be loaded into memory every time it refreshes.  Instead, you should use a sequence and calculate it once there and store the result in a variable that is displayed.  Of course this will still require DAQFactory to load 31 million data points requiring 504meg of memory, so you should think about alternative ways to do this.  For example, load and analyze the data in smaller chunks:

 

Private x

Private maxs

for (x = 0, x < 100, x++)

  try

    maxs[x] = max(MyChannel[x * 100000,(x+1)*100000-1])

  catch()

  endcatch

endfor

return(max(maxs))

In this example, we determine the max of chunks of 100,000 data points and put it in the maxs array, then find the max of that array.  This keeps our maximum memory load to 1.6 meg instead of 160 meg required if we did it in one step.  Notice that we use a try/catch block as we may subset past the end of the persistence file and don't want an error, just an empty array item.  You could also use ignore("all") if you didn't need any other error handling.  A better way would be to simply use GetHistoryCount() to avoid subsetting past the end.

There are other ways of doing this.  For example, you could precalc the max on startup and then use the channel event to update the max if needed.  With channel persistence, DAQFactory makes very large data sets easy to deal with, but you still have to think about what you are doing and what DAQFactory is doing for you.  Because systems are different, we cannot make it all completely automatic.  

· A common use for channel persistence is to allow the user to scroll the graph back a long distance in time.  To get the graph to retrieve the persist data you will need to subset.  Fortunately, you can reference the bottom axis scaling timestamps by using the two variables: bottomAxis.currentScaleFrom and bottomAxis.currentScaleTo.  These variables hold the current actual scaling parameters, which would include when the graph is frozen due to zooming or panning.  These variables are different from strScaleFrom and strScaleTo which are just the Expressions used for scaling.   So, for example you could set your graph expression to MyChannel[BottomAxis.currentScaleFrom,BottomAxis.currentScaleTo] vs Time.  If you don't subset like this you will waste a lot of processor power loading data from the disk that is never used, so try and only request the data that is needed.  This is actually good practice even if you are retrieving data from memory.

· Another use of channel persistence is to allow you to restart DAQFactory without losing the data used for graphs and calculations.  Since the persistence files remain when DAQFactory stops, they are available when it restarts so even though no data may have been acquired, your graphs can automatically retrieve the older data from the persist files.

· The channel table in the channel view will only display up to the History Length data points.  Again, this is so you don't inadvertently retrieve a large number of data points.