[thredds] Detaails as to how recheckEvery works, and optimizing access to large aggregate datasets

  • To: THREDDS THREDDS <thredds@xxxxxxxxxxxxxxxx>
  • Subject: [thredds] Detaails as to how recheckEvery works, and optimizing access to large aggregate datasets
  • From: Roy Mendelssohn <Roy.Mendelssohn@xxxxxxxx>
  • Date: Wed, 28 Nov 2007 14:39:58 -0800
Hi:
We are working with someone who is implementing a TDS for a large aggregated dataset, in this case output from ROMS models. The specific questions asked were:
Will recheckEvery also check existing files to see if they have changed (had records added)? I only ask because, as I mentioned, we are considering adding the hourly files to the current month's file. Therefore, the current month's file will change (have a new record) every hour.
Also, when the aggregation finds a new file (or modified one if it checks that) will the dataset need to be retouched to speed load times?
but the underlying question is also how best to set up the aggregation, is it still true that things are speeded up if on startup you "touch" each "file" so that the coordinate info is read into cache, and exactly what does get checked and how on recheckEvery. 

Thanks for any help,

-Roy
.



**********************
"The contents of this message do not reflect any position of the U.S. Government or NOAA." 
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division 
Southwest Fisheries Science Center
1352 Lighthouse Avenue
Pacific Grove, CA 93950-2097

e-mail: Roy.Mendelssohn@xxxxxxxx (Note new e-mail address)
voice: (831)-648-9029
fax: (831)-648-8440
www: http://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill."