Hi Sean,
So essentially, you're saying that it is the responsibility of the
writer of the .ncml writer to put the files in the right order, they
can't just specify the files in any order and expect netcdf-java to
order them? Okay...
In that case, the issue is only with geotools which is assuming an
increasing order (not decreasing).
Regards
Niels
On 06-10-16 21:18, Sean Arms wrote:
Hi Niles,
So, quick question - do you expect the valid time in the file
hycom_glb_regp01_old_t000.nc <http://hycom_glb_regp01_old_t000.nc> to
be before or after the valid time in hycom_glb_regp01_old_t003.nc
<http://hycom_glb_regp01_old_t003.nc>?
Here is what I could tell from the actual data in the files:
hycom_glb_regp01_old_t000.nc <http://hycom_glb_regp01_old_t000.nc>:
time -> 2014-07-08T00:00:00Z (127248.0 hours since 2000-01-01)
hycom_glb_regp01_old_t003.nc
<http://hycom_glb_regp01_old_t003.nc>: time -> 2014-07-07T03:00:00Z
(127227.0 hours since 2000-01-01)
hycom_glb_regp01_t000.nc <http://hycom_glb_regp01_t000.nc>: time
-> 2014-07-08T00:00:00Z(127248.0 hours since 2000-01-01)
hycom_glb_regp01_t003.nc <http://hycom_glb_regp01_t003.nc>: time
-> 2014-07-07T03:00:00Z (127227.0 hours since 2000-01-01)
Therefor, an aggregation listing the order such as:
hycom_glb_regp01_old_t000.nc <http://hycom_glb_regp01_old_t000.nc>
hycom_glb_regp01_old_t003.nc <http://hycom_glb_regp01_old_t003.nc>
will result in times decreasing in values (2014-07-08T00:00:00Z,
2014-07-07T03:00:00Z) because that's what is in the actual data files.
However, if the expectation is that the valid time in _t003 is after
the valid time in _t000, then things might only look correct if you
reverse the aggregation order.
Sean
On Thu, Oct 6, 2016 at 12:33 PM, Sean Arms <sarms@xxxxxxxx
<mailto:sarms@xxxxxxxx>> wrote:
Hi Niles,
If you specify individual files for aggregation, they will be in
the order you specify (which is exactly what you have found). If
you instead to a scan to aggregate the file, the files will be
aggregated in lexicographical order (for example, the order shown
by the command `ls`). This may or may not be in the appropriate
order. This is one of many tricky things about using NcML
aggregations. This is also one of the motivating reasons behind
the concept of featureCollections in the TDS. In a
featureCollection (Forecast Model Run Collections (FMRC), for
example), the coordinate systems of all of the files in a
collection are examined, and the TDS tries to "do the right thing"
with them with minimal configuration.
Cheers,
Sean
On Wed, Oct 5, 2016 at 3:32 AM, Niels Charlier <niels@xxxxxxxxx
<mailto:niels@xxxxxxxxx>> wrote:
Following this question, what is going to happen in this
regard if you do a <scan> to list the aggregation files? Will
they be put in an appropriate order?
Thanks for your answer.
Regards
Niels
On 09/30/2016 12:16 PM, Niels Charlier wrote:
Hi Everyone,
I have changed my mind on this other aggregation issue I
discussed in one of my emails (see below), I said it was
unrelated to netcdf-java but it doesn't seem to be the
case. Well, I really want to hear your opinion. The issue
is related to the order of files listed for aggregations.
I have found that the issue lies in part with geotools,
because it is making certain assumptions about time
variables being increasing rather than decreasing.
Convention does demand that a variable is either one of
the two (but not in random order).
The question is whether the list of files in an .ncml
files should also be in a certain order. It seems that
netcdf-java (which does the actual aggregating of the
variables) follows the order from the ncml code. That
means that the values inside the aggregated variable may
end up in random order.
For example, assume you have two files.
One has time values 1,2,3. The the other one has 4,5,6.
If we now place them in opposite order in our .ncml file,
netcdf-java appears to create an aggregated variable with
values 4,5,6,1,2,3
In that case, geotools cannot even assume that the
extremes are on both ends of the list, which is IMO faulty
and can cause many issues with other assumptions in the code.
Have I found something here, or do you disagree? Thank you
for responding.
Should we just demand from users to place the list of file
sin the same order as the order of values inside the .nc
files (increasing/decreasing)? is that (un)reasonable?
Kind Regards
Niels
On 08-09-16 14:51, Niels Charlier wrote:
3. changing the order of the list of files in the
inner aggregation tags.
<netcdf
location="hycom_glb_regp01_old_t003.nc
<http://hycom_glb_regp01_old_t003.nc>"/> <netcdf
location="hycom_glb_regp01_old_t000.nc
<http://hycom_glb_regp01_old_t000.nc>"/>
instead of <netcdf
location="hycom_glb_regp01_old_t000.nc
<http://hycom_glb_regp01_old_t000.nc>"/> <netcdf
location="hycom_glb_regp01_old_t003.nc
<http://hycom_glb_regp01_old_t003.nc>"/>
Otherwise the one with the earliest time
coordinates is last in the list instead of first...
and geotools fails on this. Is this normal?
This issue is definitely unrelated to netcdf-java,
but I am not sure whether this is a bug in geotools or
in the .ncml file.
_______________________________________________
NOTE: All exchanges posted to Unidata maintained email
lists are
recorded in the Unidata inquiry tracking system and made
publicly
available through the web. Users who post to any of the
lists we
maintain are reminded to remove any personal information
that they
do not want to be made public.
netcdf-java mailing list
netcdf-java@xxxxxxxxxxxxxxxx
<mailto:netcdf-java@xxxxxxxxxxxxxxxx>
For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/
<http://www.unidata.ucar.edu/mailing_lists/>
_______________________________________________
NOTE: All exchanges posted to Unidata maintained email lists are
recorded in the Unidata inquiry tracking system and made publicly
available through the web. Users who post to any of the lists we
maintain are reminded to remove any personal information that they
do not want to be made public.
netcdf-java mailing list
netcdf-java@xxxxxxxxxxxxxxxx <mailto:netcdf-java@xxxxxxxxxxxxxxxx>
For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/
<http://www.unidata.ucar.edu/mailing_lists/>
