Re: [cf-satellite] Sharing quality flags among multiple variables

The same issue occurs with World Ocean Database which consists of mainly profile data. Each profile typically consists of several variables measured along the depth. The quality flags used for all the variable are same. I have attached a snippet below of how a CF compliant netCDF would look. The over-head due to repeating the quality flags is not much when the data is stored is a multi-profile format, but when each profile is stored in an individual file, the overhead becomes significant, since these flags have to be present in each file. 

Upendra


 float temperature(z) ;
                temperature:standard_name = "sea_water_temperature" ;
                temperature:long_name = "sea water temperature" ;
                temperature:units = "degree_C" ;
                temperature:coordinates = "time lon lat z" ;
                temperature:WODprofile_flag = 0 ;
                temperature:_FillValue = -1.e+10f ;
                temperature:grid_mapping = "crs" ;
temperature:cell_methods = "time:point longitude:point latitude: 
point z:point" ;
                temperature:ancillary_variables = "temperatu
re_WODflag" ;
        int temperature_WODflag(z) ;
temperature_WODflag:standard_name = "sea_water_temperature statu 
s_flag" ;
temperature_WODflag:flag_values = "0, 1, 2, 3, 4, 5, 6, 7, 8, 9" 
 ;
temperature_WODflag:flag_meanings = "accepted_value range_outlie r failed_inversion_check failed_gradient_check observed_level_bullseye_flag_and_ zero_gradient_check combined_gradient_and_inversion_checks failed_range_and_inve rsion_checks failed_range_and_gradient_checks failed_range_and_questionable_data 
_checks failed_range_and_combined_gradient_and_inversion_checks" ;
                temperature_WODflag:reference = "" ;


        float salinity(z) ;
                salinity:standard_name = "sea_water_salinity" ;
                salinity:long_name = "sea water salinity" ;
                salinity:units = "" ;
                salinity:coordinates = "time lon lat z" ;
                salinity:WODprofile_flag = 0 ;
                salinity:_FillValue = -1.e+10f ;
                salinity:grid_mapping = "crs" ;
salinity:cell_methods = "time:point longitude:point latitude:poi 
nt z:point" ;
                salinity:ancillary_variables = "salinity_WODfla
g" ;
        int salinity_WODflag(z) ;
salinity_WODflag:standard_name = "sea_water_salinity status_flag 
" ;
salinity_WODflag:flag_values = "0, 1, 2, 3, 4, 5, 6, 7, 8, 9" ; salinity_WODflag:flag_meanings = "accepted_value range_outlier f ailed_inversion_check failed_gradient_check observed_level_bullseye_flag_and_zer o_gradient_check combined_gradient_and_inversion_checks failed_range_and_inversi on_checks failed_range_and_gradient_checks failed_range_and_questionable_data_ch 
ecks failed_range_and_combined_gradient_and_inversion_checks" ;
                salinity_WODflag:comment = "" ;


        float oxygen(z) ;
oxygen:standard_name = "volume_fraction_of_oxygen_in_sea_water" 
;
                oxygen:long_name = "Oxygen" ;
                oxygen:units = "ml/l" ;
                oxygen:coordinates = "time lon lat z" ;
                oxygen:WODprofile_flag = 0 ;
                oxygen:_FillValue = -1.e+10f ;
                oxygen:grid_mapping = "crs" ;
oxygen:cell_methods = "time:point longitude:point latitude:point 
 z:point" ;
                oxygen:ancillary_variables = "oxygen_WODflag" ;
        int oxygen_WODflag(z) ;
oxygen_WODflag:standard_name = "volume_fraction_of_oxygen_in_sea 
_water status_flag" ;
oxygen_WODflag:flag_values = "0, 1, 2, 3, 4, 5, 6, 7, 8, 9" ; oxygen_WODflag:flag_meanings = "accepted_value range_outlier fai led_inversion_check failed_gradient_check observed_level_bullseye_flag_and_zero_ gradient_check combined_gradient_and_inversion_checks failed_range_and_inversion _checks failed_range_and_gradient_checks failed_range_and_questionable_data_chec 
ks failed_range_and_combined_gradient_and_inversion_checks" ;
                oxygen_WODflag:comment = "" ;


        float phosphate(z) ;
phosphate:standard_name = "mole_concentration_of_phosphate_in_se 
a_water " ;
                phosphate:long_name = "Phosphate" ;
                phosphate:units = "umol/l" ;
                phosphate:coordinates = "time lon lat z" ;
                phosphate:WODprofile_flag = 0 ;
                phosphate:_FillValue = -1.e+10f ;
                phosphate:grid_mapping = "crs" ;
phosphate:cell_methods = "time:point longitude:point latitude:po 
int z:point" ;
                phosphate:ancillary_variables = "phosphate_WOD
flag" ;
        int phosphate_WODflag(z) ;
phosphate_WODflag:standard_name = "mole_concentration_of_phospha 
te_in_sea_water status_flag" ;
phosphate_WODflag:flag_values = "0, 1, 2, 3, 4, 5, 6, 7, 8, 9" ; phosphate_WODflag:flag_meanings = "accepted_value range_outlier failed_inversion_check failed_gradient_check observed_level_bullseye_flag_and_ze ro_gradient_check combined_gradient_and_inversion_checks failed_range_and_invers ion_checks failed_range_and_gradient_checks failed_range_and_questionable_data_c 
hecks failed_range_and_combined_gradient_and_inversion_checks" ;
                phosphate_WODflag:comment = "" ;


        float silicate(z) ;
silicate:standard_name = "mole_concentration_of_silicate_in_sea_ 
water" ;
                silicate:long_name = "Silicate" ;
                silicate:units = "umol/l" ;
                silicate:coordinates = "time lon lat z" ;
                silicate:WODprofile_flag = 0 ;
                silicate:_FillValue = -1.e+10f ;
                silicate:grid_mapping = "crs" ;
silicate:cell_methods = "time:point longitude:point latitude:point z:point" ; silicate:ancillary_variables = "silicate_sigfigs silicate_WODflag" ; 
        int silicate_WODflag(z) ;
silicate_WODflag:standard_name = "mole_concentration_of_silicate_in_sea_water status_flag" ; silicate_WODflag:flag_values = "0, 1, 2, 3, 4, 5, 6, 7, 8, 9" ; silicate_WODflag:flag_meanings = "accepted_value range_outlier failed_inversion_check failed_gradient_check observed_level_bullseye_flag_and_zero_gradient_check combined_gradient_an d_inversion_checks failed_range_and_inversion_checks failed_range_and_gradient_checks failed_range_and_questionable_data_checks failed_range_and_combined_gradient_and_inversion_checks" ; 
                silicate_WODflag:comment = "" ;


        float nitrate(z) ;
nitrate:standard_name = "mole_concentration_of_nitrate_in_sea_water" ; 
                nitrate:long_name = "Nitrate" ;
                nitrate:units = "umol/l" ;
                nitrate:coordinates = "time lon lat z" ;
                nitrate:WODprofile_flag = 0 ;
                nitrate:_FillValue = -1.e+10f ;
                nitrate:grid_mapping = "crs" ;
nitrate:cell_methods = "time:point longitude:point latitude:point z:point" ; 
                nitrate:ancillary_variables = "nitrate_WODflag" ;
        int nitrate_WODflag(z) ;
nitrate_WODflag:standard_name = "mole_concentration_of_nitrate_in_sea_water status_flag" ; nitrate_WODflag:flag_values = "0, 1, 2, 3, 4, 5, 6, 7, 8, 9" ; nitrate_WODflag:flag_meanings = "accepted_value range_outlier failed_inversion_check failed_gradient_check observed_level_bullseye_flag_and_zero_gradient_check combined_gradient_and _inversion_checks failed_range_and_inversion_checks failed_range_and_gradient_checks failed_range_and_questionable_data_checks failed_range_and_combined_gradient_and_inversion_checks" ; 
                nitrate_WODflag:comment = "" ;


        float nitrite(z) ;
                nitrite:long_name = "Nitrite" ;
                nitrite:units = "umol/l" ;
                nitrite:coordinates = "time lon lat z" ;
                nitrite:WODprofile_flag = 0 ;
                nitrite:_FillValue = -1.e+10f ;
                nitrite:grid_mapping = "crs" ;
nitrite:cell_methods = "time:point longitude:point latitude:point z:point" ; 
                nitrite:ancillary_variables = "nitrite_WODflag" ;
        int nitrite_WODflag(z) ;
nitrite_WODflag:flag_values = "0, 1, 2, 3, 4, 5, 6, 7, 8, 9" ; nitrite_WODflag:flag_meanings = "accepted_value range_outlier failed_inversion_check failed_gradient_check observed_level_bullseye_flag_and_zero_gradient_check combined_gradient_and _inversion_checks failed_range_and_inversion_checks failed_range_and_gradient_checks failed_range_and_questionable_data_checks failed_range_and_combined_gradient_and_inversion_checks" ; 
                nitrite_WODflag:comment = "" ;

On 10/31/2011 12:12 PM, Randy Horne wrote:

In the application I am working on, there are specific types of products where 
where the same quality flags apply to multiple variables.  These products will 
be contained in a single NetCDF file.

The current CF conventions dictate that quality flags are attached to specific 
variables.  The implication is that comforming with CF conventions would 
require the same quality flags to be stored multiple times in our NetCDF 
product files.  This is potentially problematic due to the resulting increase 
in product file size.  Our variables have on the order of 10s of millions of 
data points where each has a distinct quality flag.

This issue has come up before, and any design to optimize flags has been 
dismissed due to the overriding objective of having variables that are 
self-describing.

Product file compression is a potential answer, but this presents its own set 
of performance issues.

Maybe flags could be handled in a similar manner as coordinates to allow 
sharing among variables ?



..............End of Message ...............................-->





_______________________________________________
cf-satellite mailing list
cf-satellite@xxxxxxxxxxxxxxxx
For list information or to unsubscribe, visit: 
http://www.unidata.ucar.edu/mailing_lists/