Hi Chris:
Very good points about data models/formats becoming complex. There are a number of features that appear to complicate things without obvious corresponding benefit. Perhaps it would be helpful if we start to enumerate exactly what are the basic features and which not?
As I mentioned in a previous post, the checksum idea was to verify which table
was used, rather than message integrity, which can now be assumed to be handled
at a lower layer. Whether that's an idea whose complexity is worth it is
another issue!
Some bufr complexity appears to be motivated by keeping the record small. I
wonder whether a simpler data layout with dictionary based compression (eg LZW)
might not work better, now that CPUs are so much faster than IO. I will be
doing some testing with that idea, once my decoder is reliably working.
Regards,
John
Little, Chris wrote:
Hi Milan and John,
Perhaps I can usefully contribute here.
We did originally have a checksum for each of the main sections of BUFR, but
the justification was mainly to do with unreliable bit orientated telecoms, and
limits on block sizes. We addressed those by saying that the telecoms protocol
should handle them. We wanted a clean, orthogonal separation of transport
protocol and content.
We never considered correcting errors between tables, messages and
applications. We didn't even envisage more than one Master Table - multiple
master tables were a bit of an afterthought, to accommodate the oceanographers.
We never really thought through in detail how things would be administered. It
was left as a trivial exercise for the readers!
From the outside, I can see you all adding bits here and there to fix things. I would say Bufr is now in its 'baroque' or 'rococo' phase, and I would recommend some serious pruning/deprecation/simplification/re-factoring to get back to a more 'classical' structure, perhaps a 'BUFR- lite'. I think complexity, even if it is necessary, seems to be a barrier to wider take up of BUFR. I think the success of NetCDF and HDF came from simplicity and elegance first (and supplied software and APIs), and then complexity came after it had been taken up and the community recognised that the extras were needed.
I hope you can come up with a straightforward table maintenance process.
Best wishes, Chris
Chris Little International Telecommunications
Met Office FitzRoy Road Exeter Devon EX1 3PB United Kingdom
Tel: +44(0)1392 886278 Fax: +44(0)1392 885681 Mobile: +44(0)7753 880514
E-mail: chris.little@xxxxxxxxxxxxxxxx
Met Office climate change predictions can be viewed on Google Earth
http://www.metoffice.gov.uk/research/hadleycentre/google/
