How so?
To write a parser for a binary file one needs to know the detailed structure of
the file - and if that changes your code is now broken. With GML or any
XML-based encoding the file structure is not known to your application. Where
do you see the problems in parsing and understanding GML?
R
-----Original Message-----
From: Gerry Creager [mailto:gerry.creager@xxxxxxxx]
Sent: August 21, 2009 1:17 PM
To: Ron Lake
Cc: John Caron; Unidata GALEON; wcs-2.0.swg
Subject: Re: [galeon] [WCS-2.0.swg] CF-netCDF standards initiatives
I'm not so sure that's true.
Ron Lake wrote:
> Hi John:
>
> Surely the GML encoding is going to be simpler to parse and "understand" than
> any equivalent binary encoding.
>
> R
>
> -----Original Message-----
> From: galeon-bounces@xxxxxxxxxxxxxxxx
> [mailto:galeon-bounces@xxxxxxxxxxxxxxxx] On Behalf Of John Caron
> Sent: August 21, 2009 12:52 PM
> Cc: Unidata GALEON; wcs-2.0.swg
> Subject: Re: [galeon] [WCS-2.0.swg] CF-netCDF standards initiatives
>
> A few thoughts on these issues:
>
> The WCS/SWE/OGC are making major contributions in elaborating conceptual
> models and defining remote access protocols based on them. More
> problematic is managing complexity and adoption paths of existing
> communities. One of the main weaknesses of WCS is the likely
> proliferation of binary encoding formats. WMS has been lucky to be able
> to use existing, universal formats like JPEG, and partly because of
> that, has been more widely adopted.
>
> A central issue is the relation of the WCS data encoding format to its
> GML envelope. Should the data be "just bits" and let the GML encode all
> the semantics, or should a binary format also encode some/all of the
> semantics? In the first case, any client must understand GML, which is a
> big barrier to use by existing applications, plus the possibility exists
> that some essential information is offline or unavailable, as Tom W
> points out. In the second case, we have a proliferation of data encoding
> formats and the possibility that the semantics are encoded differently
> between the GML and the data.
>
> From Unidata's POV, namely that of supporting a large community of
> users of netcdf and other scientific data formats, using netCDF-CF to
> encode the data is obvious. However, this is not just a matter of one
> more group advocating for their favorite bespoke format. IMHO, netCDF is
> close to being a "maximally simple implementation" of a binary data
> encoding that allows data to be more than "just numbers". Its BNF
> grammer fits on one side of a page of paper. If one chooses to encode
> semantics into the data format, and not require a client to know GML,
> then you need something like netCDF, and you wont find an encoding
> significantly simpler. The CF Conventions group has been slowly adding
> the necessary semantics for 10 years or so. From that motivation we
> decided to recommend netCDF as a WCS encoding format.
>
> The hard part is describing in a formal way the relationship of the
> netCDF-CF binary response to the actual request and to any GML part of
> the response. Stefano et al are doing the heroic work of describing the
> mapping between the two models, but there is and perhaps should be a
> loose coupling between the two. There are always edge cases: what
> happens when the requested lat/lon bounding box crosses a discontinuity
> of the projection plane? One might get different, but reasonable, files
> back from different servers of the same dataset. So what can we
> standardize? None of this is unique to netcdf-CF or binary encodings.
>
> Taking the first path of letting GML carry all the semantics has many
> advantages also, especially for new development and GIS-centric clients,
> and as new libraries are developed that can mitigate some of the
> complexity of GML. In (our) scientific community, these are new
> possibilities, to be experimented with and prototypes tested. Once there
> are real services delivering "gotta have" data we will do whatever we
> need to do to "get me some of that" ;^)
>
> John Caron
> Unidata
>
> Steve Hankin wrote:
>>
>> Robin, Alexandre wrote:
>>> Hi Steve,
>>>
>>> Just to clarify when I said NetCDF was a "NEW standard" I meant a new
>>> standard in OGC.
>>>
>>> As I was telling Ben in an offline email, I am totally conscious of
>>> its penetration and usefulness in certain communities.
>>>
>>> However, _I am not convinced that /having two standards doing the
>>> same thing in OGC /is sending the right message and is the best way
>>> to go for a standardization organization_.
>>>
>> Hi Robin,
>>
>> I confess that I was aware of using a cheap rhetorical device when I
>> twisted your intended meaning of "NEW". (Begging your tolerance.) It
>> was helpful in order to raise more fundamental questions. You have
>> alluded to a key question just above. Is it really best to think of
>> the target of OGC as a the development of a single, definitive
>> standard? one that is more general and more powerful than all existing
>> standards? Or is it better to think of OGC as a process, through which
>> the forces of divergence in geospatial IT systems can be weakened
>> leading towards convergence over time? The notion that there can be a
>> single OGC solution is already patently an illusion. Which one would
>> you pick? WFS? WCS? SOS with SWE Common? SOS with its many other XML
>> schema? (Lets not even look into the profusion of WFS application
>> schema.) I trust that we are not pinning our hopes on a future
>> consolidation of all of these. There is little evidence to indicate
>> that we can sustain the focus necessary to traverse that path. The
>> underlying technology is not standing still.
>>
>> What Ben (and David Arctur and others) have proposed through seeking
>> to put an OGC stamp of approval on netCDF-CF technology is similar to
>> what OGC has achieved through putting its stamp on KML ("There are
>> sound business and policy reasons for doing so.") It is to create a
>> process -- a technical conversation if you will -- which will lead to
>> interoperability pathways that bridge technologies and communities.
>> Real-world interoperability.
>>> There has been a lot of experimentation with SWE technologies as well
>>> that you may not know about and in many communities, especially in
>>> earth science.
>>>
>>> What I'm saying is that perhaps it is worth testing bridging NetCDF
>>> to SWE before we go the way of stamping two 100% overlapping
>>> standards as OGC compliant.
>>>
>> Complete agreement that this sort of testing ought to occur. And
>> interest to hear more about what has been achieved. But great
>> skepticism that there is this degree of overlap between the
>> approaches. And disagreement that this testing ought to be a
>> precondition to OGC recognition of a significant ,community-proven
>> interoperability mechanism like netCDF. OGC standardization of netCDF
>> will provide a forum for testing and experimentation to occur much
>> more rapidly and for a 2-way transfer of the best ideas between
>> approaches. NetCDF & co. (its API, data model, CF, DAP) have a great
>> deal to offer to OGC.
>>
>> - Steve
>>> Regards,
>>>
>>> -------------------------------------------------
>>>
>>> **Alexandre Robin**
>>>
>>> Spot Image, Web and E-Business
>>>
>>> Tel: +33 (0)5 62 19 43 62
>>>
>>> Fax: +33 (0)5 62 19 43 43
>>>
>>> http://www.spotimage.com
>>>
>>> Before printing, think about the environment
>>>
>>> ------------------------------------------------------------------------
>>>
>>> *De :* Steve Hankin [mailto:Steven.C.Hankin@xxxxxxxx]
>>> *Envoyé :* jeudi 20 août 2009 20:58
>>> *À :* Tom Whittaker
>>> *Cc :* Robin, Alexandre; Ben Domenico; Unidata GALEON; wcs-2.0.swg
>>> *Objet :* Re: [galeon] [WCS-2.0.swg] CF-netCDF standards initiatives
>>>
>>> Hi Tom,
>>>
>>> I am grateful to you for opening the door to comments "from 10
>>> thousand feet" -- fundamental truths that we know from many years of
>>> experience, but that we fear may be getting short shrift in
>>> discussions of a new technology. I'd like to offer a comment of that
>>> sort regarding the interplay of ideas today between Robin ("/I hope
>>> we don't have to define a NEW standard .../") and Carl Reed ("/there
>>> are other organizations interested in bringing legacy spatial
>>> encodings into the OGC. There are sound business and policy reasons
>>> for doing so./").
>>>
>>> The NEW standard in this discussion is arguably SWE, rather than
>>> netCDF. NetCDF has decades of practice behind it; huge bodies of data
>>> based upon it; a wide range of applications capable of accessing it
>>> (both locally and remotely); and communities that depend vitally upon
>>> it. As Ben points out, netCDF also has its own de jure pedigree.
>>>
>>> A key peril shared by most IT standards committees -- a lesson that
>>> has been learned, forgotten, relearned and forgotten again so many
>>> times that it is clearly an issue of basic human behavior -- is that
>>> they will try to innovate. Too-common committee behavior is to
>>> propose, discuss and document new and intriguing technologies, and
>>> then advance those documents through a de jure standards process,
>>> despite an insufficient level of testing. The OGC testbed process
>>> exists to address this, but we see continually how large the gap is
>>> between the testbed process and the pace and complexity of
>>> innovations emerging from committees.
>>>
>>> Excellent reading on this subject is the essay by Michi Henning, /The
>>> Rise and Fall of CORBA/ (2006 --
>>> http://queue.acm.org/detail.cfm?id=1142044). Among the many insights
>>> he offers is
>>>
>>> **'Standards consortia need iron-clad rules to ensure that they
>>> standardize existing best practice.** There is no room for innovation
>>> in standards_._ Throwing in "just that extra little feature"
>>> inevitably causes unforeseen technical problems, despite the best
>>> intentions.'
>>>
>>> While it adds weight to an argument to be able to quote from an
>>> in-print source, this is a self-evident truth. We need only reflect
>>> on the recent history of IT. What we need is to work together to find
>>> ways to prevent ourselves from continually forgetting it.
>>>
>>> There is little question in my mind that putting an OGC stamp of
>>> approval on netCDF is a win-win process -- for the met/ocean/climate
>>> community and for the broader geospatial community. It will be a path
>>> to greater interoperability in the long run and it deserves to go
>>> forward. The merits of SWE (or GML) as an alternative approach to the
>>> same functionality also deserve to be explored and tested in
>>> situations of realistic complexity. But this exploration should be
>>> understood initially as a process of R&D -- a required step before a
>>> "standards process" is considered. If that exploration has already
>>> been done it should be widely disseminated, discussed and evaluated.
>>>
>>> - Steve
>>>
>>> ==================================
>>>
>>> Tom Whittaker wrote:
>>>
>>> I may be ignorant about these issues, so please forgive me if I am
>>> completely out-of-line....but when I looked at the examples, I got
>>> very concerned since the metadata needed to interpret the data values
>>> in the "data files" is apparently not actually in the file, but
>>> somewhere else. We've been here before: One of the single biggest
>>> mistakes that the meteorological community made in defining a
>>> distribution format for realtime, streaming data was BUFR -- because
>>> the "tables" needed to interpret the contents of the files are
>>> somewhere else....and sometimes, end users cannot find them!
>>>
>>> NetCDF and ncML maintain the essential metadata within the files:
>>> types, units, coordinates -- and I strongly urge you (or whomever) not
>>> to make the "BUFR mistake" again -- put the metadata into the files!
>>> Do not require the end user to have to have an internet connection to
>>> simply "read" the data....many people download the files and then
>>> "take them along" when traveling, for example.
>>>
>>> If I simply downloaded the file at
>>> <http://schemas.opengis.net/om/1.0.0/examples/weatherObservation.xml>
>>> I would not be able to read it. In fact, it looks like even if I also
>>> got the "metadata" file at:
>>> <http://schemas.opengis.net/om/1.0.0/examples/weatherRecord1.xml>
>>> I would still not be able to read it, since it also refers to other
>>> servers in the universe to obtain essential metadata.
>>>
>>> That is my 2 cents worth....and I hope I am wrong about what I saw in
>>> the examples....
>>>
>>> tom
>>>
>>>
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> galeon mailing list
>> galeon@xxxxxxxxxxxxxxxx
>> For list information, to unsubscribe, visit:
>> http://www.unidata.ucar.edu/mailing_lists/
>
> _______________________________________________
> galeon mailing list
> galeon@xxxxxxxxxxxxxxxx
> For list information, to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
>
> _______________________________________________
> galeon mailing list
> galeon@xxxxxxxxxxxxxxxx
> For list information, to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
--
Gerry Creager -- gerry.creager@xxxxxxxx
Texas Mesonet -- AATLT, Texas A&M University
Cell: 979.229.5301 Office: 979.458.4020 FAX: 979.862.3983
Office: 1700 Research Parkway Ste 160, TAMU, College Station, TX 77843
