Re: [netcdfgroup] ZFP compression for NetCDF4

  • To: Leigh Orf <leigh.orf@xxxxxxxxx>
  • Subject: Re: [netcdfgroup] ZFP compression for NetCDF4
  • From: "H. Joe Lee" <hyoklee@xxxxxxxxxxxx>
  • Date: Tue, 23 Mar 2021 12:46:58 -0500
  • Authentication-results: unidata.ucar.edu; dkim=none (message not signed) header.d=none;unidata.ucar.edu; dmarc=none action=none header.from=hdfgroup.org;
Please check this out:

https://github.com/HDFGroup/hdf5_plugins/

- Joe
---
Deploy XaaS quickly using HDF in Distributed Cloud.




On Tue, Mar 23, 2021 at 12:03 PM Dennis Heimbigner <dmh@xxxxxxxx> wrote:

> If you have the HDF5 ZFP plugin available, then you can certainly
> use it in netcdf.
> Some documentation on netcdf filter support is here:
>  >https://github.com/Unidata/netcdf/blob/master/NUG/filters.md
>
> =Dennis Heimbigner
>    Unidata
>
>
> On 3/23/2021 10:52 AM, Leigh Orf wrote:
> > Hello,
> >
> > I'm a long time user of HDF and netcdf. I create lots of data in my
> > research (severe weather simulation) and ZFP compression in HDF5 files
> > has been great. I'd like to be able to create netcdf4 files
> > (preferably using the netcdf API) with 3D fields compressed using the
> > lossy ZFP compression algorithm. I know that netcdf4 is HDF5 so it's
> > certainly possible.
> >
> > I'm wondering whether anyone could provide any pointers. I see 3
> > options really.
> >
> > 1. Create ZFP compressed netcdf4 files using the normal netCDF-C API
> > (my code is mostly C)
> > 2. Create ZFP compressed netcdf4 files using the HDF5 API (including
> > all the HDF5 special sauce that makes netcdf4 netcdf4)
> > 3. Use a version of nccopy that understands ZFP
> >
> > Are any of these options possible right now? If not where are we on
> > efforts to include ZFP into netCDF? I did grab the latest code on
> > github and saw references to ZFP but no filter code.
> >
> > Leigh Orf
> > UW-Madison
> >
> > _______________________________________________
> > NOTE: All exchanges posted to Unidata maintained email lists are
> > recorded in the Unidata inquiry tracking system and made publicly
> > available through the web.  Users who post to any of the lists we
> > maintain are reminded to remove any personal information that they
> > do not want to be made public.
> >
> >
> > netcdfgroup mailing list
> > netcdfgroup@xxxxxxxxxxxxxxxx
> > For list information or to unsubscribe,  visit:
> https://www.unidata.ucar.edu/mailing_lists/
>
> _______________________________________________
> NOTE: All exchanges posted to Unidata maintained email lists are
> recorded in the Unidata inquiry tracking system and made publicly
> available through the web.  Users who post to any of the lists we
> maintain are reminded to remove any personal information that they
> do not want to be made public.
>
>
> netcdfgroup mailing list
> netcdfgroup@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe,  visit:
> https://www.unidata.ucar.edu/mailing_lists/
>