Hi David,
NetCDF4 is using parallel HDF5 underneath to do parallel IO. As far
as I know, I don't think HDF5 is doing anything like blocking in its
layer. Other experts, please correct me if I am wrong.
If seems that your application fits to what we called irregular
selection of data. Parallel HDF5 does support irregular selection of
data via collective IO.
However, the current netcdf-4 API may not support this yet. One
workaround solution is see whether you can combine your cells into
several regular selections and then use NF90_PUT_VAR. From your
example, you can put (1 2) of cell 1 and (3 4) of cell 2 into one
regular selection and call NF90_PUT_VAR.
Kent Yang
The HDF Group
At 10:39 AM 4/26/2007, David Stuebe wrote:
Hi NETCDF folks
I work on an unstructured finite volume coastal ocean model, FVCOM,
which is parallel (using MPICH2). The Read Write is a major slow
down for our large cases. On our cluster, we have one large storage
device, an emc raid array. The network is infini-band - the network
is much faster than the raid array.
For our model we need to read large initial condition data sets, and
single frames of forcing data while running. We also need to write
single frames of data for output (frequently), and large restart
files (less frequently).
I am considering two options for recoding the IO from the model. One
is based around the future F90 netcdf 4 parallel interface which
would allow a symmetric code- every processor does the same thing.
The other option is to use netcdf 3, let the master processor
read/write the data and distribute it to each node, -an asymmetric coding.
What I need to know- are netcdf 4 parallel IO operations blocking?
The problem - the order of cells and nodes in our data set does not
allow for a simple start, count read format. A data array might have
dimensions (time,layers,cells). As an example, in a 2 processor
case with 8 cells, proc1 has cells(1 2 5 7) while proc2 has cells (3
4 6 8) - write operations would have to be in a do loop to write
each cell individually from the processor that owns it.
For a model with 300,000 cells on 30 processors, this would be
10,000 calls to NF90_PUT_VAR on each processor. Even if the calls
are non-blocking this seems dangerous.
Any thoughts?
David
==============================================================================
To unsubscribe netcdf-hdf, visit:
http://www.unidata.ucar.edu/mailing-list-delete-form.html
==============================================================================
