John:
It seems that we are going to a review of MEAD project. MEAD is an NSF
expedition project back to 4 years ago. At first, we didn't committed to
provide either an operational parallel or a sequential IO module.s. What we
did is to do a kind of expedition, and further more to investigate the
performance of HDF5. However, at the end of the project, John Mialakes
would like to have our parallel version into WRF release. Since I have been
spending limited time maintaining the module even after the project ended a
year ago. Sequential version has never been into WRF release since WRF has
already had several sequential IO modules. I did remember I checked the
output carefully about dim. scale at that time; I should realize the
reverse order of dim-rank. Somehow it just slipped away from my finger. In
the future I need to review them again, I just don't have time now. As for
parallelzation, WRF does parallelism in
spatially(domain decompositions). Time won't affect parallelzation.
As for the last few questions you ask:
We can "extend" the data this way but just not use chunking storage in WRF.
I am not sure your last few questions. The bottom line is to store data in
contiguous storage won't affect parallelzation at all. It just add a little
bit more meta-data.
If you want to know more about this, you can go to
http://hdf.ncsa.uiuc.edu/apps/WRF-ROMS to read more report about this work;
especially the article at
http://hdf.ncsa.uiuc.edu/apps/WRF-ROMS/performance-report.pdf may help
answer your questions.
Kent
At 09:32 AM 3/31/2005, you wrote:
Hi Kent:
So do you think no one will use this particular format? If so, I will
wait for new version..
I don't know. Somebody from OU contacted with me about using WRF-PHDF5.
But I didn't hear from them about this issue.
since its also in the sequential version, i guess the question is, do you
know if WRF is planning on using either version?
BTW, in the parallel output (but not the sequential), you put each time
step in its own group. was this to optimize parallel performance or something ?
The reason is: we are using contiguous storage for parallel HDF5 because
of performance issue. So that's why we put each time step in to its own
group. I hope that in the future I can have time or maybe a little
funding to modify the WRF-HDF5 IO modules with 1.8 release.
so you use contiguous, not chunked storage. so you cant extend the data ?
so you create a new group as each time step completes ?
how do you parellelize the tasks? does each thread have its own seperate
region of the array?
