These are chat archives for Fortran-FOSS-Programmers/General-Discussion
Dear all, I am going to define the IO format for a new CFD code :tada: :tada: :tada:
I am really in doubt for selecting a convenient approach for my needs. Essentially the new code is a multi-block, Chimera-AMR, finite volume. In the past I have adopted two different approaches:
In fact, the new code will have AMR capability (the others did not) thus the work-load on each processor is not static for the whole simulation, rather it changes step by step. Thus, each processor has a varying number of blocks in charge thus I cannot simply have the pattern 1-processor=> 1-multi-block-file, as I did in the past.
The simplest solution for me is a new pattern 1-block=>1-file : surely it will be parallel-efficient. The cons is that for production simulation I will have billions of files for time-accurate unsteady sims (1000-blocks X 1000... steps...) :cry:
Other possibilities that I see are:
Any suggestions are much more than welcome.
My best regards.
I think I found a compromise: each processor/image will IO its own 1-file containing the blocks handled at that time step. The file format will be a sort of XML tagged format to allow dynamic workload and un-ordered list of block-id/refinement-level, namely to allow each processor/image to change its workload at runtime. However, this will require an extra post-processor for collect the simulation results in order to be re-used by other runs with different number of processors/images.
I think that to have only 1 file independently of the number of processors/blocks only MPI IO or HDF5 can help. I considered also direct-access file format (each processor/image access to only its own part of the file to avoid race condiction) but it sound tricky...
@giacombum I think that HDF5 and CAF can be mixed almost smoothly, maybe @zbeekman or @rouson can comment this.
MPI IO: I never used it, I could re-learn, but due to the fact that I hope to switch soon from MPI to CAF, I am really in doubt if it is worth to invest my time on it;
HDF5: I never used it, but when @francescosalvadore did show it to me it looked amazing; one issue is that some of my bosses have issues (old CentOS-based workstations) on installing exotic libraries (I had issues in installing Python 2.7 that has about 10 years...): remain in Fortran/MPI field could be better for them.
These are the two solutions I would recommend. Also, N.B. CAF can be safely mixed with MPI (in theory I have yet to try) at least for OpenCoarrays. The distributed shared memory OpenCoarrays implemention is built ontop of MPI, but we create private communicators, so it should be safe to use with other MPI elements. I have little experience with MPI-IO but it may give you the best fine grained control if that is critical to your application. I tried parallel HDF5 (also built on MPI, also should be safe to mix with OpenCoarrays) a few years ago and hit some painful bugs with LustreFS and HDF5. Hopefully these are all fixed now and you can use parallel HDF5 without issues. I know, for example, US3D uses HDF5 for it's file format.