These are chat archives for Fortran-FOSS-Programmers/General-Discussion

8th
Feb 2017
Stefano Zaghi
@szaghi
Feb 08 2017 05:19

@/all

Dear all, I am going to define the IO format for a new CFD code :tada: :tada: :tada:

I am really in doubt for selecting a convenient approach for my needs. Essentially the new code is a multi-block, Chimera-AMR, finite volume. In the past I have adopted two different approaches:

  1. a single file which IO is synced by master processor: this is not a reall parallel IO approach due to the needed barrier, thus it is not efficient for large scaled cluster;
  2. each processor takes into account its own IO: this is very efficient because the IO is really parallel, but for the new code there are some issues...

In fact, the new code will have AMR capability (the others did not) thus the work-load on each processor is not static for the whole simulation, rather it changes step by step. Thus, each processor has a varying number of blocks in charge thus I cannot simply have the pattern 1-processor=> 1-multi-block-file, as I did in the past.

The simplest solution for me is a new pattern 1-block=>1-file : surely it will be parallel-efficient. The cons is that for production simulation I will have billions of files for time-accurate unsteady sims (1000-blocks X 1000... steps...) :cry:

Other possibilities that I see are:

  1. MPI IO: I never used it, I could re-learn, but due to the fact that I hope to switch soon from MPI to CAF, I am really in doubt if it is worth to invest my time on it;
  2. HDF5: I never used it, but when @francescosalvadore did show it to me it looked amazing; one issue is that some of my bosses have issues (old CentOS-based workstations) on installing exotic libraries (I had issues in installing Python 2.7 that has about 10 years...): remain in Fortran/MPI field could be better for them.

Any suggestions are much more than welcome.

My best regards.

Stefano Zaghi
@szaghi
Feb 08 2017 05:26
@zbeekman Indeed, our chimera approach (based on a sort of volume-forcing interpolation) does a good job for the conservation preservation: surely, some is lost in the chimera interpolation (and the fully conservative scheme is a chimera...), but it is not dramatic and it is really comparable with other sources of error for challenging simulations like a ship maneuvering in formed sea. The interpolation which Giacomo is referring to, happens on each single block, not in the chimera interpolation. We use WENO interpolation for high order fluxes reconstruction into the block, while the block-to-block chimera interpolation is done with other approach.
Giacomo Rossi
@giacombum
Feb 08 2017 09:16
@szaghi well, I think you already know my answer... If you have to re-learn MPI IO, well, why don't learn HDF5? And if your bosses use out of date software, well, this isn't your problem... :smile:
Stefano Zaghi
@szaghi
Feb 08 2017 09:43
@giacombum Indeed, it is one of my problem :cry:
Giacomo Rossi
@giacombum
Feb 08 2017 09:50
@szaghi You know that I know :cry: But I think that when a new CFD code is in development, the use of very out of date operating systems in which the code could run can't be a mandatory requirement. If the code is new, it can take advantages from all the new formats, paradigms and softwares that are available at the moment. And if you want to use CAF instead of MPI (very very very good choiche), don't you think you have to to face the same problems of HDF5 file format?
Stefano Zaghi
@szaghi
Feb 08 2017 14:47

@/all

I think I found a compromise: each processor/image will IO its own 1-file containing the blocks handled at that time step. The file format will be a sort of XML tagged format to allow dynamic workload and un-ordered list of block-id/refinement-level, namely to allow each processor/image to change its workload at runtime. However, this will require an extra post-processor for collect the simulation results in order to be re-used by other runs with different number of processors/images.

I think that to have only 1 file independently of the number of processors/blocks only MPI IO or HDF5 can help. I considered also direct-access file format (each processor/image access to only its own part of the file to avoid race condiction) but it sound tricky...

@giacombum I think that HDF5 and CAF can be mixed almost smoothly, maybe @zbeekman or @rouson can comment this.

Cheers

Giacomo Rossi
@giacombum
Feb 08 2017 14:52
@szaghi when I refer to the problem of the HDF5 format, I refer to the old workstation with CentOS: do you think that on that machines is simple to install CAF? And as you can see here, there are already pre-built binaries for CentOS, so... please, consider HDF5 as the future file format for your new code!
Stefano Zaghi
@szaghi
Feb 08 2017 14:55
@giacombum We have gfortran6.2 on that CentOS. I guess that with the great install script of OpenCoarrays CAF are easy to support for Roberto & Co. On the contrary, I guess that if I force them to use PETSc, HDF5, ecc.. they will leave me alone :cry: sadly, there is an inertia on our Institute to learn/adopt new standard
Izaak "Zaak" Beekman
@zbeekman
Feb 08 2017 19:23

MPI IO: I never used it, I could re-learn, but due to the fact that I hope to switch soon from MPI to CAF, I am really in doubt if it is worth to invest my time on it;
HDF5: I never used it, but when @francescosalvadore did show it to me it looked amazing; one issue is that some of my bosses have issues (old CentOS-based workstations) on installing exotic libraries (I had issues in installing Python 2.7 that has about 10 years...): remain in Fortran/MPI field could be better for them.

These are the two solutions I would recommend. Also, N.B. CAF can be safely mixed with MPI (in theory I have yet to try) at least for OpenCoarrays. The distributed shared memory OpenCoarrays implemention is built ontop of MPI, but we create private communicators, so it should be safe to use with other MPI elements. I have little experience with MPI-IO but it may give you the best fine grained control if that is critical to your application. I tried parallel HDF5 (also built on MPI, also should be safe to mix with OpenCoarrays) a few years ago and hit some painful bugs with LustreFS and HDF5. Hopefully these are all fixed now and you can use parallel HDF5 without issues. I know, for example, US3D uses HDF5 for it's file format.

Izaak "Zaak" Beekman
@zbeekman
Feb 08 2017 19:31
@szaghi another idea I like is store data very compactly with direct access IO binary files. Then store meta-data in a self describing way using JSON (JSON-Fortran) or, if you must, XML. So you can store raw blocks of variables (an ordered list of vectors of node or cell centered variables) and then either have 1 file per processor or use direct access to index into each file in the appropriate location. Obviously the connectivity info is also challenging if you're using unstructured.
@/all Travis-CI supports community contributed and supported languages. See: https://github.com/travis-ci/docs-travis-ci-com/blob/gh-pages/user/languages/community-supported-languages.md They require a team of at least three people to be maintainers and contributors with the expertise etc. It would be great if @szaghi @rouson @jacobwilliams @milancurcic @cmacmackin or anyone else here were interested in helping out with this. Setting up a sane config for fortran would hopefully make it less of a yak shave to get started using Fortran on Travis-CI. Please let me know if you are interested.
Stefano Zaghi
@szaghi
Feb 08 2017 20:29
@zbeekman Thanks for the hints on IO, I'll discuss about this with @giacombum and @francescosalvadore For the Travis support I surely offer my (poor and scattered) support, but I could not be up to the task: I have to carefully read the mission, later today I'll give you an answer. Cheers