0. You need a working MPI environment, preferably MPI-2, but MPI-1.X will be fine, too. 1. you need to compile with -DWITH_MPI set and the appropriate libraries for MPI (normally, using mpicc instead of gcc/icc/cc will do the right thing) 2. you need to execute with mpiexec -n NCPU ./AHFstep AHFstep.input if you are not using MPI-2, use mpirun instead with the correct -machinefile parameter. -> note that there is no redirect of stdin to AHFstep anymore! okay, that's simple, but here's what else to consider... 1. domain decomposition ----------------------- - the domain is decomposed according to a Peano-Hilbert curve utilizing a grid (2^LOADBALANCE_DOMAIN_LEVEL)^3 where LOADBALANCE_DOMAIN_LEVEL is a parameter to be set in src/param.h - this parameter should be set in a way to ensure that the physical spacing of the grid (i.e., BoxSizeInMpch/2^LOADBALANCE_DOMAIN_LEVEL) is >= Rvir where Rvir is the virial radius of the largest object expected to be found in the box; or in other words, the buffer zone about each domain segment is one cell! If you do not comply with this rule you may end up with twisted halos...but if you set this parameter too small, each CPU will receive all particles and you gain nothing! Set it wisely to have fun! 2. analysis ----------- - each CPU writes its own AHF output files, but you can just simply use 'cat' to merge them all together - at the moment the newly written libio can handle AMIGA binaries, all sorts of GADGET files and TIPSY binaries. But there is a tiny difference: you do not need to tell AMIGA upon compile time the format of the input file! You do that in the first line of AHFstep.input: if you simply give the name of the file in that line libio expects to find an AMIGA binary, if that filename is followed by a 60 you will read a single GADGET file, and if that number is 61 you will read multiple GADGET files, and if followed by 90 TIPSY format is assumed. Note: It is encouraged to specify 0 in the case you want to read AMIGA binaries, even though this is implicitly assumed when not setting anything. - the number of CPU's reading and analysing does not need to be the same, however, N_reading <= N_analysing! 3. example ---------- Just as an example of the above: Assume your (multiple GADGET) data resides in '/work/me/data' and consists of 32 GADGET files per snapshot -- so there should be e.g. snapshot_012.{0-31} in the data directory. Further let us assume that you are in the directory '/work/me/analysis' and the properly compiled AHFstep is in '/home/me/bin'. You could now start the P-AHF analysis with: mpiexec -n 4 /home/me/bin/AHFstep analysis-snapshot_012.input where analysis-snapshot_012.input reads: /work/me/data/snapshot_012. 61 2 TEST-snapshot_12 256 5.0 5.0 0 0 1 1 So you are giving here in the first line the 'filename' of the datafiles up to the last common character. P-AHF will then detect the ending numbers correctly then, as you told it with the '61' that it is a multiple GADGET data file. And the following '2' tells AHF to use 2 CPU's for reading. The rest of the parameters are exactly as you know them. If you didn't change anything in param.h, the particle splitting will be done on a 64^3 grid. Depending on your dataset that might be appropriate, or not. In the course of running, P-AHF will create a couple of logfiles, namely 'TEST-snapshot_12.00.log' up to 'TEST-snapshot_12.03.log'; note that '.XX.log' gets appended automatically and the number refers to the rank of the MPI task writing this file. In the -.00.log file you should see a bunch of information concerning the input file, the other tasks currently don't bother with reading data. Also note that the MPI version of AHF will append to the end of the log file, you can search forward to the line saying '--- NEW ENTRY STARTS HERE ---' to get the different entries. concluding remarks ------------------ All the credits for MPI version of AHF go to Steffen Knollmann! He worked very hard during the last couple of months to get this version up and running...