SIONlib: Scalable I/O library for parallel access to task-local files

Version (1.1p9) available (see also Release notes)

SIONlib is a small library for writing and reading binary data to/from from several thousands of processors into one or a small number of physical files. The library provides global open and close functions for accessing one file for writing and reading in parallel. Only these function are collective, writing and reading to/from this file can be done asynchronously. sionlib provides a sion file handle and also an file pointer (FILE *) which allows to use the standard C file I/O operations (POSIX). This file pointer points to a unique position in the file.

SIONlib provides a simplified file handling for a parallel program that formerly had to read or write binary data in parallel into separate files (task-local files). After opening the file with sionlib, the I/O can be done with the standard C-I/O functionality (fwrite, fread) without changing the existing program. Instead of maintaining thousands of separate files for storing the data, there is only one large file containing all the data.

SIONlib requires information on the BLOCKSIZE of the underlying filesystem, i.e., the BLOCKSIZE is the smallest part of a file which can be locked. For instance, IBM GPFS has blocksizes of one ore two Megabytes.

Given that each processor writes its data to same file, sionlib assigns different regions (e.g., data blocks) of the file to a processor. Hence, for each processor, sionlib needs the estimated data size which would be written to the file (chunk size). sionlib can also handle extensions (additional chunks), if the end of a data block is reached.

Both, the estimated chunk size and filesystem's blocksize are used to calculate the individual start position in the file for each processor. While a certain block of the file is locked only by one processor, efficient parallel writing and reading is feasible.

Further informations:
Sion API
Sion File Format
Examples
Utilities
Further Documentation
Installation, Debugging, Error Messages
Release Notes & Copyright

Contact: sionlib.jsc@kfa-juelich.de
WWW: http://www.fz-juelich.de/jsc/sionlib