VlsWriter Class Reference

Detailed Description

VLSV file format

Vlasov simulation writes data into its own binary file format. The file format is quite flexible on datatypes. For example, in some files the data may be written in 4-byte floating point values, and 8-byte floating point values in another file(s). For this reason (and portability as well) each vlsv file contains the instructions on how to read the data.

It is expected that vlsv files can be efficiently written in parallel, for example by using MPI I/O.

1. Endianness

Endianness basically means the byte order of within longer data words, i.e. whether the most (big-endian) or least (little-endian) significant byte is written first. For example, floats and ints are typically 4-byte wide. There is no universal agreement on the order of the "sub-bytes". According to Wikipedia,

On some machines, while integers are represented in little-endian
form, floating point numbers are represented in big-endian form. 

Thus, a portable file format has to take care of endianness for integer and floating point datatypes separately.

Most modern computers (x86) use little-endian notation. As a side note, VisIt visualization tool natively reads VTK files which, in binary format, need to be written using big-endian notation.

The endianness of datatypes in a VLSV file is given in the header section. As a suggestion, these two fields should be the first two entries so that rest of the data can be read successfully. This behaviour is not required, however.

2. Header

The header portion consists of [{size}, {tag ID}, {value}] tuples. Each member of the tuple is an array, with number of elements and element byte sizes given in Table 1.

Table 1. The structure of header entries.

Array Content	Array Size	Element Byte Size	Element Datatype
Size	1	1	unsigned integer
Header ID	1	1	unsigned integer
Value	Size	1	depends on Header ID

The header should be read as follows:

1. Read one byte. This is the size fields.
2. If size is zero, stop reading header. Otherwise continue to next step.
3. Read one byte. This is the ID field.
4. Read size bytes. This is the value field.
5. Go back to step 1.

It is thus possible to read the header of a VLSV file without understanding its contents. However, in order to read the rest of the file correctly, some values given in header have to be parsed correctly. The possible header tags are given in a Table 2.

Table 2. List of header entries, and the datatypes of their value fields.

Tag Name	Value Datatype
BYTES_PER_CELL_CRD	unsigned integer
BYTES_PER_CELL_GID	unsigned integer
BYTES_PER_VARNAME_SIZE	unsigned integer
DIMENSIONS	unsigned integer
ENDIANNESS_FLOAT	unsigned integer
ENDIANNESS_INT	unsigned integer
VERSION	character array

Morale: The {size} and {tag %ID} fields are one byte wide entries.
This guarantees that the header can be read correctly whether or not
the reader's endianness agrees with the endianness of datatypes in the
file. The header then contains instructions on how to read the rest of
the file.

3. Static-Size Variable Description

A "static-size variable" here means that for each cell, such variable has the same (byte) size, in oppose to a "dynamic-size variables". A bona fide example of a static-size variable is the number density of particles, which for each cell is just a scalar value. An example of a dynamic-size variable is the velocity space grid stored in each cell. If the velocity space grid is adapted, it may have a different (byte) size for each spatial cell.

The description part consists of [{name size}, {name}, {varType}, {element size}] tuples. You can think that each member in the tuple is an array, with number of elements and element byte size given in Table 3.

Note that

• Variable descriptions are in same order as they appear in the cell data (see Section 4).
• "Variable Type" contains one of the values defined in namespace VlsVariable.
• The data for each variable is an array.
• The number of elements in the data array can be deduced from the variable's type.
• Byte size of data array elements is given in "Data Slement Size" field.

Table 3. Contents of an entry containing a description of a static-size variable stored in a VLSV file.

Array Content	Array Size	Element Byte Size	Element Datatype
Name Size	1	BYTES_PER_VARNAME_SIZE	unsigned integer
Variable Name	Name Size	1	character
Variable Type	1	1	unsigned integer
Data Element Size	1	1	unsigned integer

The variable descriptions should be read as follows:

1. Read BYTES_PER_VARNAME_SIZE bytes. This is the name size field.
2. If name size has a zero value, stop reading descriptions. Otherwise continue to next step.
3. Read name size bytes. This is the name field, which should be treated as a character array.
4. Read one byte. This is the variable type field.
5. Read one byte. This is the element size field.
6. Go back to step 1.

The variable descriptions are requested from DataReducer, via a call to DataReducer::getDescription, when a VLSV file is written.

Morale 1: Static-size variables are separated from dynamic-size
variables in order to reduce file size. Each dynamic-size variable
needs to be accompanied by its size for each cell. If the size field
was included for each static-size variable in each cell, the size of
cell data might increase by 25% in worst-case scenario. It is also, of
course, faster to read smaller files.

Morale 2: In the case of Vlasov simulations the user might want to get
the full six-dimensional distribution function for some cells. Writing
the distribution function for every cell results in HUGE output files
- 100 computation nodes having 16 GB memory each would already lead to
1.6 TB files!

After the variable descriptions have been read, the total byte size of static variable data per cell should be calculated, as the total size is needed when reading cell entries. Number of elements in a data array per variable type are given in Table 4.

Example: Variable description contained three entries: a SCALAR with element size 4, a VECTOR3 with element size 8, and a TENSOR33 with element size 4. Total size of static variable data per cell is thus 4

• 4*8 + 9*4 = 72 bytes.

Table 4. Variable types defined in namespace VlsHeader, and the number of elements in the data arrays.

Variable Type	Number of Elements
NULLVARIABLE	0
SCALAR	1
VECTOR2	2
VECTOR3	3
TENSOR22	4
TENSOR23	6
TENSOR32	6
TENSOR33	9

4. Cell coordinates and static-size data

This part of a VLSV file contains the physical coordinate values of each cell, as well as the static-size data. The contents of a cell entry depend on the dimensionality of the data, and on the definitions of static-size variables.

A cell entry consists of [{cell ID}, {cell crd}, {cell size}, {static data}] tuples. It is easiest to think of each element of the tuple as an array. The array sizes, byte sizes of each array element, and the element datatypes are given in Table 5.

Note that cell entries are in no particular order. This makes parallel writing of VLSV files convenient, as processes do not need to care about the order in which they write their local data to the file. The data per process can, however, be written one, few, or all cells at a time.

Table 5. Description of the cell coordinate entry in VLSV file. Consider an entry to consist of four arrays with numbers of elements, and element byte sizes, given here.

Array Contents	Array Size	Element Byte Size	Element Datatype
Cell ID	1	BYTES_PER_CELL_GID	unsigned integer
Cell crd	DIMENSIONS	BYTES_PER_CELL_CRD	floating point
Cell size	DIMENSIONS	BYTES_PER_CELL_CRD	floating point
Static Data	See Section 3.	See Section 3.	floating point

The cell data entries should be read as follows:

1. Read BYTES_PER_CELL_GID bytes, this gives the spatial cell global ID..
2. If each byte in ID has a value 255, i.e. all bits have unit value, stop reading. Otherwise continue to next step.
3. Read 2*DIMENSIONS*BYTES_PER_CELL_CRD bytes. These are the x_min, y_min, z_min, dx, dy, dz values for the cell (in the case of three-dimensional data).
4. Read static-size variable data. See Section 3 how to calculate the byte size.
5. Go back to step 1.

---

The documentation for this class was generated from the following file:

• doc/vlsv_format.txt