The data storage uses a memory buffer and additionally a file buffer that can be enabled optionally.

The description applies to all types of data stores that transfer data to external systems and where temporary accessibility and available bandwidth issues may occur, such as:

  • ibaHD timebased/eventbased/lengthbased

  • DB/Cloud timebased

  • Kafka cluster timebased

  • MQTT timebased

  • MindSphere timebased

  • InfluxDB timebased

Data to be sent to the target system always passes through the internal ibaPDA memory buffer. If the connection to the target system exists, the data is sent there from the memory buffer immediately. If the connection is lost, or the data cannot be sent out fast enough, the data remain in the memory buffer. The memory buffer is located in the RAM of the ibaPDA computer and is therefore limited and volatile. If, for example, the acquisition is restarted, the buffered data is lost. If the memory buffer grows beyond the configured size during ongoing acquisition, the oldest values are deleted and thus lost.

To improve this, a file buffer can additionally be enabled, which can buffer much larger amounts of data. The data is stored in files in a directory in a local drive of the ibaPDA server. When the file buffer is enabled, data is transferred from the overflowing memory buffer to the file buffer. If the acquisition is finished or restarted (e.g. by applying a modified IO configuration), data that may be in the memory buffer at this time is also transferred to the file buffer.

After reconnecting to the target system, the oldest data is always transferred first. Newer values are added to the buffer in the meantime. If there is still buffered data in the file buffer when the acquisition is started, it is handled and processed in the same way. The data is saved in the format that was configured in the data store at the time of buffering and it is also sent in this format when the connection is established again.

You configure the buffering in the Buffer node of the respective data store. The figure below shows the buffer configuration using the example timebased data store.

Memory buffer

The memory buffer is always enabled. You cannot deactivate it because data to be transmitted always passes through the buffer before being forwarded to the target system.

Maximum size

Enter here the maximum total size for items buffered in memory. If the maximum size is exceeded, there are 2 options:

  • When file buffering is disabled, the oldest item in memory is deleted (and is lost forever).

  • When file buffering is enabled, the oldest part of the buffer memory is moved to a buffer file.

Periodically persist memory buffer every … s

You can only enable this option if file buffering is enabled. If the option is enabled, the entire memory buffer is periodically swapped to a buffer file.

Enter a duration after which the memory buffer is periodically stored. It must be between 10 s and 600 s.

With this option you can ensure that as little data as possible is lost in case of a system failure.

Current memory configuration

Display of the approximate time period that can be temporarily stored in the memory buffer with the configured settings. Specified in d.hh:mm:ss.
File buffer

Use file buffering

By default, the file buffer is not used. Here you can enable file buffering.

Current file configuration

Display of the approximate time period that can be temporarily stored in the file buffer with the configured settings. Specified in d.hh:mm:ss.

File storage path

In the File storage path field, you can select a location for the files. You can enter the directory directly into the text field, or select it via the browse button <…>. The configured file directory must be located on a local hard disk of the ibaPDA server computer.

You can use the same file directory for several data stores because the buffer files of a data store have a unique name. Files from different data stores can thus be distinguished by their name.

Maximum size

You can configure the maximum total size of the buffer files of a data store. The buffer files themselves have the file extension .buf, the index file for managing the buffer files has the extension .info. The maximum size is the total size of all these files. If the maximum buffer size is exceeded, the oldest buffer file is deleted.
Other buffer settings

Maximum time

Stored data older than the maximum time is not transferred to the target system. Files older than the maximum time can be deleted. You can enter a value between 1 and 1000 hours.
Memory buffer diagnostics/File buffer diagnostics

Last item removed

Indicates when the last item was taken from this part of the buffer.

Fill level

The fill level indicates what percentage of the buffer size is currently filled with buffered data.

Unprocessed level

Items transferred to the target system are not deleted immediately in the file buffer. Only when a buffer file is completely read, it is deleted. Therefore, it is possible that only a part of a buffer file contains data that has not yet been transferred. The fill level refers to the existing buffer files, while the "unprocessed level" indicates the percentage of data in the file buffer that has not yet been transferred.