Pulse Data Recorder "Wow"

Overview of measurement data acquisition


Amount of data

The pulse data recorder has been designed for generating typically less than 100 MByte measurement data a day, independently from the sampling frequency and from the total number of values received from the analog/digital converter (ADC). This is achieved by pre-processing a block of sample data into a four-component value which stores the minimal and maximal values inside a sample block as well as the total value and a travel value.

For storage efficiency, the four components of a measurement value are stored as integers. In order to avoid loss of accuracy, the total value summarizes all values that belong to a sample block. Taking the known number of samples inside a sample block into account, the average can be calculated whenever it is needed with any desired accuracy.

The travel signal

Similar to the total value, the travel value summarizes the differences between adjacent measurement values inside a sample block, also taking the difference between the last value of a sample block and the first value of the next sample block into account. This value is useful as an indicator of the dynamics inside a sample block. Whenever an additional signal appears inside the noise signal of a sample block the travel value will get higher. This is true even if the signal is below the noise level or if the signal itself is noise or changes the noise characteristics. Trigger can conveniently be derived from that travel signal for limiting the amount of data to the really interesting events.

System characteristics for a "MEphisto Scope" device

For example, a "MEphisto Scope" sampling device generates a total amount of about 35 Gigabytes data a day. After pre-processing that data stream by the Pulse Data Recorder "Wow" the amount of data is reduced to about 75 Megabytes a day which is about 0,2% of the original amount of data. Nevertheless the recorded data contains the run of the signal with 10 values per second consisting of the minimal, maximal, total and travel value inside each sample block. Additionally, a full sample block is available at the beginning of each minute and further for each unusual event in the data stream which exposes highs or lows in the monitored signal and/or in the derived travel signal whenever those signals exceeded a given threshold.

Data acquisition overview

The overall data acquisition process runs according to the following layout:

Data acquisition process

A sample block of N measurement values is acquired by the sampling device at a certain sampling frequency. In a next step all data is transferred from the buffer inside the sampling device to the RAM of a PC where the Pulse Data Recorder program runs. Ideally the data transfer should take place in parallel to the sample process which would allow to work with a continuous data stream where all samples are equally spaced. However, not all sampling devices allow for this highly desired procedure but transfer data to the PC when a sample block has finished. A next sample block can start but when the transfer has finished.

As it is shown on the chart a sample block is pre-processed into the four components: minimal, maximal, total and travel value which become one measurement value in the data acquisition storage. Subsequently, all those values inside a data overview cycle are further processed into a single four-component value in the data overview storage.

Data storage

The data files generated at the described three levels: sample block, data acquisition and data overview are stored in a FTLight repository . The filenames in a FTLight repository are generated based on timestamps and the files themselves are put into a hierarchical folder structure which names are as well derived from timestamps. In addition to the ability of an automated processing of this pre-defined folder structure a FTLight repository is also very convenient for managing the over and over increasing amount of stored data with common tools that operate on file systems.

Characteristics and application

The advantage of the described two-step pre-processing procedure is - besides the already mentioned data reduction - a very fast access to an overview of all measurement data from a period of interest. Once an unusual high or low value of one of the four components has been detected the data can be further investigated by simply stepping down to the data in the acquisition storage and further to a sample block which finally will reveal the run of the signal at highest resolution. This raw data may be processed further into a spectrum view in order to see the one or multiple frequencies which possibly caused the deviation in the overview values. In addition, with a somewhat longer calculation process the scheduled sample blocks which are available for example each minute can be processed into a waterfall diagram. This kind of data representation is useful for detecting extremely weak but long lasting signals at a fix or slowly varying frequency.

Interferometry setup

The highest demand for amount of data as well as for sampling accuracy exists for interferometry setups where the signals of two or more antenna sites are combined into a single result data set. Usually sampling will take place continuously without break for a scheduled monitoring period, e.g. when the signals from a celestial object are visible to all receiving stations. During this scheduled monitoring period all data will be written to disk at high speed. The combined processing of signals, however, takes place after the monitoring period in a post-processing step where data will be exchanged between receiving stations at a speed which is usually limited by network (Internet) connectivity.

The Pulse Data Recorder "Wow" will work identically to the previously described procedure except that during a scheduled interferometry session in addition to previously described data a complete storage of all sample blocks will be done. This way the data analysis process during post-processing can benefit from all the before mentioned advantages and having nevertheless the complete set of data available for combining them into a compound result, e.g. fringes generated by earth rotation from a celestial object or multi-dimensional presentations of a celestial object in case of advanced research projects.


Created: 2007-07-16 by Eckhard Kantz