Hi,
I'd like to write interleaved data to a TDMS file, and at the same time (in a separate loop) read that data back. This is similar to what the example "Concurrent Access to TDMS File" shows, however the main difference is that the data is interleaved. Data is written a single value at a time, to multiple channels across multiple groups. This is similar to how a CSV file would be written, with a column per channel and data written to the file a single row at a time.
The problem with this approach is that using the standard TDMS functions causes the TDMS file becomes very large, and the index file is a similar size to the TDMS file. I.e. the file is fragmented. I'm aware of this issue from previous projects, and it is discussed here in more depth: https://forums.ni.com/t5/forums/v3_1/forumtopicpage/board-id/170/thread-id/811943/page/1 . Despite the one header only feature being enabled by default, it only works properly in a very narrow set of circumstances. The solution I've used in past projects is to use the advanced TDMS functions to create the file, write the header data information once, then write the interleaved data using the advanced synchronous write function. This has worked very well, resulting is very efficiently packed files that are fast to write to, can use different data types across multiple channels and multiple groups.
However, when trying to perform concurrent TDMS access, the read loop must also use the advanced TDMS functions. The advanced TDMS synchronous read function returns the data, but it appears to be raw/interleaved. I.e. the read function is essentially useless, unless you are prepared to re-implement the standard TDMS read code in a subVI to decode the file yourself... More information about that is in this post: https://forums.ni.com/t5/LabVIEW/How-to-use-TDMS-Advanced-functions-properly/m-p/2053854#M669302 .
To play with these conecpts, I've modifed the concurrent TDMS example (See attached and below):
![Test - Concurrent Access to TDMS File.png]( "Test - Concurrent Access to TDMS File.png")
Observations:
- Advanced TDMS write can only be read by a standard TDMS read after the file has been closed, not concurrently.
- The Advanced TDMS read always reads in data raw (underlying file format), regardless of being concurrent or read after the file was closed.
- The standard TDMS write can be read by the standard TDMS read, but:
- The write must be set to interleaved, otherwise data read back is still interleaved (channels mixed together).
- With a write mode of decimated:
- The read loop only returns correct data if no channel properties were written (no header), and the last run of the VI was with interleaved mode or decimated that returned correct values. Perhaps this is a bug in the TDMS functions?
- The read after the file was closed always returns interleaved data.
- Standard Interleaved writes always work, however the file is fragmented with very large index file.
- For example, 10,000 values per channel in the attached VI returns a 1944kB file that defragments to 313kB...
I can't see any way arround this problem, other than using the standard interleaved writes and defragmenting the file after closing it. I'm only writing a data point every 1 or more seconds, and to approximately 100 to 1000 channels spread over 5 to 10 groups. So write speed isn't an issue, and the temporarily fragmented TDMS file shouldn't cause problems. It's just not ideal.
Does anyone have any better suggestions?
Thoughts/ideas regarding TDMS:
- Is there a way to deinterleave the data as it is written, producing a file with sequential channel values?
- See the double array shift registers in the write loop of the attached VI: If I was doing this in memory, I'd preallocate an array per channel, then replace each value as it comes in. Is there a TDMS equivelent? If not, is it a feature that could be added?
- For a TDMS file, I imagine that would require either preallocating the file on disk (TDMS reserve functions), or by having the TDMS file add fixed length blocks (where channel length is known) as the file expands. I'm aware that TDMS can support this, and does that when writing large waveforms to disk.
- TDMS as a format:
- One advantage I see for using a TDMS file is that it can directly replace common formats like CSV, with the benefit of more efficient binary storage (in some cases the underlying TDMS file structure can be made to look like a binary version of a CSV, when using the advanced TDMS functions). This is overall good.
- Compared to other formats like HDF5, TDMS lacks flexibility in it's structure. I.e. everything is a 1D channel of data, with limited hierarcy. The tradeoff that I can see for this reduced structure is that you get the benefit of efficient high speed streaming, and minimal file fragmentation when doing so. I.e. TDMS works really well as a living file with large volumes of data being constantly written to it over time.
I'm a little disappointed that this use case of TDMS (mimicking a CSV file with columns of data written row by row) cannot work with concurrent reads while keeping the file structure clean.
Gabriel
