Implementation of Data Recording

This page focuses on the software architecture of AqNWB for implementing data recording and is mainly aimed at software developers. The recording system in AqNWB is built around several key concepts:

1. Efficient data recording for individual datasets via BaseRecordingData objects discussed in Recording datasets with BaseRecordingData
2. Consistent multi-dataset recording through convenience methods defined on individual RegisteredType objects (e.g., TimeSeries::writeData) discussed in TimeSeries Convenience Methods for Consistent Recording
3. Managing collections of recording objects through RecordingObjects, discussed in RecordingObjects for Managing Collections

Recording datasets with BaseRecordingData

AqNWB records datasets efficiently via BaseRecordingData objects. The main components involved in writing data to an NWB file via AqNWB are:

1. DEFINE_DATASET_FIELD Macro
   • The DEFINE_DATASET_FIELD macro not only defines methods for reading datasets for a particular neurodata_type (as described in Implementation of Data Read 📤), but also defines methods for retrieving BaseRecordingData objects that are used for recording to individual datasets. For each dataset field defined with this macro, a corresponding method is generated that returns a BaseRecordingData object configured for that specific dataset. The BaseRecordingData objects are then cached by RegisteredType as described below.
2. BaseRecordingData
   • BaseRecordingData is a class that manages the recording process for a dataset.
   • It keeps track of the current position in the dataset where data should be written next via the m_position member.
   • It provides methods for writing data blocks to the dataset, such as writeDataBlock, which can handle different data types and dimensions.
3. RegisteredType
   • RegisteredType maintains a cache of BaseRecordingData objects via the m_recordingDataCache member. This cache allows reusing the same BaseRecordingData object when it is requested multiple times, improving performance and retaining the recording position. The cache is essential for writing data to the dataset in a streaming fashion, as it ensures that each write continues from where the previous write left off. The cache also avoids the need for manually maintaining the objects and allows caching of an arbitrary number of BaseRecordingData object such that the individual neurodata_type classes do not need to worry about maintaining their recording state.
4. BaseIO
   • BaseIO and its implementations (e.g., HDF5IO) are responsible for the actual writing of data to disk. They provide methods for creating datasets (e.g., createArrayDataSet) and getting existing datasets (getDataSet), both of which return BaseRecordingData objects.
5. RecordingObjects
   • All RegisteredType objects used for recording are being registered with the BaseIO::m_recording_objects instance of RecordingObjects to facilitate management of all objects used for recording.

The DEFINE_DATASET_FIELD Macro for Recording

The DEFINE_DATASET_FIELD macro not only defines methods for reading datasets but also for recording to them. For each dataset field defined with this macro, a corresponding method is generated that returns a BaseRecordingData object configured for that specific dataset.

For example, if we have a TimeSeries class with a 'data' field defined using the DEFINE_DATASET_FIELD macro:

DEFINE_DATASET_FIELD(readData, recordData, std::any, "data", The main data)

This generates not only a readData() method for reading the dataset but also a recordData() method that returns a BaseRecordingData object configured for writing to the 'data' dataset.

The generated recordData() method:

1. Checks if a BaseRecordingData object for the dataset already exists in the cache
2. If it exists and reset is false, returns the cached object
3. If it doesn't exist or reset is true, gets a new BaseRecordingData object from the IO backend
4. Calls RegisteredType::registerRecordingObject to register the current RegisteredType object for recording with the I/O to make sure the object is being finalized at the end of the recording.
5. Caches the new object and returns it

This caching mechanism is crucial for maintaining the recording state across multiple writes to the same dataset.

BaseRecordingData for Managing Recording

The BaseRecordingData class is responsible for managing the recording process for a dataset. It keeps track of the current position in the dataset where data should be written next, ensuring that data is written efficiently, especially for streaming data where multiple writes occur over time.

Key features of BaseRecordingData include:

• Position Tracking: BaseRecordingData keeps track of the current position in the dataset via the m_position member. This is particularly important for streaming data, where data is written in chunks over time.
• Data Type Handling: BaseRecordingData can handle different data types and dimensions through its writeDataBlock methods, making it flexible for various types of data.

TimeSeries Convenience Methods for Consistent Recording

Specific types like TimeSeries provide convenience methods for writing multiple datasets in a consistent manner. This ensures that related datasets (e.g., 'data' and 'timestamps' in a TimeSeries) are written consistently and simplifies the recording process.

The TimeSeries class provides:

• An initialize method that sets up all the necessary datasets and attributes for a time series, including data, timestamps, control and all their attributes, e.g., unit
• A writeData method that writes data, timestamps, and control information in a single call, ensuring consistency between these related datasets.

These convenience methods handle the details of:

• Dataset Creation: Creating the necessary datasets if they don't exist.
• Data Alignment: Ensuring that related datasets (e.g., data and timestamps) are properly aligned.
• Position Management: Managing the current position in each dataset to ensure consistent writing.
• Error Handling: Handling errors that might occur during the writing process.

RecordingObjects for Managing Collections

RecordingObjects provides an additional convenience layer for managing collections of RegisteredType Containers used for recording. This is particularly useful when recording data to multiple related containers, such as multiple TimeSeries objects.

RecordingObjects simplifies the process of:

• Container Management: Adding and retrieving containers from the collection via addRecordingObject and getRecordingObject methods.
• Finalization of recording objects: via finalize
• Clearing of cached BaseRecordingData objects via clearRecordingDataCache;

NWB I/O convenience utilities

The src/io/nwbio_utils.hpp module provides convenience methods to help coordinating the recording process across multiple containers through specialized methods like:

• writeTimeseriesData: For writing data to a TimeSeries container
• writeElectricalSeriesData: For writing data to an ElectricalSeries container
• writeSpikeEventData: For writing data to a SpikeEventSeries container
• writeAnnotationSeriesData: For writing data to an AnnotationSeries container

Object and memory management

1. The I/O owns a std::shared_ptr<RecordingObjects> m_recording_objects smart pointer to its RecordingObjects
2. The RecordingObjects instance in turn tracks the RegisteredType objects used for recroding with the I/O object by owning the RegisteredType objects stored in std::vector<std::shared_ptr<AQNWB::NWB::RegisteredType>> m_recording_objects
3. RegisteredType objects in turn have a std::weak_ptr<IO::BaseIO> m_io weak pointer to the I/O.

Note

RegisteredType::m_io is a std::weak_ptr to the I/O to avoid circular referencing between the I/O-->RecordingObjects-->RegisteredType and back to the I/O, which would prevent correct clean-up of the reference counted smart pointers, which would lead to memory leaks. This also means, that the RegisteredType object does not own or keep the I/O alive, so we need to check first that the I/O is valid when we use it.

In this logic, the I/O essentially owns the recording process and the user in turn owns the I/O object. To ensure reliable management of the objects, the constructors of all RegisteredType classes are protected, requiring the use of the RegisteredType::create factory methods. This ensures that all instance of RegisteredType and its subclasses are created as smart std::shared_ptr pointers to facilitate reliable memory management.

Note

Registration with the RecordingObjects of the I/O occurs when a RegisteredType is being created via the static RegisteredType::create factory methods.

Further Reading

• Acquiring Data 📊 provides a step-by-step overview of the typical recording process.
• Implementation of Data Read 📤 provides a complementary overview of how data read is implemented, which involves many of the same classes, but using ReadDataWrapper instead of BaseRecordingData for accessing data.
• Implementing a Registered Type 🔧 discusess the use of RegisteredType to implement writing and reading of neurodata_types for NWB.