Implementation of Data Recording

This page focuses on the software architecture of AqNWB for implementing data recording and is mainly aimed at software developers. The recording system in AqNWB is built around several key concepts:

1. Efficient data recording for individual datasets via BaseRecordingData objects discussed in Recording datasets with BaseRecordingData
2. Consistent multi-dataset recording through convenience methods defined on individual RegisteredType objects (e.g., TimeSeries::writeData) discussed in TimeSeries Convenience Methods for Consistent Recording
3. Managing collections of recording objects through RecordingContainers, discussed in RecordingContainers for Managing Collections

Recording datasets with BaseRecordingData

AqNWB records datasets efficiently via BaseRecordingData objects. The main components involved in writing data to an NWB file via AqNWB are:

1. DEFINE_DATASET_FIELD Macro
   • The DEFINE_DATASET_FIELD macro not only defines methods for reading datasets for a particular neurodata_type (as described in Implementation of Data Read 📤), but also defines methods for retrieving BaseRecordingData objects that are used for recording to individual datasets. For each dataset field defined with this macro, a corresponding method is generated that returns a BaseRecordingData object configured for that specific dataset. The BaseRecordingData objects are then cached by RegisteredType as described below.
2. BaseRecordingData
   • BaseRecordingData is a class that manages the recording process for a dataset.
   • It keeps track of the current position in the dataset where data should be written next via the m_position member.
   • It provides methods for writing data blocks to the dataset, such as writeDataBlock, which can handle different data types and dimensions.
3. RegisteredType
   • RegisteredType maintains a cache of BaseRecordingData objects via the m_recordingDataCache member. This cache allows reusing the same BaseRecordingData object when it is requested multiple times, improving performance and retaining the recording position. The cache is essential for writing data to the dataset in a streaming fashion, as it ensures that each write continues from where the previous write left off. The cache also avoids the need for manually maintaining the objects and allows caching of an arbitrary number of BaseRecordingData object such that the individual neurodata_type classes do not need to worry about maintaining their recording state.
4. BaseIO
   • BaseIO and its implementations (e.g., HDF5IO) are responsible for the actual writing of data to disk. They provide methods for creating datasets (e.g., createArrayDataSet) and getting existing datasets (getDataSet), both of which return BaseRecordingData objects.

The DEFINE_DATASET_FIELD Macro for Recording

The DEFINE_DATASET_FIELD macro not only defines methods for reading datasets but also for recording to them. For each dataset field defined with this macro, a corresponding method is generated that returns a BaseRecordingData object configured for that specific dataset.

For example, if we have a TimeSeries class with a 'data' field defined using the DEFINE_DATASET_FIELD macro:

DEFINE_DATASET_FIELD(readData, recordData, std::any, "data", The main data)

This generates not only a readData() method for reading the dataset but also a recordData() method that returns a BaseRecordingData object configured for writing to the 'data' dataset.

The generated recordData() method:

1. Checks if a BaseRecordingData object for the dataset already exists in the cache
2. If it exists and reset is false, returns the cached object
3. If it doesn't exist or reset is true, gets a new BaseRecordingData object from the IO backend
4. Caches the new object and returns it

This caching mechanism is crucial for maintaining the recording state across multiple writes to the same dataset.

BaseRecordingData for Managing Recording

The BaseRecordingData class is responsible for managing the recording process for a dataset. It keeps track of the current position in the dataset where data should be written next, ensuring that data is written efficiently, especially for streaming data where multiple writes occur over time.

Key features of BaseRecordingData include:

• Position Tracking: BaseRecordingData keeps track of the current position in the dataset via the m_position member. This is particularly important for streaming data, where data is written in chunks over time.
• Data Type Handling: BaseRecordingData can handle different data types and dimensions through its writeDataBlock methods, making it flexible for various types of data.

TimeSeries Convenience Methods for Consistent Recording

Specific types like TimeSeries provide convenience methods for writing multiple datasets in a consistent manner. This ensures that related datasets (e.g., 'data' and 'timestamps' in a TimeSeries) are written consistently and simplifies the recording process.

The TimeSeries class provides:

• An initialize method that sets up all the necessary datasets and attributes for a time series, including data, timestamps, control and all their attributes, e.g., unit
• A writeData method that writes data, timestamps, and control information in a single call, ensuring consistency between these related datasets.

These convenience methods handle the details of:

• Dataset Creation: Creating the necessary datasets if they don't exist.
• Data Alignment: Ensuring that related datasets (e.g., data and timestamps) are properly aligned.
• Position Management: Managing the current position in each dataset to ensure consistent writing.
• Error Handling: Handling errors that might occur during the writing process.

RecordingContainers for Managing Collections

RecordingContainers provides an additional convenience layer for managing collections of RegisteredType Containers used for recording. This is particularly useful when recording data to multiple related containers, such as multiple TimeSeries objects.

RecordingContainers simplifies the process of:

• Container Management: Adding and retrieving containers from the collection via addContainer and getContainer methods.
• Coordinated Recording: Coordinating the recording process across multiple containers through specialized methods like:
  • writeTimeseriesData: For writing data to a TimeSeries container
  • writeElectricalSeriesData: For writing data to an ElectricalSeries container
  • writeSpikeEventData: For writing data to a SpikeEventSeries container
  • writeAnnotationSeriesData: For writing data to an AnnotationSeries container
• Error Handling: Handling errors that might occur during the recording process across multiple containers.

Further Reading

• Acquiring Data 📊 provides a step-by-step overview of the typical recording process.
• Implementation of Data Read 📤 provides a complementary overview of how data read is implemented, which involves many of the same classes, but using ReadDataWrapper instead of BaseRecordingData for accessing data.
• Implementing a Registered Type 🔧 discusess the use of RegisteredType to implement writing and reading of neurodata_types for NWB.