Integrating NWB Extensions 🧩

How to Integrate a New Namespace

Integrating a new schema namespace with AqNWB (e.g., to support an extension to NWB) involves generating the necessary specification files and ensuring that the namespace is registered with the NamespaceRegistry. This process is simplified through the use of the generate_spec_files.py script.

1. Get the schema files: Download or create the schema for the namespace in YAML format.

   Note

   If you are creating a new extension, please see the NWB Extension Tutorial for more information on how to create data schema for NWB.

2. Convert the schema files to C++: Run the resources/utils/generate_spec_files.py script on your schema files to generate the necessary C++ header files. This script processes the schema and creates the appropriate C++ header files that include the namespace definitions and registration.

   Note

   To learn more about how to use the script and its parameters, you can view the help doc by running:

   python resources/utils/generate_spec_files.py --help

3. Include the Generated Header Files: In your C++ code that uses AqNWB, include the generated header files. The header files will automatically take care of registering the namespace with the NamespaceRegistry by calling the REGISTER_NAMESPACE macro. Once the extension has been registered, NWBFile::initialize will automatically take care of caching your schema in the NWB file when creating a new file.
4. Implement appropriate RegisteredType classes: Follow the tutorial on Implementing a Registered Type 🔧 to define appropriate interfaces for the neurodata_types defined in your new namespace. Using the schematype_to_aqnwb.py Utility can also provide additional help by providing a simple utility that can automatically generate skeleton AqNWB C++ classes for neurodata_types directly from JSON/YAML schema files.

How the NamespaceRegistry Works

The NamespaceRegistry is a singleton class that manages the registration and lookup of namespace information.

Registering a Namespace

The REGISTER_NAMESPACE macro is used to register a namespace with the NamespaceRegistry. This macro is typically placed in the generated header file for the namespace schema and ensures that the namespace is registered when the header file is included. For example:

REGISTER_NAMESPACE("your_namespace", "1.0.0", specVariables);

would register the namespace your_namespace with version 1.0.0 and the specified specVariables.

Looking up Namespaces

To retrieve information about a registered namespace, we can use the getNamespaceInfo method. For example:

const NamespaceInfo* info = NamespaceRegistry::instance().getNamespaceInfo("your_namespace");
if (info) {
    // Use the namespace information
}

Alternatively, we can also use getAllNamespaces to get a reference to all registered namespaces. For example:

const auto& allNamespaces = NamespaceRegistry::instance().getAllNamespaces();
for (const auto& [name, info] : allNamespaces) {
    // Process each namespace
}

The returned info object is, in both cases, a NamespaceInfo struct with the name, version, and specVariables.

LabMetaData Extension Demo

The following example illustrated the key steps to implement and use an extension to NWB via AqNWB. The complete sources for this demo are available as part of the AqNWB GitHub repo in the demo/labmetadata_extension_demo folder.

Step 1: Get the Schema of the Extension

The demo uses the schema from ndx-labmetadata-example as an example. For convenience, we copied the relevant schema files already to the demo/labmetadata_extension_demo/spec folder.

Step 2: Convert the Schema to C++

The schema files are converted to C++ using the resources/utils/generate_spec_files.py script:

mkdir demo/labmetadata_extension_demo/src
python resources/utils/generate_spec_files.py demo/labmetadata_extension_demo/spec demo/labmetadata_extension_demo/src

This generates the following new header file in the demo/labmetadata_extension_demo/src folder:

ndx_labmetadata_example.hpp

#pragma once
 
#include <vector>
#include <string>
#include <string_view>
#include "spec/NamespaceRegistry.hpp"
 
namespace AQNWB::SPEC::NDX_LABMETADATA_EXAMPLE
{
 
const std::string  namespaceName = "ndx-labmetadata-example";
 
const std::string version = "0.1.0";
 
constexpr std::string_view ndx_labmetadata_example_extensions = R"delimiter(
{"groups":[{"neurodata_type_def":"LabMetaDataExtensionExample","neurodata_type_inc":"LabMetaData","name":"custom_lab_metadata","doc":"Example extension type for storing lab metadata","datasets":[{"name":"tissue_preparation","dtype":"text","doc":"Lab-specific description of the preparation of the tissue","quantity":"?"}]}]})delimiter";
 
constexpr std::string_view namespaces = R"delimiter(
{"namespaces":[{"author":["Oliver Ruebel"],"contact":["oruebel@lbl.gov"],"doc":" Example extension to illustrate how to extend LabMetaData for adding lab-specific metadata","name":"ndx-labmetadata-example","schema":[{"namespace":"core","neurodata_types":["LabMetaData"]},{"source":"ndx-labmetadata-example.extensions.yaml"}],"version":"0.1.0"}]})delimiter";
 
const std::vector<std::pair<std::string_view, std::string_view>>
    specVariables {{
  {"ndx-labmetadata-example.extensions", ndx_labmetadata_example_extensions},
  {"namespace", namespaces}
 
}};
 
// Register this namespace with the global registry
REGISTER_NAMESPACE(namespaceName, version, specVariables)
 
}  // namespace AQNWB::SPEC::NDX_LABMETADATA_EXAMPLE

Step 3: Create RegisteredType Classes for all neurodata_types

Next, a RegisteredType class is implemented for the LabMetaDataExtensionExample neurodata_type defined in the schema. Please see the Implementing a Registered Type 🔧 tutorial for more details.

LabMetaDataExtensionExample.hpp

#pragma once
 
#include "nwb/hdmf/base/Container.hpp"
#include "ndx_labmetadata_example.hpp"
 
class LabMetaDataExtensionExample : public AQNWB::NWB::Container
{
public:
    // Constructor with path and io inputs required by RegisteredType
    LabMetaDataExtensionExample(
        const std::string& path, 
        std::shared_ptr<AQNWB::IO::BaseIO> io);
 
    // Method for initializing and writing the data in the NWB file
    Status initialize(const std::string& tissuePreparation);
 
    // Define methods for reading custom extension fields
    DEFINE_FIELD(
        readTissuePreparation, 
        AQNWB::NWB::DatasetField, 
        std::string, 
        "tissue_preparation", 
        Lab-specific description of the preparation of the tissue)
 
    // Register the class with the type registry
    REGISTER_SUBCLASS(
        LabMetaDataExtensionExample, 
        AQNWB::SPEC::NDX_LABMETADATA_EXAMPLE::namespaceName)
 
private:
    // LabMetaData objects are stored in the NWB file at this location
    const std::string m_nwbBasePath = "/general";
};

LabMetaDataExtensionExample.cpp

#include "LabMetaDataExtensionExample.hpp"
#include "Utils.hpp"
#include <iostream>
 
using namespace AQNWB::NWB;
 
// Initialize the static registered_ member to trigger registration
REGISTER_SUBCLASS_IMPL(LabMetaDataExtensionExample)
 
 
LabMetaDataExtensionExample::LabMetaDataExtensionExample(
    const std::string& path,
    std::shared_ptr<AQNWB::IO::BaseIO> io)
    : AQNWB::NWB::Container(path, io)
{
    // Check that our path points to expected location in the NWB file
    if (path.find(m_nwbBasePath) != 0) {
        std::cerr << "LabMetaData path expected to appear in "
                  << m_nwbBasePath <<  " in the NWB file" << std::endl;
    }
}

Status LabMetaDataExtensionExample::initialize(const std::string& tissuePreparation)
{
  Status containerStatus = Container::initialize();
  auto tissuePrepPath = AQNWB::mergePaths(m_path, "tissue_preparation");
  Status tissueDataStatus = m_io->createStringDataSet(tissuePrepPath, tissuePreparation);
  return containerStatus && tissueDataStatus;
}

Note

As also discussed in more detail in the Implementing a Registered Type 🔧 tutorial, a few key points to note when implementing our derived RegisteredType class:

1. Selecting the base class:
   • At the very least our class must inherit from RegisteredType
   • When the neurodata_type is a dataset then it is often useful to inherit from Data to inherit its base initialize and read methods.
   • When a neurodata_type is a group then it is often useful to inherit from Container to inherit its base initialize and read methods (this is what we use here)
   • Always inherit from the closest parent if possible. Note: Technically we also should have here created a class for LabMetaData itself and then inherited from it. However since LabMetaData is only an empty group and does not add any fields or features, we can just use Container as our base class instead. The key purpose of the LabMetaData type is that it has a reserved space in the /general group in the NWB file.
2. Ensure the registration macros are called: These macros ensure that the class is registered with the RegisteredType type registry.
   • REGISTER_SUBCLASS must appear in the header file
   • REGISTER_SUBCLASS_IMPL must appear in the cpp file
3. Field Definitions: The DEFINE_FIELD and DEFINE_REGISTERED_FIELD macros simplify reading known fields from the file.
4. Schema Registration: The generated header file automatically registers the namespace with the NamespaceRegistry using the REGISTER_NAMESPACE macro. This ensures that as soon as we include the header in our app that the schema will be cached in the NWB file.

Step 4: Using the Extension

Finally, we can use our extension with AqNWB like any other type. The following example app illustrates how we can use our extension for writing and reading data with the new LabMetaDataExtensionExample type:

main.cpp

#include "LabMetaDataExtensionExample.hpp"
#include "nwb/NWBFile.hpp"
#include "io/hdf5/HDF5IO.hpp"
#include "Utils.hpp"
#include <iostream>
 
 
// The main routine implementing the main workflow of this demo application
int main(int argc, char* argv[])
{
    // Create the HDF5 I/O object for write
    std::string filePath = "testLabMetaDataExtensionExample.nwb";
    
    // Print progress status
    std::cout << std::endl << "Opening NWB file: " << filePath << std::endl;
 
    // Create an IO object for writing the NWB file
    auto io = AQNWB::createIO("HDF5", filePath);
    io->open(AQNWB::IO::FileMode::Overwrite);
 
    // Create the NWBFile
    auto nwbfile = AQNWB::NWB::NWBFile(io);
    nwbfile.initialize("test_identifier", "Test NWB File", "Data collection info");
 
    // Create the LabMetaDataExtensionExample object
    std::string labMetaDataPath = AQNWB::mergePaths("/general", "custom_lab_metadata");
    auto labMetaData = LabMetaDataExtensionExample(labMetaDataPath, io);
    std::cout << "Writing "<< labMetaData.getPath() << " extension data" << std::endl;
    labMetaData.initialize("Tissue preparation details");
 
    // Close the file
    io->close();
    std::cout << "Finished data write. Starting read." <<std::endl;
 
    // Create a new HDF5 I/O object for read
    auto readIO = AQNWB::createIO("HDF5", filePath);
    readIO->open(AQNWB::IO::FileMode::ReadOnly);
    
    // Read the LabMetaDataExtensionExample object from the file
    auto readLabMetaData = LabMetaDataExtensionExample(labMetaDataPath, readIO);
 
    // Read the LabMetaDataExtensionExample.readTissuePreparation field
    auto tissuePreparationWrapper = readLabMetaData.readTissuePreparation();
    std::string tissuePreparation = tissuePreparationWrapper->values().data[0];
    std::cout << "Read Tissue Preparation: " << tissuePreparation << std::endl;
 
    // Close the read file
    readIO->close();
    
    return 0;
}