Reading Remote Data ☁️

AqNWB supports reading data from remote sources using different HDF5 Virtual File Drivers (VFDs). Depending on the data source and your HDF5 installation, you can use the ROS3 VFD (for Amazon S3) or the remfile VFD (compatible with any HTTP(S) server supporting byte-range requests).

Warning

Both remote drivers require specific build-time configurations (see the Installation page):

• The ROS3 VFD must be enabled in your HDF5 installation. AqNWB checks this via the H5_HAVE_ROS3_VFD macro.
• The remfile VFD requires libcurl and must be enabled with the AQNWB_USE_REMFILE CMake option.

Reading via the ROS3 VFD

AqNWB supports reading data directly from Amazon S3 using the HDF5 ROS3 Virtual File Driver (VFD). This allows you to access NWB files stored in the cloud without having to download the entire file first.

Opening a file from S3

To open a file from S3, we create an HDF5IO object with the S3 URL and then call openS3 with the AWS region.

    std::string s3Url =
        "https://dandiarchive.s3.amazonaws.com/blobs/fec/8a6/"
        "fec8a690-2ece-4437-8877-8a002ff8bd8a";
    auto readio = std::make_shared<AQNWB::IO::HDF5::HDF5IO>(s3Url);
    Status status = readio->openS3("us-east-2");

Reading objects and data

Once the file is open, reading objects and data works exactly the same as reading from a local file. For example, we can read the NWBFile object:

    // Note, no data is actually downloaded here.
    auto nwbFile = AQNWB::NWB::NWBFile::create("/", readio);

We can also search for objects, e.g., here to find the TimeSeries objects.

    // Find all TimeSeries objects in the file
    auto timeSeriesFound = nwbFile->findOwnedTypes(
        {"core::TimeSeries"}, IO::SearchMode::CONTINUE_ON_TYPE);
    REQUIRE(timeSeriesFound.size() == 1);
    // Open the TimeSeries object
    auto timeSeries =
        AQNWB::NWB::TimeSeries::create(timeSeriesFound.begin()->first, readio);
    REQUIRE(timeSeries != nullptr);

We can then also read data subsets as usual. Because we are using the ROS3 VFD, only the metadata and the specific data chunks requested are downloaded from S3, making this approach efficient for accessing specific parts of large NWB files in the cloud.

    // Read the first 10 samples from the first channel of the TimeSeries.data
    auto readWrapper = timeSeries->readData<int16_t>();
    SizeArray start = {0, 0};
    SizeArray count = {10, 1};
    auto dataSlice = readWrapper->values(start, count);
    REQUIRE(dataSlice.data.size() == 10);
    readio->close();

Reading via the remfile VFD

As an alternative to ROS3, AqNWB can read remote files with the remfile-cpp virtual file driver, a C++ port of the Python remfile package. Instead of openS3, call openRemote:

    std::string s3Url =
        "https://dandiarchive.s3.amazonaws.com/blobs/fec/8a6/"
        "fec8a690-2ece-4437-8877-8a002ff8bd8a";
    auto readio = std::make_shared<AQNWB::IO::HDF5::HDF5IO>(s3Url);
    Status status = readio->openRemote();

Once opened, reading from the file works exactly like in the ROS3 example above. For a complete example see tests/examples/test_remfile_read_example.cpp.

Compared to ROS3, the remfile VFD:

• does not require HDF5 to be built with ROS3 support. It only requires libcurl and the light-weight remfile-cpp. remfile support is enabled with the AQNWB_USE_REMFILE CMake option (OFF by default). AqNWB uses an installed remfile-cpp package when available; otherwise, CMake downloads and builds a pinned remfile-cpp release. When fetched this way, remfile-cpp is installed into the same prefix as AqNWB.
• works with any HTTP(S) server that supports byte-range requests, not just S3, including presigned URLs
• uses adaptive read-ahead caching, which coalesces sequential reads into geometrically growing range requests and is typically faster for large sequential reads

Availability at compile time is indicated by the AQNWB_HAVE_REMFILE_VFD macro.

Further reading

• Reading Data 📤 discusses the general process of reading data from NWB files in more detail.