This repository contains the source code for a Storage Module for the Apache Strap-Down Pilotage program.
Required: The ASDP_Storage_Module requires the ASDP_Core_API library to have been
installed before it is built. This installs by default in a known system location that can
be found automatically. The library source can be obtained using:
git clone https://github.com/arizonaCameraLab/ASDP_Core_API
Build: ASDP_Storage_Module uses CMake to configure the builds (though other build systems could be used). On Ubuntu Linux, this can be done as follows
sudo apt install cmake
cd; mkdir src; cd src; git clone https://github.com/arizonaCameraLab/ASDP_Storage_Module
cd; mkdir -p build/ASDP_Storage_Module; cd build/ASDP_Storage_Module
cmake ../../src/ASDP_Storage_Module
make
Packet reduction: Until the FPGA sending camera packets is updated to send multiple lines per message (or multiple messages per packet), the utility program Batch_Image_Messages must be run on each of the stream[1+].dat files stored from a run (it is not run on stream0.dat). This will group multiple image messages per packet, which is required for the Storage Module to be able to keep up with the full data rate. There is no loss when the packets are being stored to disk, but they cannot be played back quickly enough until they are grouped.
Run: The ASDP_Storage_Module can be run from the build directory using the
following command: ./ASDP_Storage_Module NICNameIn NICNameOut StorageRoot
where NICNameIn is the DNS name or dotted decimal network address of the network
interface to use to connect to a Core_Module (may be localhost to ignore) and
NicNameOut is the DNS name or dotted decimal network address of the network
interface to act as a server on (may be localhost to ignore).
The dotted decimal name can be found using the ifconfig command on Linux and ipconfig on Windows.
NICNameIn and NICNameOut cannot be the same, to avoid connecting to itself.
StorageRoot specifies the root directory where data should be stored. This will
create a configuration file named config.json in the StorageRoot directory if
there is not one already present. Note: There can be more than one StorageRoot
specified. If more than one is specified, the camera stream files for each stream
will be stored round-robin fashion in the specified StorageRoot directories. This is
useful when multiple disks are used to increase the overall write capacity or speed.
A directory named with the serial number of each camera will be created when the camera is first connected. Within that directory, a subdirectory named with the session index (starting at 1) will be created for each recording of the camera.
After each run from a camera that stores a single line per packet, the Batch_Image_Messages utility must be run on the stream1+.dat files to group the image messages into packets with multiple messages per packet. This is required to keep up with the full data rate of the camera during replay.
Analysis: If there is a config.json file in the directory with the serial number of the camera is, and if it contains the entry 'analysisPort' with a port number, then the Storage Module will send analysis packets to that port when they are available. For a givin recording session, the presence of an 'analysis' directory indicates that there is analysis data available. Each file in the analysis directory that ends with '.json' will be opened using a JSONStringReceiver and its data will be time adjusted and sent as analysis packets to the specified port. Note that messages from all files will be sent through the same JSONStringSender, so object names must be uniqu across all of them to prevent interference between the files.
Documentation: The primary documentation is available in DOxygen, and is generated by as part of the build process when DOxygen is available. On Ubuntu Linux, this can be generated as follows:
sudo apt install doxygen- When you build using CMake, it will build DOxygen by default.
- Open doc_doxygen/html/index.html with a web browser to view the documentation.
Test: CMake includes the concept of test applications. You can run the tests
by running make test in the build directory.
Architecture: The ASDP_Storage_Module.cpp contains the main program that is run to start the Storage Module. It makes using of the Storage_Module library. The Storage_Module.h file contains the class definitions for the classes in the Storage_Module library and Storage_Module.cpp has the implementations of these classes.
-
Storage_Module: The Storage_Module class is the main class for the Storage Module. It implements the client side of the Storage Module and contains one more more Storage_Module_Server objects to handle the server side of the Storage Module for each serial numbered Core Module that has been recorded or that is running live.
-
Storage_Module_Server: Implements the server side of the Storage Module. Sends Discovery packets to advertise the Storage Module and listens for incoming connection requests, with each on its own port.
Files: The StorageRoot directory will contain a directory for the serial number of Core Modules that have been connected. Each of these contains zero or more directories for each session that has been recorded. There is a file named config.json in the root directory that contains the configuration information for the Storage Module. The default file (created if there is no file) is shown below:
{
"diskBlockSize": 1024,
"highWaterMark": 515288,
"storingAtRestart": false,
"totalBufferSize": 524288
}
The util directory contains a number of utilities.
- Batch_Image_Messsages is a utility to pack the single-line-per-packet image messages from a stored .dat stream file (stream1+.dat, not stream0.dat) to have 3 messages per packet. This enables full-speed replay.
- Time_Disk_Writes is a utility to measure the time it takes to write frames to disk.
- Time_Disk_Reads is a utility to measure the time it takes to read frames from disk.