[0pt] intermediate step: make graph-prototype pre OP-usable for FAIR/OpenDigitizer-UI

[RalphSteinhagen]

This issue is to track intermediate issues once the following is completed:

• [0pt,6pt] Clean up the prototype code for presenting to wider GR audience #45

goal: make graph-prototype usable from an OpenDigitizer-UI perspective

• SUB-EPIC: full port, block, graph, scheduler API gnuradio4#148
• we (@ivan-cukic, @wirew0rm, @RalphSteinhagen) agreed to revisit and investigate the std::graph proposal presented at the last @ CppCon'22 conference (video, slides, draft paper) to develop a clearer picture w.r.t. the design and relationship between the graph and scheduler classes.
• finalise/establish DataSet<T> interface @wirew0rm, @mormj, @RalphSteinhagen and GR architecture team
• graph <-> yaml import/export (tbd. direct use in UI (less overhead) or via yaml (decoupling) Functions for reading and writing graphs from and to YAML files gnuradio4#95
• optional: graph -> dot & Mermaid export N.B. doesn't turn out to be helpful for GitHub/issue documentation
• optional (at this stage): move OpenCMW's BasicThreadPool implementation to graph-prototype.
  • fix non-closing/already closed std::jthreads (N.B. has "finished" function seems to be buggy/is erroneously used)
  • verify compatibility/use with Emscripten/WASM
  • verify compatibility MacOS and Windows (maybe we can get some help from GR folks w.r.t. the latter)
• optional (at this stage): write BFS-based scheduler using multiple cores
• write examples for the most basic but very commonly used block functionalities (if not already done)
  • interpolator/decimator blocks #67
  • graph-prototype: handling of input & output history inside nodes #68
  • FIR & IIR filters
    • N.B. @RalphSteinhagen drives initial prototype (partially done) -> benchmark -> Matthias will see how/whether these can be optimised using SIMD
• GR4.0: FFT block implementation #69
• handling of settings tag, get/set via std::map<string, pmt::pmtv> PoCo (OpenCMW-like, @jsallay is working on a prototype for this: Struct reflection gnuradio/pmt#81 Struct refl gnuradio/pmt#84
• [6pt, 8pt] implement selector block/node (switch matrix like run-time plumbing between input->output ports) gnuradio4#69
• stream tag handling -- easy: second (potentially smaller) buffer for each port where tag is annotated which sample it is attached to.
• message passing -- easy: e.g. additional pair of in/out streaming port(s) but with multiple producer feeding one input port-> need to agree on callback API and transition between domains (e.g. CPU <-> GPU, etc.) - proposal: use dedicated explicit converter/adapter blocks as for the data streams
• simulated signal sources for general testing and unit-tests
• handling of cyclic graphs -- initially handle only one-sample delay case
  • runtime: probably only cycle detection and init-problem (i.e. generating the first zero-padded sample in the buffer where the cycle has been detected)
  • compile-time merge API: ... needs some work/input from Matthias
• [3pt, 3pt] graph/block profiling (enable/disable at compile-time) -- proposal: re-use chrome tracing format's & UI infrastructure (e.g. chrome://tracing/ and Perfetto) #71
• [0pt,9pt] porting gr-digitizer to the new API as poc for real-hw handling #72

This all in the spirit of 'eating-our-own-dog-food' with the focus that the graph-prototype can be used in the UI. This should also help to identify usability and API issues early on. N.B. the service would -- for the time being -- stick with the default GR 4.0 until the graph-prototype has been merged with GR 4.0.

Steps afterward (EPIC: #42)

• adding pybind11-based Python bindings -> needs feedback/proposal/help from the GR architecture team (@mormj et al.)
• define preferred Python work(...) API -> needs feedback/proposal/help from the GR architecture team (@mormj et al.)

[RalphSteinhagen]

The DataSet concept was presented at the last GNU Radio architecture meeting on 2023-03-24.
See slides for details (pdf with animations, pdf, odp):

It was identified that there is no 'one-size-fits-all' solution and that we shall establish a complementary composite type concept of Packet<T>->Tensor<T>->DataSet<T>, that covers ~80% of the use cases, and overlap and extend each other (via C++ concepts):

For example, if one needs a:

• vector + meta information (previously 'PDUs') -> Packet<T> or std::vector<T> + tag_t,
• tensor (e.g. matrix, volumetric, n-dim data, etc.) + meta information -> Tensor<T> (i.e. equivalent to Packet<T> and including rank+layout info), or
• numeric or measurement data (time series, images, etc) + meta information and series of tag_t -> DataSet<T> (i.e. Tensor<T> + signal names, units, timing info, (non-equidistant) axis info, ...)

The meta-information would be stored using std::map<std::string, pmt::pmtv>s. The rationale is to be both efficient and to target an 80/20 solution that is type-safe and intuitively to be used by both new users and novice developers. N.B. The implementation for the three types can be easily handled within the same block::work() function using constexpr-if (i.e. a block handling Packet<T> equally applies to Tensor<T> and DataSet<T>, a block handling Tensor<T> applies equally to any DataSet<T>).

More details including some practical data layout examples can be found in the slides attached above.

[RalphSteinhagen]

Largely finished, the remainder (notably only: 'handling of cyclic graphs') have be moved to:
fair-acc/gnuradio4#148