Part 2: Capture Data
In this second part, we will extend the demapper plugin module to capture both inputs and outputs of the demapper to files. The captured data can then be used for analysis, training, and testing of a neural demapper implementation. We focus specifically on capturing data for QPSK and 16-QAM modulation schemes, while allowing the default implementation to handle other modulation formats. Rather than reimplementing the QPSK and 16-QAM demapping algorithms, we leverage the existing OpenAirInterface implementations within our plugin.
The following sections detail the modifications required for data capture, including timestamping and multi-threading support, as well as instructions for accessing the collected data from within the containers.
Adding New Plugin Variant
The full code of the capture plugin is in tutorials/neural_demapper/nr_demapper_capture.c, and also included at the end of this tutorial. To compile this additional variant, we only need to add the following to the CMakeLists.txt file. The rest is handled by the already present demapper plugin. If you installed the tutorials via the provided installation scripts, this is already included.
1# capture
2set(PHY_DEMAPPER_CAPTURE_SRC
3 ${OPENAIR1_DIR}/PHY/NR_TRANSPORT/nr_ulsch_llr_computation.c
4 nr_demapper_capture.c
5)
6
7add_library(demapper_capture MODULE ${PHY_DEMAPPER_CAPTURE_SRC})
8target_link_libraries(demapper_capture PRIVATE pthread)
9
10add_dependencies(nr-softmodem demapper_capture)
11add_dependencies(nr-uesoftmodem demapper_capture)
12add_dependencies(demapper_capture generate_T)
Using New Plugin Variant
After compiling the new variant, you can use it by passing the option --loader.demapper.shlibversion _capture to the executable. Alternatively, you can add the following line to the .env file in your configuration directory:
GNB_EXTRA_OPTIONS="--loader.demapper.shlibversion _capture"
To verify that the capture variant is being used, check the gNB logs for the following message:
[LOADER] library libdemapper_capture.so successfully loaded
Access Captured Files
The module captures inputs and outputs to two files: demapper_in.txt and demapper_out.txt. To access these capture files from the container, you need to mount them to your host system. This can be done by adding volume mappings in your docker-compose.override.yaml file in the corresponding config folder:
cp config/common/docker-compose.override.yaml.template config/b200_arm64/docker-compose.override.yaml
And edit the file accordingly:
services:
oai-gnb:
volumes:
- ../../logs/demapper_in.txt:/opt/oai-gnb/demapper_in.txt
- ../../logs/demapper_out.txt:/opt/oai-gnb/demapper_out.txt
Create the files before running the container:
# Create the logs directory
mkdir -p logs
# Create the files
touch logs/demapper_in.txt
touch logs/demapper_out.txt
# Make the files writable by the container
chmod 666 logs/demapper_in.txt
chmod 666 logs/demapper_out.txt
Note that the host filepaths can be arbitrary, as long as they are accessible from the host system. The files must exist before running the container.
Capture Format
The captured data is stored in two text files: one containing the input symbols to the demapper and another containing the corresponding demapped output values. This simple text-based format allows for easy inspection and post-processing of the captured data.
The input file format is as follows:
1/*
2 * Input file format (f_in):
3 * time source resolution (sec.nanosec) - only first line
4 * timestamp (sec.nanosec)
5 * modulation scheme (string: QPSK, QAM16)
6 * nb_re (int32)
7 * real imag (for QPSK: int16 <space> int16)
8 * real imag ch_mag.r ch_mag.i (for QAM16: int16 <space> int16 <space> int16 <space> int16)̦̦̦̦̦̦
9 * ... (done nb_re times)
10 */
0.000000001 # Time source resolution (sec.nanosec)
1373853.185968662 # Timestamp (sec.nanosec)
QPSK # Modulation scheme (QPSK or QAM16)
96 # Number of symbols (resource elements)
177 -179 # Input symbols (real, imag)
-179 176
-180 -177
176 178
-177 -177
-177 177
And the output file format is as follows:
1/*
2 * Output file format (f_out):
3 * time source resolution (sec.nanosec) - only first line
4 * timestamp (sec.nanosec)
5 * modulation scheme (string: QPSK, QAM16)
6 * nb_re (int32)
7 * real imag llr.r llr.i (int16 <space> int16 <space> int16 <space> int16 )
8 * --->number of elements in each row depend on the modulation scheme: QPSK: 2 ; QAM16: 4
9 * ... (done nb_re times)
10 */
With the following sample output format:
0.000000001 # Time source resolution (sec.nanosec)
1373853.185968662 # Timestamp (sec.nanosec)
QPSK # Modulation scheme (QPSK or QAM16)
96 # Number of symbols (resource elements)
22 -23 # Output symbols (llr1, llr2)
-23 22
-23 -23
22 22
-23 -23
-23 22
An example to read the input and output files with Python is provided in the Integration of a Neural Demapper tutorial.
Add Timestamps
Including timestamps in the data capture enables precise analysis of the data stream by improving synchronization and sequence ordering of data blocks. Here are the key considerations when adding timestamps:
Resolution: Each file includes the system’s clock resolution to handle platforms with varying timing precision.
Monotonicity: Rapid captures can result in identical timestamps. To prevent this, we use CLOCK_MONOTONIC as the clock source, which guarantees strictly increasing timestamps.
Regularity: We capture timestamps at the module’s entry point directly before processing begins. This ensures the timestamp closely reflects the actual processing time, unaffected by processing duration or write operations.
Account for Multi-Threading
Since this is a multi-threaded application, we need to handle concurrent file writes carefully. Multiple threads will attempt to write data simultaneously, which could lead to corrupted output. Although the data processing is complete by the time we write to files, we still need to synchronize the actual file operations. The solution is straightforward - we use a mutex to ensure only one thread can write at a time. We initialize the mutex in our initialization functions and protect all file write operations with mutex locks:
static pthread_mutex_t capture_lock = PTHREAD_MUTEX_INITIALIZER;
// ...
void capture_function( ... )
{
pthread_mutex_lock( &capture_lock );
// write to files
pthread_mutex_unlock( &capture_lock );
}
Final Source Code
The original demapping function remains active while the capture function runs in parallel, allowing data collection without impacting the normal operation of the connection. The complete implementation of the capture plugin is shown below.
1static char *filename_in = "demapper_in.txt";
2static char *filename_out = "demapper_out.txt";
3static FILE *f_in;
4static FILE *f_out;
5static pthread_mutex_t capture_lock = PTHREAD_MUTEX_INITIALIZER;
6
7// import default implementations
8void nr_ulsch_qpsk_llr(int32_t *rxdataF_comp,
9 int16_t *ulsch_llr,
10 uint32_t nb_re,
11 uint8_t symbol);
12void nr_ulsch_16qam_llr(int32_t *rxdataF_comp, int32_t *ul_ch_mag, int16_t *ulsch_llr, uint32_t nb_re, uint8_t symbol);
13
14/* configure which clock source to use, will affect resolution */
15#define TIMESTAMP_CLOCK_SOURCE CLOCK_MONOTONIC
16
17/* time processing functions */
18void fprint_time( FILE* file, struct timespec *ts )
19{
20 fprintf( file, "%jd.%09ld\n", ts->tv_sec, ts->tv_nsec );
21 return;
22}
23
24// START marker-capture-input-format
25/*
26 * Input file format (f_in):
27 * time source resolution (sec.nanosec) - only first line
28 * timestamp (sec.nanosec)
29 * modulation scheme (string: QPSK, QAM16)
30 * nb_re (int32)
31 * real imag (for QPSK: int16 <space> int16)
32 * real imag ch_mag.r ch_mag.i (for QAM16: int16 <space> int16 <space> int16 <space> int16)̦̦̦̦̦̦
33 * ... (done nb_re times)
34 */
35// END marker-capture-input-format
36
37// START marker-capture-output-format
38/*
39 * Output file format (f_out):
40 * time source resolution (sec.nanosec) - only first line
41 * timestamp (sec.nanosec)
42 * modulation scheme (string: QPSK, QAM16)
43 * nb_re (int32)
44 * real imag llr.r llr.i (int16 <space> int16 <space> int16 <space> int16 )
45 * --->number of elements in each row depend on the modulation scheme: QPSK: 2 ; QAM16: 4
46 * ... (done nb_re times)
47 */
48// END marker-capture-output-format
49
50
51void capture_qpsk(
52 int32_t *rxdataF_comp,
53 int16_t *ulsch_llr,
54 uint32_t nb_re,
55 struct timespec *ts )
56{
57 pthread_mutex_lock( &capture_lock );
58
59 c16_t *rxF = (c16_t *)rxdataF_comp;
60
61 fprint_time( f_in, ts );
62 fprintf( f_in, "QPSK\n" );
63 fprintf( f_in, "%d\n", nb_re );
64 for(int i = 0; i < nb_re; i++ )
65 fprintf( f_in, "%hd %hd\n", rxF[i].r, rxF[i].i );
66 fflush( f_in );
67
68 fprint_time( f_out, ts );
69 fprintf( f_out, "QPSK\n" );
70 fprintf( f_out, "%d\n", nb_re );
71 for(int i = 0; i < nb_re; i++ )
72 fprintf( f_out, "%hd %hd\n", ulsch_llr[2*i+0], ulsch_llr[2*i+1] );
73 fflush( f_out );
74
75 pthread_mutex_unlock( &capture_lock );
76}
77
78/*
79 * from the reference implementation of test_llr.cpp
80 * in: rxdataF_comp[nb_re] , each element is a complex number with .r and .i components. Each component is int16_t
81 * in: ul_ch_mag[nb_re] , casted then to int16_t it becomes an array of [2*nb_re]. first component is the mag_real, second is mag_imag. both in int16_t
82 * out: ulsch_llr[4*nb_re]. each group of 4 are: .r, .i , saturating_sub(mag_real, std::abs(.r) , saturating_sub(mag_imag, std::abs(.i)
83 */
84
85void capture_qam16(
86 int32_t *rxdataF_comp,
87 int32_t *ul_ch_mag,
88 int16_t *ulsch_llr,
89 uint32_t nb_re,
90 struct timespec *ts )
91{
92 pthread_mutex_lock( &capture_lock );
93
94 c16_t *rxF = (c16_t *)rxdataF_comp;
95 int16_t *ul_ch_mag_i16 = (int16_t *)ul_ch_mag;
96
97 fprint_time( f_in, ts );
98 fprintf( f_in, "QAM16\n" );
99 fprintf( f_in, "%d\n", nb_re );
100 for(int i = 0; i < nb_re; i++ )
101 fprintf( f_in, "%hd %hd %hd %hd\n", rxF[i].r, rxF[i].i, ul_ch_mag_i16[2*i+0], ul_ch_mag_i16[2*i+1] );
102 fflush( f_in );
103
104 fprint_time( f_out, ts );
105 fprintf( f_out, "QAM16\n" );
106 fprintf( f_out, "%d\n", nb_re );
107 for(int i = 0; i < nb_re; i++ )
108 fprintf( f_out, "%hd %hd %hd %hd\n", ulsch_llr[4*i+0], ulsch_llr[4*i+1], ulsch_llr[4*i+2], ulsch_llr[4*i+3] );
109 fflush( f_out );
110
111 pthread_mutex_unlock( &capture_lock );
112}
113
114// Plugin Init / Shutdown
115
116int32_t demapper_init( void )
117{
118 // initialize capture mutex
119 pthread_mutex_init( &capture_lock, NULL );
120
121 // open capture files
122 f_in = fopen( filename_in, "w" );
123 AssertFatal( f_in != NULL, "Cannot open file %s for writing\n", filename_in );
124
125 f_out = fopen( filename_out, "w");
126 AssertFatal( f_out != NULL, "Cannot open file %s for writing\n", filename_out );
127
128 // print clock resolution
129 struct timespec ts;
130 clock_getres( TIMESTAMP_CLOCK_SOURCE, &ts );
131 fprint_time( f_in, &ts );
132 fprint_time( f_out, &ts );
133
134 return 0;
135}
136
137int32_t demapper_shutdown( void )
138{
139 fclose( f_in );
140 fclose( f_out );
141
142 pthread_mutex_destroy( &capture_lock );
143
144 return 0;
145}
146
147int demapper_compute_llr(int32_t *rxdataF_comp,
148 int32_t *ul_ch_mag,
149 int32_t *ul_ch_magb,
150 int32_t *ul_ch_magc,
151 int16_t *ulsch_llr,
152 uint32_t nb_re,
153 uint8_t symbol,
154 uint8_t mod_order) {
155 struct timespec ts;
156 clock_gettime( TIMESTAMP_CLOCK_SOURCE, &ts );
157
158 switch(mod_order){
159 case 2:
160 nr_ulsch_qpsk_llr(rxdataF_comp,
161 ulsch_llr,
162 nb_re,
163 symbol);
164 capture_qpsk(rxdataF_comp, ulsch_llr, nb_re , &ts);
165 break;
166 case 4:
167 nr_ulsch_16qam_llr(rxdataF_comp,
168 ul_ch_mag,
169 ulsch_llr,
170 nb_re,
171 symbol);
172 capture_qam16(rxdataF_comp, ul_ch_mag, ulsch_llr, nb_re, &ts);
173 break;
174
175#if 0 // disabled
176 case 6:
177 nr_ulsch_64qam_llr(rxdataF_comp,
178 ul_ch_mag,
179 ul_ch_magb,
180 ulsch_llr,
181 nb_re,
182 symbol);
183 break;
184 case 8:
185 nr_ulsch_256qam_llr(rxdataF_comp,
186 ul_ch_mag,
187 ul_ch_magb,
188 ul_ch_magc,
189 ulsch_llr,
190 nb_re,
191 symbol);
192 break;
193#endif
194
195 default:
196 AssertFatal(1==0,"capture_demapper_compute_llr: invalid/unhandled Qm value, symbol = %d, Qm = %d\n",symbol, mod_order);
197 return 0; // unhandled
198 }
199 return 1; // handled
200}