add support for recording the voltage beam pre-T-engine

[jaycedowell]

Have a way to record the voltage beam pre-T-engine so that a higher dynamic range can be achieved.

Note: This is being worked on in the reason branch.

[jaycedowell]

Near Term Plan:

Add a new dr_vbeam.py recorder that can be run instead of dr_tengine.py to enable recording a subset of the full voltage beam. dr_vbeam.py will write out a stream of raw "IBeam1" packets¹ containing up to 200 contiguous channels (~4.8 MHz). This recorder will be controlled through two commands:

• bnd selects the frequency range to save via a center frequency and a bandwidth and
• raw_record schedules a recording with a fixed duration.

Long Term Plan:

Merge the functionality of dr_vbeam.py into dr_tengine.py so that observers can use one or the other or both without having to go in and trade out dr_tengine.py for dr_vbeam.py and vice versa. The commands will probably stay the same since there isn't a naming conflict.

Challenges:

• There's packet loss on dr_tengine.py which is common to dr_vbeam.py. It's not clear where this comes from but it could be upstream of the T-engine machine.
• There is a constraint on the bandwidth that the raw voltage mode can record. This is partially set by the packet format (the number of channels is of type uint8_t) and partially from the constraints of wanting to be able to T-engine + raw record at the same time.
• Having a second voltage beam occasionally comes up. That may or may not work, both here and with the x-engines.

Footnotes

1. See https://github.com/ledatelescope/bifrost/blob/master/src/formats/ibeam.hpp for format details. ↩

[jaycedowell]

So far I have a dr_vbeam.py that seems to work. On November 6 I ran a 4 hour endurance test which seemed to go ok (overall packet loss was at ~2% which is comparable to what the T-engine usually sees). Today (November 8) I ran a short comparison test between the raw voltage beam and the power beam 1 (this is the same stream of data).

Plotting waterfalls:

These look similar but it's hard to tell. There's this interesting even in the voltage beam data which doesn't seem to have a correspondence in the power beam. This could be test related since I was having a hard time capturing power beam data.

Average over time and ratio:

This is a much better comparison. The structure in XX and YY match but there is a small scale offset. Could this be due to packet loss? Slightly different integration methods? NB: The x-axis labels in this last plot should be Frequency in Hz.

[jaycedowell]

If anyone wants to try to use this mode, the steps are:

1. Setup the voltage beam as you normally would for a standard T-engine run.
2. Log into lwateng, systemctl --user stop dr-tengine, and then systemctl --user start dr-vbeam.
3. Open a Python session to command dr_vbeam via mnc.mcs.Client.
4. After you are done, log back into lwateng to stop dr_vbeam and start dr_tengine.

Here's an example Python session for recording 4 MHz of bandwidth centered on 30 MHz for 60 s:

from mnc.mcs import Client
c = Client()
c.send_command('drr1', 'bnd', beam=1, central_freq=30e6, bw=4e6)
c.send_command('drr1', 'record', beam=1, start_mjd='now', start_mpm='now', duration_ms=60*1000)    

The files will be written to /data/vbeam01 on the T-engine machine.

You can also use the Python session to see if the T-engine or raw voltage beam recorder is running:

$ python3
>>> from mnc.mcs import Client
>>> c=Client()
>>> c.read_monitor_point('summary', 'drt1')
<MonitorPoint timestamp='2024-11-06 15:46:28.128470',
    value='shutdown', unit=''>
>>> c.read_monitor_point('summary', 'drr1')
<MonitorPoint timestamp='2024-11-08 21:53:06.866901', value='normal',
    unit=''>

This shows that the T-engine (drt1) was shutdown and hadn't updated its summary field since the 6th while the raw voltage beam recorder (drr1) is listed as "normal".

[jaycedowell]

There are also a couple of tools for looking at the raw voltage beam data under utils/:

• show_voltage_beam_info.py provides basic info about the contents of a file and
• plot_voltage_beam.py makes plots XX and YY waterfalls/integrated spectra.

As an example show_voltage_beam_info.py will tell you something like:

$ python3 show_voltage_beam_info.py 060622_1654555884982557643 
Setup:
  nchan: 168 = 4.020 MHz
  nbeam: 1
  chan0: 2424 = 57.996 MHz
  seq:   41417558513800 = 2024-11-08 16:54:54.617600
Frames Read: 1483720 = 62.013 s

[jaycedowell]

I took a 15 minute observation on B1919+21 yesterday:

python3 show_voltage_beam_info.py /tmp/060633_2224496425050e0f097 
Setup:
  nchan: 189 = 4.522 MHz
  nbeam: 1
  chan0: 2414 = 57.757 MHz
  seq:   41440771164240 = 2024-11-19 22:24:48.660480

Frames Read: 21581560 = 902.021 s

Looks like a pulsar:

The PSRFITS conversion software is from lwa-project/pulsar@baae102.

[jaycedowell]

For the B1919+21 test I did:

$ python3
>>> from mnc.xengine_beamformer_control import BeamPointingControl
>>> import time
>>> from mnc.mcs import Client
>>> b=BeamPointingControl(1)
>>> c=Client()
>>> c.send_command('drr1', 'bnd', beam=1, central_freq=60e6, bw=4.5e6)
>>> c.send_command('drr1', 'record', beam=1, start_mjd='now', start_mpm='now', duration_ms=900*1000)
>>> while True:
...     b.set_beam_target("PSR B1919+21")
...     time.sleep(20)
... 

[caseyjlaw]

Great progress! Thanks for documenting that.
I don't think it makes sense to add this logic to the scheduler yet, since users will need to manually change the recorder anyway. But this should be easy to automate/schedule once the features are implemented in a single recorder service.

[jaycedowell]

Part 1 is going in at #52.

[jaycedowell]

Now that #52 is in the reason branch is for developing the combined pre and post T-engine recorder. Key changes with this version:

• It's all under drt1 now.
• Pre T-engine recording is now scheduled with "raw_record".
• cancel and delete only work for post T-engine recordings.
  • That may not be 100% accurate. I think delete works for pre files but it could delete a file that is in progress.
• Pre T-engine files now end in ".raw".

[jaycedowell]

Practice session:

$ python3
>>> from mnc.mcs import Client
>>> c=Client()
>>> c.send_command('drt1', 'drx', beam=1, tuning=1, central_freq=60e6, filter=1, gain=6)
(True, {'sequence_id': '39edcf78ab6c11ef90703cecef5ef4c6', 'timestamp': 1732566678.8472424, 'status': 'success', 'response': 'success'})
>>> c.send_command('drt1', 'drx', beam=1, tuning=2, central_freq=60.1e6, filter=1, gain=6)
(True, {'sequence_id': '3b3f1b48ab6c11ef90703cecef5ef4c6', 'timestamp': 1732566681.0587673, 'status': 'success', 'response': 'success'})
>>> c.send_command('drt1', 'bnd', beam=1, central_freq=60.1e6, bw=250e3)
(True, {'sequence_id': '3ce3034cab6c11ef90703cecef5ef4c6', 'timestamp': 1732566683.8101947, 'status': 'success', 'response': 'success'})
>>> c.send_command('drt1', 'record', beam=1, start_mjd='now', start_mpm='now', duration_ms=60*1e3)
(True, {'sequence_id': '440c9a98ab6c11ef90703cecef5ef4c6', 'timestamp': 1732566695.8271933, 'status': 'success', 'response': {'filename': '/data/beam01/060639_203150827227440c9a9'}})
>>> c.send_command('drt1', 'raw_record', beam=1, start_mjd='now', start_mpm='now', duration_ms=60*1e3)
(True, {'sequence_id': '451e0688ab6c11ef90703cecef5ef4c6', 'timestamp': 1732566697.6177013, 'status': 'success', 'response': {'filename': '/data/beam01/060639_203152617735451e068.raw'}})

There are files:

$ ll /data/beam01/060639_203150827227440c9a9 /data/beam01/060639_203152617735451e068.raw
-rw-rw-r-- 1 ubuntu ubuntu  62497920 Nov 25 20:32 /data/beam01/060639_203150827227440c9a9
-rw-rw-r-- 1 ubuntu ubuntu 283390520 Nov 25 20:32 /data/beam01/060639_203152617735451e068.raw

Interesting results on time flow in the DRX file:

$ python3 ~/ovro_data_recorder/utils/show_drx_info.py /data/beam01/060639_203150827227440c9a9
Time Range:
  Start: 339583075181186816 -> 2024-11-25 20:31:50.108096
  Stop:  339583087332609792 -> 2024-11-25 20:32:52.105152
  Total frames: 15140
Frequency Range:
  Tuning 1: 1314785907 -> 60.000 MHz @ 0.250 MHz
  Tuning 2: 1316977217 -> 60.100 MHz @ 0.250 MHz
Frame Sets:
  Beam 1, tuning 1, pol. 0
  Beam 1, tuning 1, pol. 1
  Beam 1, tuning 2, pol. 0
  Beam 1, tuning 2, pol. 1
Time Flow:
  Beam 1, tuning 1, pol. 0
    Timetag step of 3211263: 17 occurances 
    Timetag step of 3211264: 222 occurances *
    Timetag step of 3211265: 17 occurances 
  Beam 1, tuning 1, pol. 1
    Timetag step of 3211263: 17 occurances 
    Timetag step of 3211264: 222 occurances *
    Timetag step of 3211265: 17 occurances 
  Beam 1, tuning 2, pol. 0
    Timetag step of 3211263: 17 occurances 
    Timetag step of 3211264: 220 occurances *
    Timetag step of 3211265: 17 occurances 
  Beam 1, tuning 2, pol. 1
    Timetag step of 3211263: 17 occurances 
    Timetag step of 3211264: 220 occurances *
    Timetag step of 3211265: 17 occurances

Raw voltage beam file seems ok though:

$ python3 ~/ovro_data_recorder/utils/show_voltage_beam_info.py /data/beam01/060639_203152617735451e068.raw
Setup:
  nchan: 11 = 0.263 MHz
  nbeam: 1
  chan0: 2507 = 59.982 MHz
  seq:   41453012144320 = 2024-11-25 20:31:51.664640
Frames Read: 1483720 = 62.013 s

Trying another DRX-only recording:

>>> c.send_command('drt1', 'record', beam=1, start_mjd='now', start_mpm='now', duration_ms=60*1e3)
(True, {'sequence_id': '324f6168ab6d11ef90703cecef5ef4c6', 'timestamp': 1732567095.563737, 'status': 'success', 'response': {'filename': '/data/beam01/060639_203830563768324f616'}})
>>> c.read_monitor_point('op-type', 'drt1')
<MonitorPoint timestamp='2024-11-25 20:30:23.061085', value='idle',
    unit=''>

It's strange that "op-type" says nothing is happening but I see recording in the log and there is a file. The file looks the same:

$ python3 ~/ovro_data_recorder/utils/show_drx_info.py /data/beam01/060639_203830563768324f616 
Time Range:
  Start: 339583153519972096 -> 2024-11-25 20:38:29.795776
  Stop:  339583165671395072 -> 2024-11-25 20:39:31.792832
  Total frames: 15140
Frequency Range:
  Tuning 1: 1314785907 -> 60.000 MHz @ 0.250 MHz
  Tuning 2: 1316977217 -> 60.100 MHz @ 0.250 MHz
Frame Sets:
  Beam 1, tuning 1, pol. 0
  Beam 1, tuning 1, pol. 1
  Beam 1, tuning 2, pol. 0
  Beam 1, tuning 2, pol. 1
Time Flow:
  Beam 1, tuning 1, pol. 0
    Timetag step of 3211263: 16 occurances 
    Timetag step of 3211264: 224 occurances *
    Timetag step of 3211265: 16 occurances 
  Beam 1, tuning 1, pol. 1
    Timetag step of 3211263: 16 occurances 
    Timetag step of 3211264: 224 occurances *
    Timetag step of 3211265: 16 occurances 
  Beam 1, tuning 2, pol. 0
    Timetag step of 3211263: 16 occurances 
    Timetag step of 3211264: 222 occurances *
    Timetag step of 3211265: 16 occurances 
  Beam 1, tuning 2, pol. 1
    Timetag step of 3211263: 16 occurances 
    Timetag step of 3211264: 222 occurances *
    Timetag step of 3211265: 16 occurances 

[jaycedowell]

FWIW the "op-type" reporting seems fixed now.

[jaycedowell]

Results for this afternoon's B1919+21 test:
DRX @ 19.6 MHz bandwidth:

Raw Voltage Beam @ 4.5 MHz bandwidth:

DRX @ 4.5 MHz of bandwidth:

[caseyjlaw]

Two PRs to support this mode with the scheduler:

• can handle recorder name with "raw" ovro-lwa/mnc_python#75
• using obs_mode VOLTRAW to record with "raw_record" command on drt1 ovro-lwa/lwa-observing#29

[jaycedowell]

I started experimenting with expanding the bandwidth that can be recorded in the raw mode. This is a (somewhat ratty) detection on B1919+21 with 19.6 MHz of bandwidth:

I'm not sure how far I can push this. I'm running up against two problems:

1. There might be a conflict for disk bandwidth when trying to run higher bandwidth raw modes at the same time as the standard DRX data. For the record this 19.6 MHz of bandwidth is about 300 MB/s.
2. The current IBeam1 format requires a lot of extra re-packetizing of the data since nchan is a unit8_t and that limits the packets to 255 channels.

Note: I was only able to get 300 MB/s with local modification's to Bifrost.

[jaycedowell]

I've switched the raw voltage beam recording from the IBeam1 packet format to RBeam1¹. The new format seems ok and I was able to run a 15 minute recording with 24 MHz of bandwidth and detect PSR B1919+21. For reference 24 MHz corresponds to ~370 MB/s going to disk.

I tried to push the bandwidth further but it looks like dr_tengine.py is maxing out at about 28 MHz for a raw voltage beam-only recording. It looks like it's being limited by the DownSelectOp block but I'm not sure why. All that block does is copy some data between two rings.

Footnotes

1. The only difference between the two is that nchan goes from uint8_t in IBeam to uint16_t in RBeam1. ↩