Skip to main content

Status Assessment and Troubleshooting

Healthy UE attach and data with OAI RAN​

  1. On LITEON RU Once the gNodeB connects to RU, the DuConnected field should change to Ready as shown below when we run: brc extract logs <liteon-element-name>
Sync State  : SYNCHRONIZED
RF State    : Ready
DPD         : Ready
DuConnected : Ready
  1. On Benetel Once the gNodeB connects to RU, the antena transmission can be checked using: TXMeanPower and if all 4 in the 4RX4TX configuration are up, one should see:
TX 1 Mean Power: 10.7029 dBm
TX 2 Mean Power: 4.37732 dBm
TX 3 Mean Power: 9.40766 dBm
TX 4 Mean Power: 10.2364 dBm

In addition, KPI measurements can be checked using: kpi.sh

Good transmission means the LATE and EARLY packet counters are zeros as shown in the example:

SAMPLE_TIME          | RX_TOTAL             | RX_ON_TIME           | RX_EARLY             | RX_LATE              | RX_ON_TIME_C         | RX_EARLY_C           | RX_LATE_C            | TX_TOTAL
10:32:25.066903      | 92959                | 83396                | -0                   | -0                   | 9604                 | -0                   | -0                   | 32400
10:32:26.065888      | 92672                | 83032                | -0                   | -0                   | 9592                 | -0                   | -0                   | 32400
10:32:27.065919      | 92883                | 83272                | -0                   | -0                   | 9596                 | -0                   | -0                   | 32400
10:32:28.067132      | 92930                | 83136                | -0                   | -0                   | 9608                 | -0                   | -0                   | 32400
10:32:29.070562      | 92837                | 83384                | -0                   | -0                   | 9620                 | -0                   | -0                   | 32505

If the RX_ON_TIME counters is 0, with RX_EARLY , RX_LATE, populated as shown below:

SAMPLE_TIME          | RX_TOTAL             | RX_ON_TIME           | RX_EARLY             | RX_LATE              | RX_ON_TIME_C         | RX_EARLY_C           | RX_LATE_C            | TX_TOTAL
 11:07:48.053547      | 92978                | -0                   | 41868                | 41600                | -0                   | 4408                 | 4424                 | -0
 11:07:49.048914      | 92817                | -0                   | 41340                | 41600                | -0                   | 4392                 | 4384                 | -0
 11:07:50.046409      | 92844                | -0                   | 41720                | 41600                | -0                   | 4400                 | 4392                 | -0
 11:07:51.048411      | 92781                | -0                   | 41518                | 41600                | -0                   | 4400                 | 4416                 | -0

Resolve by checking the ptp4l and phc2sys services in the DU server; If any of them is in a failed state, restart:

sudo systemctl restart ptp4l.service
sudo systemctl restart phc2sys.service
  1. OAI gNB/CU-DU

On successful DU RU connection, you should see the following tx rx counters in the gnb log:

pps  127488 kbps 4214777][Total Msgs_Rcvd 14284785]
[NR_PHY]   [o_du0][pusch0 3225585 prach0  345632]
[NR_PHY]   [o_du0][pusch1 3225592 prach1  345600]
[NR_PHY]   [o_du0][pusch2 3225590 prach2  345600]
[NR_PHY]   [o_du0][pusch3 3225586 prach3  345600]
[NR_MAC]   Frame.Slot 256.0

UE RNTI 8f6d CU-UE-1 ID 1 in-sync PH 47 dB PCMAX 24 dBm, average RSRP -71 (32 meas)
     CQI 15, RI 2, PMI (0,1)
UE 8f6d: dlsch_rounds 1278/692/678/665, dlsch_errors 597, pucch0_DTX 2632, BLER 0.54206 MCS (1) 3
UE 8f6d: dlsch_rounds 1118/117/74, dlsch_errors 3, ulsch_DTX 0, BLER 0.00000 MCS (1) 6
UE 8f6d: MAC: TX =     175737 RX =     165266 bytes
 LCID 1: TX       907 RX     3878 bytes
 LCID 2: TX       126 RX       28 bytes
 LCID 4: TX      4371 RX     1578 bytes
  • RSRP close to -70 dBm (typical indoor)
  • Low uplink BLER (0.00000) – good uplink channel
  • Downlink BLER ~0.53 – moderate downlink quality
  • MAC layer TX/RX activity confirms user-plane traffic

Troubleshooting​

  • If below logs is observed, with zero tx and rx counters, the devmem commands that set the transmission mode and the PRACH compression may not be well set. Set extras.t9s.io/liteon-ru-reboot to true and re-deploy your network so that the RU is rebooted and devmem settings are applied. If the above devmem settings are not made, pusch and prach counters will have zeros as below;
[NR_PHY]   [o-du 0][rx       0 pps       0 kbps       0][tx 1020100 pps  127488 kbps 4717971][Total Msgs_Rcvd 0]
[NR_PHY]   [o_du0][pusch0       0 prach0       0]
[NR_PHY]   [o_du0][pusch1       0 prach1       0]
[NR_PHY]   [o_du0][pusch2       0 prach2       0]
[NR_PHY]   [o_du0][pusch3       0 prach3       0]
  • If above problem persists and/or errors regarding received time not matching the expected time:
[PHY]   Received Time doesn't correspond to the time we think it is (slot mismatch, received 480.5, expected 475.8)
[PHY]   Received Time doesn't correspond to the time we think it is (frame mismatch, 480.5 , expected 475.5)
[PHY]   Jump in frame counter last_frame 480 => 519, slot 19
[PHY]   Received Time doesn't correspond to the time we think it is (slot mismatch, received 519.19, expected 480.12)

Restart the ptp4l and phc2sys services on the gNB/CU-DU node with:

sudo systemctl restart ptp4l.service
sudo systemctl restart phc2sys.service

After restarting these services, remove and re-install the network with brc remove network <network.yaml> and brc install network <network.yaml>.

Checking Status of CU-DU nodes​

Here are the commands for CU-DU (gNB) nodes using O-RAN 7.2 to be run after each reboot.

  • Check the IP and interfaces: ip address show

  • Initialize the DPDK (after every reboot):

    sudo dpdk-setup # produces the PCI addresses created

    you may view the contents of the script via: sudo vim /bin/dpdk-setup

  • Check the virtual MAC addresses

    ip link show # under dpdk0, vf0 and vf1

  • Synchronization checks:

    sudo systemctl status ptp4l.service

    should show low numerical values for the diff -- Ideally -10 < x < +10

    sudo systemctl status phc2sys.service

    Should show low numerical values for the diff -- Ideally -10 < x < +10

  • If restarting the services is needed, you may run:

    sudo systemctl restart ptp4l.service

    sudo systemctl restart phc2sys.service

  • Check the journal logs for the PTP or PHC2sys sudo journalctl --unit ptp4l --unit phc2sys --follow

Last updated on by Risma