[openqa-revtui] High CPU usage of openqa-reftui after running >~20h

[cfconrad]

I just let openqa-revtui open for long time period. When I come back to my PC the CPU usage is high:

clemix@linux-7k5u:~> top -b -n 1 -H -p 11003
top - 08:54:50 up 23:42, 20 users,  load average: 3.89, 3.30, 2.87
Threads:  12 total,   1 running,  11 sleeping,   0 stopped,   0 zombie
%Cpu(s): 46.7 us,  5.0 sy,  0.0 ni, 48.3 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem : 15886.94+total,  655.859 free, 7543.238 used, 7687.844 buff/cache
MiB Swap: 15919.99+total, 15913.74+free,    6.250 used. 6626.691 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
11006 clemix    20   0 1529364  24996   6956 S 26.67 0.154  55:57.31 openqa-revtui
11007 clemix    20   0 1529364  24996   6956 S 26.67 0.154  68:35.43 openqa-revtui
11013 clemix    20   0 1529364  24996   6956 R 26.67 0.154  55:01.20 openqa-revtui
20084 clemix    20   0 1529364  24996   6956 S 26.67 0.154  50:35.34 openqa-revtui
14706 clemix    20   0 1529364  24996   6956 S 20.00 0.154  65:42.32 openqa-revtui
11003 clemix    20   0 1529364  24996   6956 S 0.000 0.154   0:00.02 openqa-revtui
11004 clemix    20   0 1529364  24996   6956 S 0.000 0.154   0:54.14 openqa-revtui
11005 clemix    20   0 1529364  24996   6956 S 0.000 0.154   0:00.03 openqa-revtui
11008 clemix    20   0 1529364  24996   6956 S 0.000 0.154   0:00.00 openqa-revtui
11009 clemix    20   0 1529364  24996   6956 S 0.000 0.154   0:00.00 openqa-revtui
11010 clemix    20   0 1529364  24996   6956 S 0.000 0.154  69:08.24 openqa-revtui
11012 clemix    20   0 1529364  24996   6956 S 0.000 0.154  65:02.26 openqa-revtui

clemix@linux-7k5u:~> top -n1
top - 08:57:39 up 23:45, 20 users,  load average: 4.38, 3.81, 3.14
Tasks: 397 total,   3 running, 394 sleeping,   0 stopped,   0 zombie
%Cpu(s): 46.5 us,  7.0 sy,  0.0 ni, 46.5 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem : 15886.94+total,  570.668 free, 7629.473 used, 7686.801 buff/cache
MiB Swap: 15919.99+total, 15913.74+free,    6.250 used. 6556.410 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                                         
11003 clemix    20   0 1529364  24732   6956 S 125.0 0.152 434:14.98 openqa-revtui                                                                                                   
 4572 clemix    20   0 24.722g 527796 273988 S 12.50 3.244 122:18.04 rocketchat-desk                                                                                                 
 7967 clemix    20   0 16.782g 104844  67292 S 12.50 0.644 227:47.92 chromium                                                                                                        
 8081 clemix    20   0 40.824g 367316 133096 S 12.50 2.258 457:33.13 chromium                                                                                                        
 2947 clemix     9 -11 3377936  18044   9848 S 6.250 0.111 152:31.71 pulseaudio                              

The output of openqa-reftui was this:

openqa Review TUI Dashboard v0.2b
q:Quit   r:Refresh   h:Hide/Show jobs   m:Toggle RabbitMQ tracker   s:Switch sorting    Arrows:Move up/down

===== openSUSE Tumbleweed AArch64 ====================

 2021-08-17-07:42:38     1876675 softfailed   https://openqa.opensuse.org/tests/1876675 | microos-Tumbleweed-MicroOS-Image-aarch64-Build20210816-microos@aarch64_lse_smp4
 2021-08-17-07:50:51     1876674 softfailed   https://openqa.opensuse.org/tests/1876674 | microos-Tumbleweed-MicroOS-Image-aarch64-Build20210816-microos-old2microosnext@aarch64_lse_smp4
 2021-08-17-09:05:53     1876673 softfailed   https://openqa.opensuse.org/tests/1876673 | microos-Tumbleweed-MicroOS-Image-aarch64-Build20210816-microos2microosnext@aarch64_lse_smp4
                         1876678 softfailed   https://openqa.opensuse.org/tests/1876678 | microos-Tumbleweed-MicroOS-Image-ContainerHost-aarch64-Build20210816-container-host@aarch64_lse_smp4
                         1876677 softfailed   https://openqa.opensuse.org/tests/1876677 | microos-Tumbleweed-MicroOS-Image-ContainerHost-aarch64-Build20210816-container-host-old2microosnext@aarch64_lse_smp4
                         1876676 softfailed   https://openqa.opensuse.org/tests/1876676 | microos-Tumbleweed-MicroOS-Image-ContainerHost-aarch64-Build20210816-container-host2microosnext@aarch64_lse_smp4
                         1876679 softfailed   https://openqa.opensuse.org/tests/1876679 | microos-Tumbleweed-MicroOS-Image-Kubic-kubeadm-aarch64-Build20210816-kubeadm@aarch64_lse_smp4
Total: 7, softfailed: 7


Error while refreshing: Get "https://openqa.opensuse.org/api/v1/jobs/1876675": read tcp [2001:16b8:5c90:c600:1265:30ff:fea6:b450]:46804->[2001:67c:2178:8::16]:443: read: permission denied

Maybe that error course the loop...

[cfconrad]

Also with these errors, I get high CPU usage:

Error while refreshing: http status code 502^Cinterrupt

Error while refreshing: http status code 503^Cinterrupt

[grisu48]

I am observing this issue also at random after some time (perhaps 1h). The program does not recover from this issue and I have not been successfully to find the reason for it.

[grisu48]

I'm observing the same issue now with openqa-mon and enabled RabbitMQ, so I guess the issue lies therein. My primary suspect is the connection drops and the library is in some kind of endless loop, trying to receive stuff from a closed socket or something like this.

[grisu48]

This is likely to be related to https://github.com/grisu48/gopenqa/issues/17

[grisu48]

https://github.com/grisu48/openqa-mon/pull/115 improves the situation, but it's not yet fully resolved.