NullPointerException error reports

[sherif-fanous]

I've been getting these error reports and corresponding Prometheus alerts since setting up Piraeus Operator + Linstor on K8s a couple of days ago.

╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ Id                    ┊ Datetime            ┊ Node             ┊ Exception                                                                          ┊
╞═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ 6636AF5E-FAD29-000003 ┊ 2024-05-04 22:01:08 ┊ S|talos-hybrid-1 ┊ NullPointerException: Cannot invoke "java.lang.CharSequence.length()" because "... ┊
┊ 6636AF5E-FAD29-000004 ┊ 2024-05-04 22:01:17 ┊ S|talos-hybrid-1 ┊ NullPointerException: Cannot invoke "java.lang.CharSequence.length()" because "... ┊
┊ 6636AF5E-FAD29-000005 ┊ 2024-05-04 22:01:47 ┊ S|talos-hybrid-1 ┊ NullPointerException: Cannot invoke "java.lang.CharSequence.length()" because "... ┊
┊ 6636AF5E-FAD29-000006 ┊ 2024-05-04 22:02:48 ┊ S|talos-hybrid-1 ┊ NullPointerException: Cannot invoke "java.lang.CharSequence.length()" because "... ┊
┊ 6636AF5E-FAD29-000007 ┊ 2024-05-04 22:04:47 ┊ S|talos-hybrid-1 ┊ NullPointerException: Cannot invoke "java.lang.CharSequence.length()" because "... ┊
┊ 6636AF5E-FAD29-000008 ┊ 2024-05-04 22:09:48 ┊ S|talos-hybrid-1 ┊ NullPointerException: Cannot invoke "java.lang.CharSequence.length()" because "... ┊
┊ 6636AF5E-FAD29-000009 ┊ 2024-05-04 22:19:47 ┊ S|talos-hybrid-1 ┊ NullPointerException: Cannot invoke "java.lang.CharSequence.length()" because "... ┊
┊ 6636AF5E-FAD29-000010 ┊ 2024-05-04 22:49:47 ┊ S|talos-hybrid-1 ┊ NullPointerException                                                               ┊
┊ 6636C2CC-FAD29-000000 ┊ 2024-05-04 23:34:42 ┊ S|talos-hybrid-1 ┊ NullPointerException: Cannot invoke "java.lang.CharSequence.length()" because "... ┊
┊ 6636C2CC-FAD29-000001 ┊ 2024-05-04 23:34:54 ┊ S|talos-hybrid-1 ┊ NullPointerException: Cannot invoke "java.lang.CharSequence.length()" because "... ┊
┊ 6636C2CC-FAD29-000002 ┊ 2024-05-04 23:35:18 ┊ S|talos-hybrid-1 ┊ NullPointerException: Cannot invoke "java.lang.CharSequence.length()" because "... ┊
┊ 6636C2CC-FAD29-000003 ┊ 2024-05-04 23:35:20 ┊ S|talos-hybrid-1 ┊ NullPointerException: Cannot invoke "java.lang.CharSequence.length()" because "... ┊
┊ 6636C2CC-FAD29-000004 ┊ 2024-05-04 23:36:18 ┊ S|talos-hybrid-1 ┊ NullPointerException: Cannot invoke "java.lang.CharSequence.length()" because "... ┊
┊ 6636C2CC-FAD29-000005 ┊ 2024-05-04 23:38:20 ┊ S|talos-hybrid-1 ┊ NullPointerException: Cannot invoke "java.lang.CharSequence.length()" because "... ┊
┊ 6636C2CC-FAD29-000006 ┊ 2024-05-04 23:43:18 ┊ S|talos-hybrid-1 ┊ NullPointerException: Cannot invoke "java.lang.CharSequence.length()" because "... ┊
┊ 6636C2CC-FAD29-000007 ┊ 2024-05-04 23:53:19 ┊ S|talos-hybrid-1 ┊ NullPointerException: Cannot invoke "java.lang.CharSequence.length()" because "... ┊
┊ 6636C2CC-FAD29-000008 ┊ 2024-05-05 00:23:18 ┊ S|talos-hybrid-1 ┊ NullPointerException: Cannot invoke "java.lang.CharSequence.length()" because "... ┊
┊ 6636C2CC-FAD29-000009 ┊ 2024-05-05 01:23:18 ┊ S|talos-hybrid-1 ┊ NullPointerException                                                               ┊
┊ 6636C2CC-FAD29-000010 ┊ 2024-05-05 05:23:18 ┊ S|talos-hybrid-1 ┊ NullPointerException                                                               ┊
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

There are 2 different error reports as follows

ERROR REPORT 6636C2CC-FAD29-000008

============================================================

Application:                        LINBIT? LINSTOR
Module:                             Satellite
Version:                            1.27.1
Build ID:                           c6f8ceed9d50da2c4d37ae8ce20d09daf3046464
Build time:                         2024-04-25T11:12:13+00:00
Error time:                         2024-05-05 00:23:18
Node:                               talos-hybrid-1
Thread:                             DeviceManager

============================================================

Reported error:
===============

Category:                           RuntimeException
Class name:                         NullPointerException
Class canonical name:               java.lang.NullPointerException
Generated at:                       Method 'getTextLength', Source file 'Matcher.java', Line #1769

Error message:                      Cannot invoke "java.lang.CharSequence.length()" because "this.text" is null

Error context:
        An error occurred while processing resource 'Node: 'talos-hybrid-1', Rsc: 'pvc-07b05ee1-a95a-42d2-8c7c-252c4ee10988''
Call backtrace:

    Method                                   Native Class:Line number
    getTextLength                            N      java.util.regex.Matcher:1769
    reset                                    N      java.util.regex.Matcher:415
    <init>                                   N      java.util.regex.Matcher:252
    matcher                                  N      java.util.regex.Pattern:1134
    shellQuote                               N      com.linbit.utils.StringUtils:93
    accept                                   N      java.util.stream.ReferencePipeline$3$1:197
    forEachRemaining                         N      java.util.Spliterators$ArraySpliterator:992
    copyInto                                 N      java.util.stream.AbstractPipeline:509
    wrapAndCopyInto                          N      java.util.stream.AbstractPipeline:499
    evaluate                                 N      java.util.stream.AbstractPipeline:575
    evaluateToArrayNode                      N      java.util.stream.AbstractPipeline:260
    toArray                                  N      java.util.stream.ReferencePipeline:616
    toArray                                  N      java.util.stream.ReferencePipeline:622
    joinShellQuote                           N      com.linbit.utils.StringUtils:117
    exec                                     N      com.linbit.extproc.ExtCmd:111
    exec                                     N      com.linbit.extproc.ExtCmd:89
    getCurrentGID                            N      com.linbit.linstor.layer.drbd.utils.DrbdAdm:371
    allVlmsMetaDataNew                       N      com.linbit.linstor.layer.drbd.DrbdLayer:1708
    condInitialOrSkipSync                    N      com.linbit.linstor.layer.drbd.DrbdLayer:1614
    adjustDrbd                               N      com.linbit.linstor.layer.drbd.DrbdLayer:713
    processResource                          N      com.linbit.linstor.layer.drbd.DrbdLayer:250
    lambda$processResource$4                 N      com.linbit.linstor.core.devmgr.DeviceHandlerImpl:908
    processGeneric                           N      com.linbit.linstor.core.devmgr.DeviceHandlerImpl:949
    processResource                          N      com.linbit.linstor.core.devmgr.DeviceHandlerImpl:904
    processResources                         N      com.linbit.linstor.core.devmgr.DeviceHandlerImpl:370
    dispatchResources                        N      com.linbit.linstor.core.devmgr.DeviceHandlerImpl:217
    dispatchResources                        N      com.linbit.linstor.core.devmgr.DeviceManagerImpl:331
    phaseDispatchDeviceHandlers              N      com.linbit.linstor.core.devmgr.DeviceManagerImpl:1204
    devMgrLoop                               N      com.linbit.linstor.core.devmgr.DeviceManagerImpl:778
    run                                      N      com.linbit.linstor.core.devmgr.DeviceManagerImpl:672
    run                                      N      java.lang.Thread:840


END OF ERROR REPORT.

And

ERROR REPORT 6636C2CC-FAD29-000010

============================================================

Application:                        LINBIT? LINSTOR
Module:                             Satellite
Version:                            1.27.1
Build ID:                           c6f8ceed9d50da2c4d37ae8ce20d09daf3046464
Build time:                         2024-04-25T11:12:13+00:00
Error time:                         2024-05-05 05:23:18
Node:                               talos-hybrid-1
Thread:                             DeviceManager

============================================================

Reported error:
===============

Category:                           RuntimeException
Class name:                         NullPointerException
Class canonical name:               java.lang.NullPointerException
Generated at:                       <UNKNOWN>


Error context:
        An error occurred while processing resource 'Node: 'talos-hybrid-1', Rsc: 'pvc-07b05ee1-a95a-42d2-8c7c-252c4ee10988''
Call backtrace:

    Method                                   Native Class:Line number


END OF ERROR REPORT.

Both error reports pertain to PVC pvc-07b05ee1-a95a-42d2-8c7c-252c4ee10988 but this PVC doesn't even exist in my cluster. It might have been a test PVC I created earlier and then deleted.

Thanks

[ghernadi]

In case this issue persists after restarting this one satellite and possibly also the controller, I'd like to ask for TRACE logs as next step (using linstor c set-log-level TRACE --global) and send me the logs (or an SOS report using linstor sos dl) via email for example. (Feel free to disable TRACE logging with linstor c set-log-level INFO --global after you have collected the logs)