HPO 659: Mix read/write with md5 checking halts all I/O on all metalLB IP's and does not recover unless noobaa pod restart

[MonicaLemay]

Environment info

• NooBaa Version:
  [root@c83f1-app1 ~]# noobaa status
  INFO[0000] CLI version: 5.9.2
  INFO[0000] noobaa-image: noobaa/noobaa-core:nsfs_backport_5.9-20220331
  INFO[0000] operator-image: quay.io/rhceph-dev/odf4-mcg-rhel8-operator@sha256:01a31a47a43f01c333981056526317dfec70d1072dbd335c8386e0b3f63ef052
  INFO[0000] noobaa-db-image: quay.io/rhceph-dev/rhel8-postgresql-12@sha256:98990a28bec6aa05b70411ea5bd9c332939aea02d9d61eedf7422a32cfa0be54
• Platform:
  [root@c83f1-app1 ~]# oc get csv
  NAME DISPLAY VERSION REPLACES PHASE
  mcg-operator.v4.9.5 NooBaa Operator 4.9.5 mcg-operator.v4.9.4 Succeeded
  ocs-operator.v4.9.5 OpenShift Container Storage 4.9.5 ocs-operator.v4.9.4 Succeeded
  odf-operator.v4.9.5 OpenShift Data Foundation 4.9.5 odf-operator.v4.9.4 Succeeded

Actual behavior

This is not the same as issue 6930.

In this issue that I am opening, it is true that the node remained in the Ready state so I don't expect any IP failover. This defect is not about metallb IP's not failing over. In this defect, I/O was running to metallb IP 172.20.100.31 which is for node master1. On node master0 , in the CNSA scale core pod, (namespace ibm-spectrum-scale), mmshutdown was issued for just that node. The other nodes remained active and with the filesystem mounted. Master0 has metallb IP 172.20.100.30. There was no I/O going to that IP.

What was observed after mmshutdown on master0 was that all I/O going to 172.20.100.31 stopped. Because of issue 6930, there was no failover. That is fine and expected. But what is not expected is for all I/O to stop.

sh-4.4# date; mmshutdown
Tue Apr  5 17:10:28 UTC 2022
Tue Apr  5 17:10:28 UTC 2022: mmshutdown: Starting force unmount of GPFS file systems
Tue Apr  5 17:10:34 UTC 2022: mmshutdown: Shutting down GPFS daemons
Shutting down!

When mmshutdown was issued, the noobaa endpoint pods only error was Stale file handle

Logs show stale file handle

pod/noobaa-endpoint-7fdb5b75fd-t99nd/endpoint] Apr-5 17:12:26.930 [Endpoint/14] [ERROR] core.endpoint.s3.s3_rest:: S3 ERROR <?xml version="1.0" encoding="UTF-8"?><Error><Code>InternalError</Code><Message>We encountered an internal error. Please try again.</Message><Resource>/s5001b85</Resource><RequestId>l1mefzt6-3wj6yz-8x</RequestId></Error> PUT /s5001b85 {"host":"172.20.100.30","accept-encoding":"identity","user-agent":"aws-cli/2.3.2 Python/3.8.8 Linux/4.18.0-240.el8.x86_64 exe/x86_64.rhel.8 prompt/off command/s3.mb","x-amz-date":"20220405T171225Z","x-amz-content-sha256":"61d056dc66f1882c0f4053be381523a7a28d384abde04fcf5b0021c716bb0ea1","authorization":"AWS4-HMAC-SHA256 Credential=QzhyXj9wVDH9DvnK97L9/20220405/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=6d9fa5c22501bfed4f312ac47621b6cec691bf1cf8f719e8250fcdc0f61522f1","content-length":"154"} Error: Stale file handle
[pod/noobaa-endpoint-7fdb5b75fd-t99nd/endpoint] Apr-5 17:12:27.589 [Endpoint/14]    [L0] core.sdk.bucketspace_nb:: could not create underlying directory - nsfs, deleting bucket [Error: Stale file handle] { code: 'Unknown system error -116' }
[pod/noobaa-endpoint-7fdb5b75fd-t99nd/endpoint] Apr-5 17:12:27.793 [Endpoint/14] [ERROR] core.endpoint.s3.s3_rest:: S3 ERROR <?xml version="1.0" encoding="UTF-8"?><Error><Code>InternalError</Code><Message>We encountered an internal error. Please try again.</Message><Resource>/s5001b85</Resource><RequestId>l1meg1if-7apxvy-1bas</RequestId></Error> PUT /s5001b85 {"host":"172.20.100.30","accept-encoding":"identity","user-agent":"aws-cli/2.3.2 Python/3.8.8 Linux/4.18.0-240.el8.x86_64 exe/x86_64.rhel.8 prompt/off command/s3.mb","x-amz-date":"20220405T171227Z","x-amz-content-sha256":"61d056dc66f1882c0f4053be381523a7a28d384abde04fcf5b0021c716bb0ea1","authorization":"AWS4-HMAC-SHA256 Credential=QzhyXj9wVDH9DvnK97L9/20220405/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=bb34c5aa7ed665ae73a2d172ab33d4056c2611ad2c09331ece80089cff46df05","content-length":"154"} Error: Stale file handle
[root@c83f1-app1 ~

This error is a bit odd because it is on the endpoint pod that was for master0. Master0's metallb IP was 172.20.100.30. Cosbench workload was only set up for 172.20.100.31.

An additional observation is that s3 command for list will work but not for write.

[root@c83f1-dan4 RW_workloads]# date; s5001_2_31 ls
Tue Apr  5 18:33:40 EDT 2022
2022-04-05 18:33:43 s5001b100
2022-04-05 18:33:43 s5001b63
2022-04-05 18:33:43 s5001b62
2022-04-05 18:33:43 s5001b61


[root@c83f1-dan4 RW_workloads]# date;  s5001_2_31 cp alias_commands s3://s5001b1
Tue Apr  5 18:35:36 EDT 2022
^C^Cfatal error:
^CError in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "concurrent/futures/thread.py", line 40, in _python_exit
  File "threading.py", line 1011, in join
  File "threading.py", line 1027, in _wait_for_tstate_lock
KeyboardInterrupt
[root@c83f1-dan4 RW_workloads]# date
Tue Apr  5 18:36:43 EDT 2022
[root@c83f1-dan4 RW_workloads]#

All future PUT to 172.20.100.31 and 172.20.100.32, get a timeout (if I don't CTL-C) and the endpoint pods record a "Error: Semaphore Timeout"
From 31 and 32, we can do GET and we can read from the Noobaa database. If we rsh into the endpoint pods for the IP's of 172.20100.31 and 172.20.100.32, we see that Spectrum Scale is still mounted in the correct place and we can write to it manually with touch file. So, this tells us that the IP's 31 and 32 are still alive and that the noobaa db is still online. It also tells us that the Spectrum Scale filesystem is still mounted and writable. The timeout on the subsequent PUT tell us that it makes a connection request but never gets a response.

The endpoint pods never restarted and they sill have their labels.

Also, In the scale core pod we run mmhealth node show -N all and we see that everything is HEALTHY, except of course on the one node that we did mmshutdown.

  sh-4.4# mmhealth node show

Node name:      master1-daemon
Node status:    HEALTHY
Status Change:  1 day ago

Component      Status        Status Change     Reasons & Notices
----------------------------------------------------------------
GPFS           HEALTHY       1 day ago         -
GUI            HEALTHY       1 day ago         -
NETWORK        HEALTHY       9 days ago        -
FILESYSTEM     HEALTHY       9 days ago        -
NOOBAA         HEALTHY       26 min. ago       -
PERFMON        HEALTHY       1 day ago         -
THRESHOLD      HEALTHY       9 days ago        -
sh-4.4# set -o vi
sh-4.4# mmhealth node show -N all

Node name:      master0-daemon
Node status:    FAILED
Status Change:  36 min. ago

Component      Status        Status Change     Reasons & Notices
--------------------------------------------------------------------------------
GPFS           FAILED        36 min. ago       gpfs_down, quorum_down
NETWORK        HEALTHY       1 day ago         -
FILESYSTEM     DEPEND        36 min. ago       unmounted_fs_check(remote-sample)
PERFMON        HEALTHY       1 day ago         -
THRESHOLD      HEALTHY       1 day ago         -

Node name:      master1-daemon
Node status:    HEALTHY
Status Change:  1 day ago

Component      Status        Status Change     Reasons & Notices
----------------------------------------------------------------
GPFS           HEALTHY       1 day ago         -
GUI            HEALTHY       1 day ago         -
NETWORK        HEALTHY       9 days ago        -
FILESYSTEM     HEALTHY       9 days ago        -
NOOBAA         HEALTHY       27 min. ago       -
PERFMON        HEALTHY       1 day ago         -
THRESHOLD      HEALTHY       9 days ago        -

Node name:      master2-daemon
Node status:    HEALTHY
Status Change:  1 day ago

Component       Status        Status Change     Reasons & Notices
-----------------------------------------------------------------
CALLHOME        HEALTHY       9 days ago        -
GPFS            HEALTHY       1 day ago         -
NETWORK         HEALTHY       9 days ago        -
FILESYSTEM      HEALTHY       9 days ago        -
GUI             HEALTHY       3 days ago        -
HEALTHCHECK     HEALTHY       9 days ago        -
PERFMON         HEALTHY       9 days ago        -
THRESHOLD       HEALTHY       9 days ago        -

Something is obviously hung in the PUT connection but logs and noobaa health don't point to anything.
When we issue mmstartup the PUT's still fail. The only way to recover is to delete the Noobaa endpoint pods and have new ones generated again.

I have been able to recreate this very easily so if it is required I can set this up on my test stand

Expected behavior

1.When doing mmshutdown on one node, it should not impact cluster wide I/O capability. It should not be an outage. If indeed an outage is expected, then mmstartup should recover I/O capability.

Steps to reproduce

1. Start a cosbench run. I can provide the xml if needed. Once I/O is running, issue mmshutdown from within one CNSA Ibm Spectrum Scale core pod.

More information - Screenshots / Logs / Other output

Must gather and noobaa diagnose in https://ibm.ent.box.com/folder/145794528783?s=uueh7fp424vxs2bt4ndrnvh7uusgu6tocd 


This issue started as HPO https://github.ibm.com/IBMSpectrumScale/hpo-core/issues/659 Screeners determined that it was with Nooobaa. I have also slacked the CNSA team for input but have not heard back.

[MonicaLemay]

I need to amend the discovery of this defect. It has nothing to do with CNSA mmshutdown on a scale core pod. The trigger for this is to have s3 service with MD5 enabled and cosbench workload with mixed read and write while doing hashCheck=true.

[root@c83f1-dan4 ~]# cat /root/cosbench/workloads/RW_workloads/rand_read_write_delete_GB_1hour/s5001_GB_random_size_num_of_objects.xml
<?xml version="1.0" encoding="UTF-8" ?>
<workload name="GBrandomsizenumobj" description="GBrandom read and write workload">
  <storage type="s3" config="accesskey=QzhyXj9wVDH9DvnK97L9;secretkey=xTwEhfGB/wRXZbk7Ymsq2fYwl52AriraQPVzrZKF;path_style_access=true;endpoint=http://metallb2:80"/>

  <workflow>

   <workstage name="prepare_rGB">
      <work name="prepare_rGB" workers="256" interval="10" rampup="0" type="prepare" ratio="100" config="cprefix=s5001b;oprefix=s5001o;osuffix=_rGB;containers=r(1,4);objects=r(1,20);sizes=r(2,3)GB;hashCheck=true"/>
    </workstage>

    <workstage name="rGB_write_read_delete">
      <work name="rGB_write_read_delete" workers="256" interval="10" rampup="0" runtime="300">
        <operation type="write" ratio="50" config="cprefix=s5001b;oprefix=s5001o;osuffix=_rGB;containers=r(5,6);objects=r(1,20);sizes=u(2,3)GB;hashCheck=true"/>
        <operation type="read" ratio="50" config="cprefix=s5001b;oprefix=s5001o;osuffix=_rGB;containers=r(1,2);objects=r(1,20);hashCheck=true"/>
      </work>


  </workflow>
</workload>

I will see that the cosbench workload fails. When I fails, a manual s3 cp times out, while s3 ls still works.

[MonicaLemay]

Can we make this a sev 1 defect? I think this is a critical problem for us. Could this be a noobaa db deadlock. I don't know the commands to check for noobaa db deadlock

[romayalon]

@MonicaLemay
Are you able to curl to 172.20.100.31/172.20.100.32?

[MonicaLemay]

Curl works when the condition exists. The IP's are reachable.

[root@c83f1-dan4 ~]# curl 172.20.100.31
<?xml version="1.0" encoding="UTF-8"?><Error><Code>AccessDenied</Code><Message>Access Denied</Message><Resource>/</Resource><RequestId>l1w6sj05-996mt5-tt8</RequestId></Error>[root@c83f1-dan4 ~]# [

Also s3 ls will work and will list the buckets. The s3 cp times out so no PUT works.

[romayalon]

@MonicaLemay

Few observations:

1. The endpoint pods that are attached to master0 are: noobaa-endpoint-7fdb5b75fd-t99nd, noobaa-endpoint-7fdb5b75fd-5k4c2
2. Stale file handle error - I searched for it in all the endpoint pods logs and could only find it in these: noobaa-endpoint-7fdb5b75fd-t99nd, noobaa-endpoint-7fdb5b75fd-5k4c2 - which are the pods that are attached to master0.
   according to google:

A filehandle becomes stale whenever the file or directory referenced by the handle is removed by another host, while your client still holds an active reference to the object

I feel this stale file handle error is expected, but maybe i'm missing something?

3. I searched for alias_commands (the object that you tried to cp) in all endpoint pods logs and couldn't find it - I suspect NooBaa never got this request and that's why the response never return and you get a timeout - NSFS debug level would help with checking it.
4. The fact that ls is working doesn't really say anything because NooBaa throws Slow Down and it's not a timeout.
5. cosbench failing on PUT requests - on the functional endpoint pods I see that NooBaa returns Slow down - which makes sense to me because now there are only 4 endpoint pods, each has 3Gi memory - and 0.75 Gi is the limit for NSFS buffers. I see that you are uploading large files, can you try it with smaller files and check if you still get this Slow Down error?

[MonicaLemay]

@romayalon Thanks Romy for the comment. I'm not as worried about Stale File handle and Cosbench slow down. What concerns me the most is after I hit this condition, I kill cosbench workload. After that I cannot do s3 cp . I can wait 12 hours and still cannot do an s3 cp file. I have to restart the noobaa pods. After I restart noobaa pods, I can do s3 cp

[romayalon]

@MonicaLemay So can you please check again the cp command connection details (specifically the endpoint address) and check if the requests were received in the endpoint pods? A Higher debug level is needed here probably.

[MonicaLemay]

Did some follow-up investigation asked for by Romy. In the process, I have refined the trigger.
So, in summary 5 things:

1. Cosbench will give a java error when the cosbench xml file has hashcheck=true AND file sizes are 3gig or bigger. In some ways, I don't consider this to be a noobaa problem . This is probably something with cosbench. When the java error occurs the file does not get sent. This can't be noobaa's fault. I don't think we need to persue the java errors from cosbench. When files are 3 gig and bigger but hashCheck is not set everything works fine.

2. Cosbench termination of workload or lettting it complete, w/out hashcheck=true AND 3 gig or bigger is OK. s3 cp file works
 and there are no java errors in the cosbench logs.


3. Cosbench w/ hashcheck=true AND files 1 - 2 Gig will causes s3 cp file to timeout. This is the error that this defect needs to persue.
4 . I checked to make sure that Cosbench is not holding the port open or blocked. I stopped cosbench on all drivers and still could not s3 cp file
.
5. If I remove the read from the Cosbench workload xml file, then s3 cp file will work. It appears to NOT be a function of file size but doing both reads (GET) and writes (PUT) with hashCheck=true .

The best xml file to reproduce this quickly is:

[root@c83f1-dan4 RW_workloads]# cat /root/cosbench/workloads/RW_workloads/rand_read_write_delete_GB_1hour/s5002_GB_random_size_num_of_objects.xml
<?xml version="1.0" encoding="UTF-8" ?>
<workload name="GBrandomsizenumobj" description="GBrandom read and write workload">
  <storage type="s3" config="accesskey=qiIImaITeJCdkdABWAsk;secretkey=c80yUC5QfG9sxK99XQI70zXjpin9cqIeSR54r47c;path_style_access=true;endpoint=http://metallb2:80"/>

  <workflow>

   <workstage name="prepare_rGB">
      <work name="prepare_rGB" workers="256" interval="10" rampup="0" type="prepare" ratio="100" config="cprefix=s5002b;oprefix=s5002o;osuffix=_rGB;containers=r(1,2);objects=r(1,3);sizes=r(1,2)GB;hashCheck=true"/>
    </workstage>

    <workstage name="rGB_write_read_delete">
      <work name="rGB_write_read_delete" workers="256" interval="10" rampup="0" runtime="300">
        <operation type="write" ratio="20" config="cprefix=s5002b;oprefix=s5002o;osuffix=_rGB;containers=r(5,6);objects=r(1,20);sizes=r(1,2)GB;hashCheck=true"/>
        <operation type="read" ratio="80" config="cprefix=s5002b;oprefix=s5002o;osuffix=_rGB;containers=r(1,2);objects=r(1,3);hashCheck=true"/>
      </work>
    </workstage>


  </workflow>
</workload>

One test was done with log level nsfs. Noobaa diagnose and must-gather are colllected and uploaded to our box note.

root@c83f1-dan4 RW_workloads]# date ; sh /root/cosbench/cli.sh submit /root/cosbench/workloads/RW_workloads/rand_read_write_delete_GB_1hour/s5002_GB_random_size_num_of_objects.xml
Wed Apr 13 15:34:25 EDT 2022
Accepted with ID: w967
[root@c83f1-dan4 RW_workloads]#




The hang occurred around
Wed Apr 13 15:36:10 EDT 2022
Grabbed logs

Now try a new file name


[root@c83f1-dan4 RW_workloads]# date; s5001_2_31 cp good.xml s3://s5001b1
Wed Apr 13 15:38:33 EDT 2022

NOW LOG LEVEL ALL



[root@c83f1-dan4 RW_workloads]# date ; sh /root/cosbench/cli.sh submit /root/cosbench/workloads/RW_workloads/rand_read_write_delete_GB_1hour/s5002_GB_random_size_num_of_objects.xml
Wed Apr 13 15:55:08 EDT 2022
Accepted with ID: w969

Now try to s3 cp a file

[root@c83f1-dan4 RW_workloads]# date; s5001_2_31 cp nohup.out s3://s5001b1
Wed Apr 13 15:56:12 EDT 2022
upload: ./nohup.out to s3://s5001b1/nohup.out
[root@c83f1-dan4 RW_workloads]# date; s5001_2_31 cp rand_read.xml s3://s5001b1
Wed Apr 13 15:57:02 EDT 2022

In all cases oc logs does not show any errors.

[guymguym]

@MonicaLemay This is a great elimination!

I probably missed it mentioned above somewhere - how much time, or which actions, it takes to recover from this situation?

Few suggestions:

• Use aws --debug s3 cp/ls ...- the debug flag will produce very verbose logs from the client side, including tracing of all http requests and responses it sends and receives.

• Check the metrics from openshift monitoring dashboards (see this blog for guidance) - I would take a look at the basic cpu/mem/network stats of the pods to see if we can spot anything. It would be relevant to see which pods max-out their CPU requests (endpoint/database perhaps) vs. which pods become idle and just wait for some event, and how things change overtime until the issue is hit, and then recovered.

• Few more things you can try once this issue occurs: 1. GET an object 2. Try to access other buckets 3. Read & Write files directly to the filesystem (on that same bucket dir and best from the cluster mount point).

[MonicaLemay]

@guymguym @romy

1. When the condition occurs it is only on metallb IP 172.20.100.31 and 172.20.100.32. The metallb IP address 172.20.100.33 does not exhibiti the problem. The reason for this is b/c my cosbench set up only has 2 drivers. I don't have 3 drivers. Driver 1 points to 172.20.100.31 and driver 2 points to 172.20.100.32. It appears it only occurrs on IP's that had simultaneous read/write with hashCheck=true.

• A file is attached for aws debug of cp file is attached. It was going to account s5002 bucket4, using 172.20.100.32.

• When the condition occurs other accounts and buckets experience the same problem.

-When the condtion occurs s3 mb and rb works. Only PUT and GET fail to the 2 IP's that were doing cosbench simultaneous read/write with hashCheck=true.

• I wrote 2 scripts. One does curl PUT to the s5001 account and the other does curl GET to the s5001 account. It uses the aws access keys and generates signature. The GET and PUT work until this GIT issue we are talking about occurs. When it occurs, the behaviour is just like cp file. Both curl GET and curl PUT hang to the 2 addresses mentioned in this post.

• Last week, I had investigated CPU and memory to see if it was overly used.

aws-debug-32.docx

[MonicaLemay]

I can provide IP addresses for storage cluster, openshift cluster and appllication nodes if you noobaa team would like to debug on my cluster.

[guymguym]

@MonicaLemay Thanks!

• You didn't mention how long it takes to recover or if you had to do something to recover from this issue.
• About the dashboard screenshot - is it filtering only that single endpoint pod? can you see all the namespace pods in the same graph to compare?
• The cpu graph of the endpoint pod shows a burst of 3 minutes - is this the time of the entire cosbench run? or just the time of the issue?