[BUG] The problem of health check in gRPC service-to-service invocation

[KenyeeC]

What happened:

I have two Kratos services that call each other through gRPC. My gRPC connection is established via a domain name, and there are multiple IPs behind this domain name, which is used for load balancing. When one of the IPs associated with this domain name goes offline, the health check of the gRPC client in Kratos will cause the other service to be inaccessible for a short period of time, but in fact, it is still accessible.

What you expected to happen:

Even if a certain IP behind the domain name goes offline, the connection should not be made unavailable. Specifically, since in this situation, the connection is usually made through only one domain name, is it possible that in the gRPC load balancing of Kratos, when it is determined that there is only one node, the load balancing can be skipped (in this way, the health check will also be avoided)

How to reproduce it (as minimally and precisely as possible):

Start two Kratos services, communicate through gRPC, with the endpoint being a domain name. This domain name points to multiple IPs. Then, keep sending requests to Kratos. At this time, randomly take one of the IPs corresponding to the domain name offline, and then the issue can be reproduced.

Anything else we need to know?:

This is caused by the health check in the grpc load balancing. Can the load balancing not take effect when there is only one node, or can users be allowed to choose the strategy by themselves?

Environment:

• Kratos version (use kratos -v): 2.x
• Go version (use go version): 1.23.x
• OS (e.g: cat /etc/os-release): mac
• Others:

[dosubot]

Hello @KenyeeC! I'm Dosu, your friendly bot here to help you solve bugs, answer questions, and assist you in becoming a contributor. I'm here while you wait for a human maintainer. Let's tackle your issue together!

To address the issue of gRPC service inaccessibility due to health checks when an IP goes offline, you can configure Kratos to avoid health checks or customize the load balancing strategy. Here are two approaches:

1. Disable Health Checks: You can disable health checks by using the WithHealthCheck option and setting it to false. This will prevent health checks from being performed, which can be useful when there is only one node.

   grpc.Dial(
       context.Background(),
       grpc.WithEndpoint("your-endpoint"),
       grpc.WithHealthCheck(false),
   )

2. Custom Load Balancing Strategy: You can allow users to choose their own strategy by using the WithNodeFilter option to apply custom node filters. This can help in selecting nodes based on custom criteria, potentially avoiding nodes that are offline.

   grpc.Dial(
       context.Background(),
       grpc.WithEndpoint("your-endpoint"),
       grpc.WithNodeFilter(customNodeFilter),
   )

Replace "your-endpoint" with the actual endpoint and customNodeFilter with your custom node filter logic. These configurations can help maintain service accessibility even when some IPs go offline ^[1].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[KenyeeC]

Hello @KenyeeC! I'm Dosu, your friendly bot here to help you solve bugs, answer questions, and assist you in becoming a contributor. I'm here while you wait for a human maintainer. Let's tackle your issue together!

To address the issue of gRPC service inaccessibility due to health checks when an IP goes offline, you can configure Kratos to avoid health checks or customize the load balancing strategy. Here are two approaches:

1. Disable Health Checks: You can disable health checks by using the WithHealthCheck option and setting it to false. This will prevent health checks from being performed, which can be useful when there is only one node.
   grpc.Dial(
   context.Background(),
   grpc.WithEndpoint("your-endpoint"),
   grpc.WithHealthCheck(false),
   )
2. Custom Load Balancing Strategy: You can allow users to choose their own strategy by using the WithNodeFilter option to apply custom node filters. This can help in selecting nodes based on custom criteria, potentially avoiding nodes that are offline.
   grpc.Dial(
   context.Background(),
   grpc.WithEndpoint("your-endpoint"),
   grpc.WithNodeFilter(customNodeFilter),
   )

Replace "your-endpoint" with the actual endpoint and customNodeFilter with your custom node filter logic. These configurations can help maintain service accessibility even when some IPs go offline [1].

To continue talking to Dosu, mention @dosu.

Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other

1. Turning off the health check can solve the problem, but I only knew this was feasible after looking at the source code. What I mean is that it should be turned off before performing load balancing to prevent users from encountering this bug without being aware of it.

2. "WithNodeFilter" cannot solve the problem because when the health check fails, there is no node available for selection here, so it cannot be resolved either.

[miyabyte]

The machines behind the DNS system may adjust their numbers based on automatic scaling strategies. Therefore, at the framework level, it is not recommended to directly request the IP after obtaining it.

[KenyeeC]

The machines behind the DNS system may adjust their numbers based on automatic scaling strategies. Therefore, at the framework level, it is not recommended to directly request the IP after obtaining it.

We did not request the IP directly after obtaining it. The IP is actually leveraged by the underlying gRPC health check mechanism. Kratos' load balancer utilizes this health check by default for service discovery and traffic distribution."