Jaeger Deployment issue with Elastisearch Backend

[agarwal62]

Hi everyone, I'm facing an error while trying to deploy Jaeger with Elasticsearch storage on top of an EKS Cluster. I tried to deploy it with the help of Helm charts as well as operators, but my Jaeger collector and query pods are not coming up and are in a CrashLoopBackOff state. The error logs I get is:

{"level":"info","ts":1727413786.1901612,"caller":"flags/admin.go:122","msg":"Admin server started","http.host-port":"[::]:16687","health-status":"unavailable"}
{"level":"fatal","ts":1727413791.2333581,"caller":"query/main.go:93","msg":"Failed to init storage factory","error":"failed to create primary Elasticsearch client: health check timeout: Head "
http://quickstart-es-http:9200\":
EOF: no Elasticsearch node available","stacktrace":"main.main.func1\n\tgithub.com/jaegertracing/jaeger/cmd/query/main.go:93\ngithub.com/spf13/cobra.(*Command).execute\n\tgithub.com/spf13/[email protected]/command.go:983\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\tgithub.com/spf13/[email protected]/command.go:1115\ngithub.com/spf13/cobra.(*Command).Execute\n\tgithub.com/spf13/[email protected]/command.go:1039\nmain.main\n\tgithub.com/jaegertracing/jaeger/cmd/query/main.go:159\nruntime.main\n\truntime/proc.go:267"}

I checked that my Elasticsearch is running and I am able to access it, but I don't know why the Jaeger components are not able to access it.

[dosubot[bot]]

Hey there, @agarwal62! I'm here to help you with your Jaeger deployment issue. I can assist you in solving bugs, answering questions, and becoming a contributor. Let's work on resolving this together.

I found some similar discussions that might help you resolve the issue:

1. helmchart - jaeger-collector & jaeger-query can't reach elasticsearch (solved) ^[1].
2. Jeager All in One with Opensearch Backend (solved) ^[2].

To resolve the CrashLoopBackOff state of Jaeger collector and query pods when deploying Jaeger with Elasticsearch storage on an EKS Cluster, given the error "failed to create primary Elasticsearch client: health check timeout", you need to ensure that the Jaeger components can properly connect to the Elasticsearch cluster. Here are the steps to troubleshoot and resolve the issue:

1. Verify Elasticsearch Endpoint: Ensure that the ES_SERVER_URLS environment variable is correctly set with the proper Elasticsearch endpoint URL.

2. Network Connectivity: Check if the Jaeger pods can reach the Elasticsearch service. You can use tools like curl or telnet from within the Jaeger pods to test connectivity to the Elasticsearch endpoint.

3. Elasticsearch Health: Ensure that the Elasticsearch cluster is healthy and accessible. You can check the health status by accessing the _cluster/health endpoint of Elasticsearch.

4. Configuration Parameters: Ensure that all required configuration parameters for Elasticsearch are correctly set in the Jaeger deployment. For example, you can configure the Jaeger collector and query service with Elasticsearch as follows:

# Jaeger Collector
apiVersion: apps/v1
kind: Deployment
metadata:
  name: jaeger-collector
spec:
  template:
    spec:
      containers:
      - name: jaeger-collector
        image: jaegertracing/jaeger-collector:{{< currentVersion >}}
        env:
        - name: SPAN_STORAGE_TYPE
          value: elasticsearch
        - name: ES_SERVER_URLS
          value: http://<ES_SERVER_IP>:<ES_SERVER_PORT>
        ports:
        - containerPort: 14269
        - containerPort: 14250
        - containerPort: 14268

# Jaeger Query Service
apiVersion: apps/v1
kind: Deployment
metadata:
  name: jaeger-query
spec:
  template:
    spec:
      containers:
      - name: jaeger-query
        image: jaegertracing/jaeger-query:{{< currentVersion >}}
        env:
        - name: SPAN_STORAGE_TYPE
          value: elasticsearch
        - name: ES_SERVER_URLS
          value: http://<ES_SERVER_IP>:<ES_SERVER_PORT>
        ports:
        - containerPort: 16685
        - containerPort: 16686
        - containerPort: 16687

Make sure to replace <ES_SERVER_IP> and <ES_SERVER_PORT> with the actual IP address and port of your Elasticsearch service.

5. TLS Configuration: If your Elasticsearch cluster requires TLS, ensure that the Jaeger components are configured with the appropriate TLS settings. For example, you can enable TLS by setting the following environment variables:

- name: ES_TLS_ENABLED
  value: "true"
- name: ES_TLS_CA
  value: "/path/to/ca.crt"
- name: ES_TLS_CERT
  value: "/path/to/client.crt"
- name: ES_TLS_KEY
  value: "/path/to/client.key"

6. Elasticsearch Configuration: Ensure that the Elasticsearch cluster is properly configured and running. You may need to adjust settings such as the number of replicas, shards, and other parameters to match your deployment requirements. For example, you can configure the number of replicas and shards as follows:

- name: es.num-replicas
  value: "1"
- name: es.num-shards
  value: "5"

7. Jaeger Configuration Options: Jaeger binaries can be configured in a number of ways, including command line arguments, environment variables, and configuration files in JSON, TOML, YAML, HCL, or Java properties formats. To see the complete list of options, run the binary with the help command or refer to the CLI Flags page for more information. Options that are specific to a certain storage backend are only listed if the storage type is selected.

If the issue persists, you may need to check the Jaeger logs for more detailed error messages and ensure that the Elasticsearch cluster is properly configured and running. For more detailed information on deploying Jaeger, you can refer to the Jaeger Deployment Documentation ^{[3][4][5][6][7]}.

To continue talking to Dosu, mention @dosu.

[agarwal62]

@dosu still same issue

[dosubot[bot]]

Ensure that the Jaeger components can properly connect to the Elasticsearch cluster by following these steps:

1. Verify Elasticsearch Endpoint: Double-check that the ES_SERVER_URLS environment variable is correctly set with the proper Elasticsearch endpoint URL.

2. Network Connectivity: From within the Jaeger pods, use tools like curl or telnet to test connectivity to the Elasticsearch endpoint.

3. Elasticsearch Health: Verify the health status of the Elasticsearch cluster by accessing the _cluster/health endpoint.

4. Configuration Parameters: Ensure all required configuration parameters for Elasticsearch are correctly set in the Jaeger deployment. Example configuration:

# Jaeger Collector
apiVersion: apps/v1
kind: Deployment
metadata:
  name: jaeger-collector
spec:
  template:
    spec:
      containers:
      - name: jaeger-collector
        image: jaegertracing/jaeger-collector:{{< currentVersion >}}
        env:
        - name: SPAN_STORAGE_TYPE
          value: elasticsearch
        - name: ES_SERVER_URLS
          value: http://<ES_SERVER_IP>:<ES_SERVER_PORT>
        ports:
        - containerPort: 14269
        - containerPort: 14250
        - containerPort: 14268

# Jaeger Query Service
apiVersion: apps/v1
kind: Deployment
metadata:
  name: jaeger-query
spec:
  template:
    spec:
      containers:
      - name: jaeger-query
        image: jaegertracing/jaeger-query:{{< currentVersion >}}
        env:
        - name: SPAN_STORAGE_TYPE
          value: elasticsearch
        - name: ES_SERVER_URLS
          value: http://<ES_SERVER_IP>:<ES_SERVER_PORT>
        ports:
        - containerPort: 16685
        - containerPort: 16686
        - containerPort: 16687

Replace <ES_SERVER_IP> and <ES_SERVER_PORT> with the actual IP address and port of your Elasticsearch service.

5. TLS Configuration: If your Elasticsearch cluster requires TLS, configure the Jaeger components with the appropriate TLS settings:

- name: ES_TLS_ENABLED
  value: "true"
- name: ES_TLS_CA
  value: "/path/to/ca.crt"
- name: ES_TLS_CERT
  value: "/path/to/client.crt"
- name: ES_TLS_KEY
  value: "/path/to/client.key"

6. Elasticsearch Configuration: Ensure the Elasticsearch cluster is properly configured and running. Adjust settings such as the number of replicas and shards as needed:

- name: es.num-replicas
  value: "1"
- name: es.num-shards
  value: "5"

7. Jaeger Configuration Options: Jaeger binaries can be configured in various ways, including command line arguments, environment variables, and configuration files in JSON, TOML, YAML, HCL, or Java properties formats. For a complete list of options, run the binary with the help command or refer to the CLI Flags page.

If the issue persists, check the Jaeger logs for more detailed error messages and ensure the Elasticsearch cluster is properly configured and running. For more detailed information on deploying Jaeger, refer to the Jaeger Deployment Documentation ^{[1][2][3][4][5]}.

To continue talking to Dosu, mention @dosu.

[agarwal62]

Elastic Search is running properly