[Bug]: Jager UI fails with HTTP Error: error reading service_names from storage: Cannot achieve consistency level LOCAL_ONE

[yitzhtal]

What happened?

I installed Jaeger with Cassandra, all pods seem to run without any issues but I get from the Jaeger UI the following error:

This looks similar to this old issue but i'm not using jaeger all in one - #513.

Steps to reproduce

Jaeger chart details:
appVersion: 1.53.0
version: 1.0.2

Expected behavior

Jaeger UI should work without UI Errors

Relevant log output

│ {"level":"error","ts":1716026433.7536812,"caller":"app/http_handler.go:492","msg":"HTTP handler, Internal Server Error","error":"error readi │
│ ng service_names from storage: Cannot achieve consistency level LOCAL_ONE","stacktrace":"github.com/jaegertracing/jaeger/cmd/query/app.(*API │
│ Handler).handleError\n\tgithub.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:492\ngithub.com/jaegertracing/jaeger/cmd/query/app.(*A │
│ PIHandler).getServices\n\tgithub.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:163\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/serv │
│ er.go:2136\ngithub.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).handleFunc.traceResponseHandler.func2\n\tgithub.com/jaegertracing/ja │
│ eger/cmd/query/app/http_handler.go:536\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2136\ngithub.com/jaegertracing/jaeger/cmd/query │
│ /app.(*APIHandler).handleFunc.WithRouteTag.func3\n\tgo.opentelemetry.io/contrib/instrumentation/net/http/[email protected]/handler.go:285\nne │
│ t/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2136\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHT │
│ TP\n\tgo.opentelemetry.io/contrib/instrumentation/net/http/[email protected]/handler.go:229\ngo.opentelemetry.io/contrib/instrumentation/net/ │
│ http/otelhttp.NewMiddleware.func1.1\n\tgo.opentelemetry.io/contrib/instrumentation/net/http/[email protected]/handler.go:81\nnet/http.Handler │
│ Func.ServeHTTP\n\tnet/http/server.go:2136\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2136\ngithub.com/gorilla/mux.(*Router).Serve │
│ HTTP\n\tgithub.com/gorilla/[email protected]/mux.go:212\ngithub.com/jaegertracing/jaeger/cmd/query/app.createHTTPServer.additionalHeadersHandler.fu │
│ nc4\n\tgithub.com/jaegertracing/jaeger/cmd/query/app/additional_headers_handler.go:28\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go: │
│ 2136\ngithub.com/jaegertracing/jaeger/cmd/query/app.createHTTPServer.CompressHandler.CompressHandlerLevel.func6\n\tgithub.com/gorilla/handle │
│ [email protected]/compress.go:141\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2136\ngithub.com/gorilla/handlers.recoveryHandler.ServeHTTP\ │
│ n\tgithub.com/gorilla/[email protected]/recovery.go:78\nnet/http.serverHandler.ServeHTTP\n\tnet/http/server.go:2938\nnet/http.(*conn).serve\n\ │
│ tnet/http/server.go:2009"}

Additional context

No response

Jaeger backend version

appVersion: 1.53.0

SDK

No response

Pipeline

No response

Stogage backend

Cassandra

Operating system

Linux

Deployment model

Kubernetes; Helm

Deployment configs

Overriding values.yaml with:

provisionDataStore:
  cassandra: true
agent:
  enabled: false
schema:
  extraEnv:
   - name: TRACE_TTL
     value: "7200"
   - name: DEPENDENCIES_TTL
     value: "7200"
query:
  nodeSelector:
    purpose: on-demand
  tolerations:
    - key: purpose
      operator: "Exists"
      effect: "NoSchedule"
  ingress:
    enabled: true
    ingressClassName: nginx
    hosts:
      - jaeger-ui-solutions.internal.lightrun.com
  config: |-
    {
      "dependencies": {
        "dagMaxNumServices": 200,
        "menuEnabled": true
      },
      "archiveEnabled": true,
      "tracking": {
        "gaID": "UA-000000-2",
        "trackErrors": true
      }
    }
cassandra:
  selector:
    nodeSelector:
      purpose: on-demand
  tolerations:
    - key: purpose
      operator: "Exists"
      effect: "NoSchedule"
collector:
  nodeSelector:
    purpose: on-demand
  tolerations:
    - key: purpose
      operator: "Exists"
      effect: "NoSchedule"
  service:
    otlp:
      grpc:
         name: otlp-grpc
         port: 4317
      http:
         name: otlp-http
         port: 4318