Faulty RemoteJwks first try will never be retried #7803

Ati59 · 2023-02-07T13:53:04Z

Gloo Edge Version

1.13.x (latest stable)

Kubernetes Version

None

Describe the bug

If the RemoteJwks url is not reachable at the extauth service launch, it will throw an error failed to fetch JWKS but will never retried to get the the jwks even if refreshInterval is set.

Then every call through a route using the AuthConfig will get a 403 error.

On ext-auth logs :

{"level":"error","ts":"2023-02-07T13:30:10.094Z","caller":"jwks/utils.go:24","msg":"failed to fetch JWKS","version":"1.12.40","error":"Get \"http://172.18.2.2:8080/realms/master/protocol/openid-connect/certs\": dia
l tcp 172.18.2.2:8080: connect: connection refused","stacktrace":"github.com/solo-io/ext-auth-service/pkg/config/utils/jwks.FetchJwksWithClient\n\t/go/pkg/mod/github.com/solo-io/[email protected]/pkg/config/
utils/jwks/utils.go:24\ngithub.com/solo-io/ext-auth-service/pkg/config/utils/jwks.FetchJwks\n\t/go/pkg/mod/github.com/solo-io/[email protected]/pkg/config/utils/jwks/utils.go:16\ngithub.com/solo-io/ext-auth-
service/pkg/config/oauth/token_validation/jwt/jwks.NewRemoteJwksSource\n\t/go/pkg/mod/github.com/solo-io/[email protected]/pkg/config/oauth/token_validation/jwt/jwks/remote.go:71\ngithub.com/solo-io/ext-auth
-service/pkg/config.(*authServiceFactory).NewOAuth2JwtAccessToken\n\t/go/pkg/mod/github.com/solo-io/[email protected]/pkg/config/factory.go:218\ngithub.com/solo-io/solo-projects/projects/extauth/pkg/config.(
*extAuthConfigTranslator).authConfigToService\n\t/go/src/github.com/solo-io/solo-projects/projects/extauth/pkg/config/translator.go:273\ngithub.com/solo-io/solo-projects/projects/extauth/pkg/config.(*extAuthConfigT
ranslator).getConfigs\n\t/go/src/github.com/solo-io/solo-projects/projects/extauth/pkg/config/translator.go:98\ngithub.com/solo-io/solo-projects/projects/extauth/pkg/config.(*extAuthConfigTranslator).Translate\n\t/
go/src/github.com/solo-io/solo-projects/projects/extauth/pkg/config/translator.go:82\ngithub.com/solo-io/solo-projects/projects/extauth/pkg/config.(*configGenerator).GenerateConfig\n\t/go/src/github.com/solo-io/sol
o-projects/projects/extauth/pkg/config/generator.go:86\ngithub.com/solo-io/solo-projects/projects/extauth/pkg/runner.(*configSource).Run.func1.1\n\t/go/src/github.com/solo-io/solo-projects/projects/extauth/pkg/runn
er/xds.go:116\ngithub.com/solo-io/gloo/projects/gloo/pkg/api/v1/enterprise/options/extauth/v1.applyExtAuthConfig.func1\n\t/go/pkg/mod/github.com/solo-io/[email protected]/projects/gloo/pkg/api/v1/enterprise/options/ext
auth/v1/ext_auth_discovery_service_xds.sk.go:111\ngithub.com/solo-io/solo-kit/pkg/api/v1/control-plane/client.(*client).Start\n\t/go/pkg/mod/github.com/solo-io/[email protected]/pkg/api/v1/control-plane/client/clien
t.go:137\ngithub.com/solo-io/solo-projects/projects/extauth/pkg/runner.(*configSource).Run.func1\n\t/go/src/github.com/solo-io/solo-projects/projects/extauth/pkg/runner/xds.go:145\ngithub.com/solo-io/go-utils/conte
xtutils.(*exponentialBackoff).Backoff\n\t/go/pkg/mod/github.com/solo-io/[email protected]/contextutils/backoff.go:70\ngithub.com/solo-io/solo-projects/projects/extauth/pkg/runner.(*configSource).Run\n\t/go/src/githu
b.com/solo-io/solo-projects/projects/extauth/pkg/runner/xds.go:154\ngithub.com/solo-io/ext-auth-service/pkg/server.Server.Run.func2\n\t/go/pkg/mod/github.com/solo-io/[email protected]/pkg/server/server.go:14
9"}
{"level":"error","ts":"2023-02-07T13:30:10.098Z","caller":"config/generator.go:114","msg":"Errors encountered while processing new server configuration","version":"1.12.40","error":"1 error occurred:\n\t* failed to
 get auth service for auth config with id [gloo-system.accesstoken-auth]; this configuration will be ignored: failed to fetch JWKS: Get \"http://172.18.2.2:8080/realms/master/protocol/openid-connect/certs\": dial t
cp 172.18.2.2:8080: connect: connection refused\n\n","stacktrace":"github.com/solo-io/solo-projects/projects/extauth/pkg/config.(*configGenerator).GenerateConfig\n\t/go/src/github.com/solo-io/solo-projects/projects
/extauth/pkg/config/generator.go:114\ngithub.com/solo-io/solo-projects/projects/extauth/pkg/runner.(*configSource).Run.func1.1\n\t/go/src/github.com/solo-io/solo-projects/projects/extauth/pkg/runner/xds.go:116\ngit
hub.com/solo-io/gloo/projects/gloo/pkg/api/v1/enterprise/options/extauth/v1.applyExtAuthConfig.func1\n\t/go/pkg/mod/github.com/solo-io/[email protected]/projects/gloo/pkg/api/v1/enterprise/options/extauth/v1/ext_auth_d
iscovery_service_xds.sk.go:111\ngithub.com/solo-io/solo-kit/pkg/api/v1/control-plane/client.(*client).Start\n\t/go/pkg/mod/github.com/solo-io/[email protected]/pkg/api/v1/control-plane/client/client.go:137\ngithub.c
om/solo-io/solo-projects/projects/extauth/pkg/runner.(*configSource).Run.func1\n\t/go/src/github.com/solo-io/solo-projects/projects/extauth/pkg/runner/xds.go:145\ngithub.com/solo-io/go-utils/contextutils.(*exponent
ialBackoff).Backoff\n\t/go/pkg/mod/github.com/solo-io/[email protected]/contextutils/backoff.go:70\ngithub.com/solo-io/solo-projects/projects/extauth/pkg/runner.(*configSource).Run\n\t/go/src/github.com/solo-io/solo
-projects/projects/extauth/pkg/runner/xds.go:154\ngithub.com/solo-io/ext-auth-service/pkg/server.Server.Run.func2\n\t/go/pkg/mod/github.com/solo-io/[email protected]/pkg/server/server.go:149"}
{"level":"error","ts":1675776616.4879637,"logger":"ext-auth.ext-auth-service","msg":"Auth Server does not contain auth configuration with the given ID","version":"undefined","x-request-id":"b9568388-3cb7-4e1b-8b0d-
135454e91eb4","requestContext":{"AuthConfigId":"gloo-system.accesstoken-auth","SourceType":"virtual_host","SourceName":"gloo-system.gateway-proxy-listener-::-8080-gloo-system_vs"},"stacktrace":"github.com/envoyprox
y/go-control-plane/envoy/service/auth/v3._Authorization_Check_Handler.func1\n\t/go/pkg/mod/github.com/envoyproxy/[email protected]/envoy/service/auth/v3/external_auth.pb.go:699\ngithub.com/solo-io/go-utils/h
ealthchecker.GrpcUnaryServerHealthCheckerInterceptor.func1\n\t/go/pkg/mod/github.com/solo-io/[email protected]/healthchecker/grpc.go:69\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.go
lang.org/[email protected]/server.go:1135\ngithub.com/solo-io/ext-auth-service/pkg/server.requestIdInterceptor.func1\n\t/go/pkg/mod/github.com/solo-io/[email protected]/pkg/server/logging.go:86\ngoogle.golang.org
/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1138\ngithub.com/grpc-ecosystem/go-grpc-middleware/logging/zap.UnaryServerInterceptor.func1\n\t/go/pkg/mod/github.com/grp
c-ecosystem/[email protected]/logging/zap/server_interceptors.go:31\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1138\ngoogle.golang.org/grp
c.chainUnaryInterceptors.func1\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1140\ngithub.com/envoyproxy/go-control-plane/envoy/service/auth/v3._Authorization_Check_Handler\n\t/go/pkg/mod/github.com/envoy
proxy/[email protected]/envoy/service/auth/v3/external_auth.pb.go:701\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1301\ngoogle.golang.org/grpc.(*
Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1642\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:938"}

On gateway-proxy logs :

[2023-02-07T13:30:59.213Z] "HEAD /get HTTP/1.1" 403 UAEX 0 0 52 - "-" "curl/7.85.0" "22916f1f-ce5e-4385-ad0d-55fcfea928f3" "httpbin.domain.local" "-"
[2023-02-07T13:31:00.414Z] "HEAD /get HTTP/1.1" 403 UAEX 0 0 11 - "-" "curl/7.85.0" "0e781a03-9441-491f-a4a7-685dd7475c38" "httpbin.domain.local" "-"
[2023-02-07T13:31:02.367Z] "HEAD /get HTTP/1.1" 403 UAEX 0 0 6 - "-" "curl/7.85.0" "c4c7eb49-473b-45a3-b601-7983f04d0d8a" "httpbin.domain.local" "-"

Steps to reproduce the bug

Define an AuthConfig as-is :

apiVersion: enterprise.gloo.solo.io/v1
kind: AuthConfig
metadata:
  name: accesstoken-auth
  namespace: gloo-system
spec:
  configs:
    - oauth2:
        accessTokenValidation:
          jwt:
            remoteJwks:
              url: ${KEYCLOAK_URL}/realms/master/protocol/openid-connect/certs
              refreshInterval: "10"

And a VirtualService that is using it :

apiVersion: gateway.solo.io/v1
kind: VirtualService
metadata:
  name: vs
  namespace: gloo-system
spec:
  virtualHost:
    domains:
      - '*'
    routes:
      - matchers:
          - prefix: /
        routeAction:
          single:
            upstream:
              name: httpbin-httpbin-8000
              namespace: gloo-system
    options:
      extauth:
        configRef:
          name: accesstoken-auth
          namespace: gloo-system

Make keycloak unavailable (I change the selector to get all endpoint out of the service)
Restart the ext-auth service, it should not get the jwks and you should see failed to fetch JWKS error on ext-auth logs
Wait for refreshInterval
Trying to curl your exposed app should end up on UAEX errors on gateway-proxy pod and Auth Server does not contain auth configuration with the given ID errors on ext-auth one
(6. If you make keycloak available again and then restart the ext-auth pod, it will fix the issue)

Expected Behavior

Ext-auth pod should retry to get the jwks based on refreshInterval value so we can get through the authentication process and end up with 200 without having to restart the ext-auth pod.
When the first try is faulty it looks like the "refresh loop" is not launch at all.

Additional Context

No response

Related Issues

ExtAuth pod should be able to automatically fetch Authconfig endpoint #8372

┆Issue is synchronized with this Asana task by Unito

The text was updated successfully, but these errors were encountered:

SantoDE · 2023-02-09T16:17:36Z

This might be a dupe

Ati59 · 2023-02-10T08:25:04Z

We have a KCS article on zendesk for this. It is probably the "dupe" ;)

sam-heilbron · 2023-07-10T12:12:46Z

Similar issue: #7528

sam-heilbron · 2023-09-13T21:48:20Z

Additional context: https://solo-io-corp.slack.com/archives/C02LQ0JCNLF/p1694578006362599
We support an on demand cache refresh policy (https://docs.solo.io/gloo-edge/latest/reference/api/github.com/solo-io/gloo/projects/gloo/api/v1/enterprise/options/extauth/v1/extauth.proto.sk/#jwksondemandcacherefreshpolicy) in OIDC that could be re-used in this service (would require code changes)

DuncanDoyle · 2024-03-01T13:03:34Z

Can still reproduce this on GE 1.15.14. The problem seems to be that when the ExtAuth server loads with an AuthConfig that points to a JWKS endpoint that is not available/reachable, the config does not get loaded. And when the config does not get loaded, the refreshInterval does not get triggered at all. So the only way to get out of that situation is to reload the AuthConfig or restart the ExtAuth server.

{"level":"info","ts":"2024-03-01T10:22:17.510Z","caller":"runner/xds.go:115","msg":"{\"auth_config_ref_name\":\"gloo-system.oauth-auth\",\"configs\":[{\"AuthConfig\":{\"Oauth2\":{\"OauthType\":{\"AccessTokenValidationConfig\":{\"ValidationType\":{\"Jwt\":{\"JwksSourceSpecifier\":{\"RemoteJwks\":{\"url\":\"http://keycloak.example.com/realms/master/protocol/openid-connect/certs\",\"refresh_interval\":{\"seconds\":10}}}}},\"ScopeValidation\":null}}}}}]}","version":"1.15.14"}
{"level":"error","ts":"2024-03-01T10:22:17.539Z","caller":"jwks/utils.go:24","msg":"failed to fetch JWKS","version":"1.15.14","error":"request failed with code 503 Service Unavailable","stacktrace":"github.com/solo-io/ext-auth-service/pkg/config/utils/jwks.FetchJwksWithClient\n\t/go/pkg/mod/github.com/solo-io/[email protected]/pkg/config/utils/jwks/utils.go:24\ngithub.com/solo-io/ext-auth-service/pkg/config/utils/jwks.FetchJwks\n\t/go/pkg/mod/github.com/solo-io/[email protected]/pkg/config/utils/jwks/utils.go:16\ngithub.com/solo-io/ext-auth-service/pkg/config/oauth/token_validation/jwt/jwks.NewRemoteJwksSource\n\t/go/pkg/mod/github.com/solo-io/[email protected]/pkg/config/oauth/token_validation/jwt/jwks/remote.go:71\ngithub.com/solo-io/ext-auth-service/pkg/config.(*authServiceFactory).NewOAuth2JwtAccessTokenAuthService\n\t/go/pkg/mod/github.com/solo-io/[email protected]/pkg/config/factory.go:318\ngithub.com/solo-io/solo-projects/projects/extauth/pkg/config.(*extAuthConfigTranslator).authConfigToService\n\t/go/src/github.com/solo-io/solo-projects/projects/extauth/pkg/config/translator.go:310\ngithub.com/solo-io/solo-projects/projects/extauth/pkg/config.(*extAuthConfigTranslator).getConfigs\n\t/go/src/github.com/solo-io/solo-projects/projects/extauth/pkg/config/translator.go:103\ngithub.com/solo-io/solo-projects/projects/extauth/pkg/config.(*extAuthConfigTranslator).Translate\n\t/go/src/github.com/solo-io/solo-projects/projects/extauth/pkg/config/translator.go:87\ngithub.com/solo-io/solo-projects/projects/extauth/pkg/config.(*configGenerator).GenerateConfig\n\t/go/src/github.com/solo-io/solo-projects/projects/extauth/pkg/config/generator.go:86\ngithub.com/solo-io/solo-projects/projects/extauth/pkg/runner.(*configSource).Run.func1.1\n\t/go/src/github.com/solo-io/solo-projects/projects/extauth/pkg/runner/xds.go:121\ngithub.com/solo-io/gloo/projects/gloo/pkg/api/v1/enterprise/options/extauth/v1.applyExtAuthConfig.func1\n\t/go/pkg/mod/github.com/solo-io/[email protected]/projects/gloo/pkg/api/v1/enterprise/options/extauth/v1/ext_auth_discovery_service_xds.sk.go:111\ngithub.com/solo-io/solo-kit/pkg/api/v1/control-plane/client.(*client).Start\n\t/go/pkg/mod/github.com/solo-io/[email protected]/pkg/api/v1/control-plane/client/client.go:137\ngithub.com/solo-io/solo-projects/projects/extauth/pkg/runner.(*configSource).Run.func1\n\t/go/src/github.com/solo-io/solo-projects/projects/extauth/pkg/runner/xds.go:148\ngithub.com/solo-io/go-utils/contextutils.(*exponentialBackoff).Backoff\n\t/go/pkg/mod/github.com/solo-io/[email protected]/contextutils/backoff.go:70\ngithub.com/solo-io/solo-projects/projects/extauth/pkg/runner.(*configSource).Run\n\t/go/src/github.com/solo-io/solo-projects/projects/extauth/pkg/runner/xds.go:157\ngithub.com/solo-io/ext-auth-service/pkg/server.Server.Run.func3\n\t/go/pkg/mod/github.com/solo-io/[email protected]/pkg/server/server.go:160"}
{"level":"error","ts":"2024-03-01T10:22:17.541Z","caller":"config/generator.go:114","msg":"Errors encountered while processing new server configuration","version":"1.15.14","error":"1 error occurred:\n\t* failed to get auth service for auth config with id [gloo-system.oauth-auth]; this configuration will be ignored: failed to fetch JWKS: request failed with code 503 Service Unavailable\n\n","stacktrace":"github.com/solo-io/solo-projects/projects/extauth/pkg/config.(*configGenerator).GenerateConfig\n\t/go/src/github.com/solo-io/solo-projects/projects/extauth/pkg/config/generator.go:114\ngithub.com/solo-io/solo-projects/projects/extauth/pkg/runner.(*configSource).Run.func1.1\n\t/go/src/github.com/solo-io/solo-projects/projects/extauth/pkg/runner/xds.go:121\ngithub.com/solo-io/gloo/projects/gloo/pkg/api/v1/enterprise/options/extauth/v1.applyExtAuthConfig.func1\n\t/go/pkg/mod/github.com/solo-io/[email protected]/projects/gloo/pkg/api/v1/enterprise/options/extauth/v1/ext_auth_discovery_service_xds.sk.go:111\ngithub.com/solo-io/solo-kit/pkg/api/v1/control-plane/client.(*client).Start\n\t/go/pkg/mod/github.com/solo-io/[email protected]/pkg/api/v1/control-plane/client/client.go:137\ngithub.com/solo-io/solo-projects/projects/extauth/pkg/runner.(*configSource).Run.func1\n\t/go/src/github.com/solo-io/solo-projects/projects/extauth/pkg/runner/xds.go:148\ngithub.com/solo-io/go-utils/contextutils.(*exponentialBackoff).Backoff\n\t/go/pkg/mod/github.com/solo-io/[email protected]/contextutils/backoff.go:70\ngithub.com/solo-io/solo-projects/projects/extauth/pkg/runner.(*configSource).Run\n\t/go/src/github.com/solo-io/solo-projects/projects/extauth/pkg/runner/xds.go:157\ngithub.com/solo-io/ext-auth-service/pkg/server.Server.Run.func3\n\t/go/pkg/mod/github.com/solo-io/[email protected]/pkg/server/server.go:160"}

Note that when you bring down Keycloak after the AuthConfig has already been loaded, the ExtAuth server will start giving errors that it can't refresh, but when you bring Keycloak back up again, it's able to refresh again.

The main question seems to be if we want to accept AuthConfigs that point to non-reachable endpoints. We can't really determine whether the AuthConfig is incorrect, or whether there is an issue with the target endpoint.

DuncanDoyle · 2024-03-05T13:11:44Z

Reproducer: https://github.com/DuncanDoyle/ge-gloo-7803

nfuden · 2024-03-05T18:29:36Z

We should have a first time start up version of extauth that forces authconfigs to keep retrying and not fail like they normally would in a case where we are applying new configuration

sync-by-unito · 2024-04-11T17:18:52Z

➤ Hanh Vu commented:

ETA of 4/12 for design review.
Implementation ETA is unknown.

sheidkamp · 2024-04-16T14:33:16Z

Outcome of design review was that the ideal approach for handling the situtation where the auth service is updated with a non-responding JWKs URL is to keep the new AuthService in a pending state until it can retrieve the URLs.

This requires changes in how we generate/translate/communicate the new AuthConfigs. The plan is to implement these structural changes in a separate PR and then add the JWKs specific changes on top of that.

This will require 3 rounds of PRs in the main branches:

solo-projects (SP1) - refactor event loop and introduce "pending" state
ext-auth-service (EXT1) - introduce mechanism to allow AuthServices to communicate readiness and implement it for Remote JWKs/ refactor Remote JWKs to (conditionally?) not fail on startup when the fetch fails
solo-projects (SP2) - changes to use the new ext-auth-service readiness functionality

sheidkamp · 2024-04-22T15:24:26Z

@DuncanDoyle - The changes needed to implement this are the type of structural changes that we usually don't like to implement in backports.

In this case we are making non-trivial modifications to the ExtAuth pod's xds event loop, and the alternative would involve breaking changes to the exported Generator or Translator interfaces that would normally only accompany a major version update. How big of an ask would it be to make these changes 1.17 only?

@kcbabo - tagging you too while Duncan is on vacation.

sync-by-unito · 2024-04-25T16:41:55Z

➤ Nathan F Solo commented:

As there are some interstitial prs hence we are pushing the final due Keith Babo

sheidkamp · 2024-04-26T19:47:31Z

First solo-projects PR merged (SP1 from #7803 (comment)) merged, EXT1 in review

sheidkamp · 2024-05-02T19:46:47Z

The Ext Auth changes have been merged, functional changes for the last PR are in place, spiffing up the e2e tests.

sheidkamp · 2024-05-03T19:37:58Z

This has been merged to solo-projects main, all PRs are complete, and the scenario in the reproducer is succeeding.

nfuden · 2024-05-06T14:02:42Z

Since this materially changes our extauth service's behavior this has been merged to main and will not be backported to 1.15.
ThereforeI have removed the 1.15 tag for now.

Ati59 added the Type: Bug Something isn't working label Feb 7, 2023

sam-heilbron added the Area: extauth label Jul 10, 2023

nfuden self-assigned this Jul 10, 2023

nfuden added Needs Investigation release-blocker-1.15 labels Jul 10, 2023

nfuden removed Needs Investigation release-blocker-1.15 labels Jul 20, 2023

sam-heilbron unassigned nfuden Aug 2, 2023

SantoDE added the release/1.15 label Aug 3, 2023

htpvu added the Prioritized Indicating issue prioritized to be worked on in RFE stream label Mar 6, 2024

DuncanDoyle added the Committed: 1.18 label Mar 8, 2024

sync-by-unito bot added the Spike label Mar 8, 2024

sam-heilbron assigned sheidkamp Mar 25, 2024

sheidkamp closed this as completed May 3, 2024

nfuden removed Spike release/1.15 labels May 6, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Faulty RemoteJwks first try will never be retried #7803

Faulty RemoteJwks first try will never be retried #7803

Ati59 commented Feb 7, 2023 •

edited by DuncanDoyle

Loading

SantoDE commented Feb 9, 2023

Ati59 commented Feb 10, 2023

sam-heilbron commented Jul 10, 2023

sam-heilbron commented Sep 13, 2023 •

edited

Loading

DuncanDoyle commented Mar 1, 2024 •

edited

Loading

DuncanDoyle commented Mar 5, 2024

nfuden commented Mar 5, 2024

sync-by-unito bot commented Apr 11, 2024

sheidkamp commented Apr 16, 2024 •

edited

Loading

sheidkamp commented Apr 22, 2024 •

edited

Loading

sync-by-unito bot commented Apr 25, 2024

sheidkamp commented Apr 26, 2024

sheidkamp commented May 2, 2024

sheidkamp commented May 3, 2024

nfuden commented May 6, 2024

Faulty RemoteJwks first try will never be retried #7803

Faulty RemoteJwks first try will never be retried #7803

Comments

Ati59 commented Feb 7, 2023 • edited by DuncanDoyle Loading

Gloo Edge Version

Kubernetes Version

Describe the bug

Steps to reproduce the bug

Expected Behavior

Additional Context

Related Issues

SantoDE commented Feb 9, 2023

Ati59 commented Feb 10, 2023

sam-heilbron commented Jul 10, 2023

sam-heilbron commented Sep 13, 2023 • edited Loading

DuncanDoyle commented Mar 1, 2024 • edited Loading

DuncanDoyle commented Mar 5, 2024

nfuden commented Mar 5, 2024

sync-by-unito bot commented Apr 11, 2024

sheidkamp commented Apr 16, 2024 • edited Loading

sheidkamp commented Apr 22, 2024 • edited Loading

sync-by-unito bot commented Apr 25, 2024

sheidkamp commented Apr 26, 2024

sheidkamp commented May 2, 2024

sheidkamp commented May 3, 2024

nfuden commented May 6, 2024

Ati59 commented Feb 7, 2023 •

edited by DuncanDoyle

Loading

sam-heilbron commented Sep 13, 2023 •

edited

Loading

DuncanDoyle commented Mar 1, 2024 •

edited

Loading

sheidkamp commented Apr 16, 2024 •

edited

Loading

sheidkamp commented Apr 22, 2024 •

edited

Loading