Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

oauth2-cc monitors not recovered after long outage #5050

Closed
1 task done
fabry006 opened this issue Aug 26, 2024 · 1 comment
Closed
1 task done

oauth2-cc monitors not recovered after long outage #5050

fabry006 opened this issue Aug 26, 2024 · 1 comment
Labels
area:monitor Everything related to monitors bug Something isn't working type:enhance-existing feature wants to enhance existing monitor

Comments

@fabry006
Copy link

πŸ“‘ I have found these related issues/pull requests

N/A

πŸ›‘οΈ Security Policy

Description

all our monitors configured with oauth2-cc detected an issue with our oAuth2 provider as expected.
The IDP had been down form multiple hours.
As soon as the IDP went back the monitors didn't auto recovered (we let them re-try for the whole weekend).
The only way to fix it was to pause / un-pause all the monitors.
Is it possible that they were trying to re-use an old token instead to ask for a new one?
In our logs we see that Uptime was providing an invalid (or expired) token. Unfortunately the token is redacted from the logs and cannot provide more info

πŸ‘Ÿ Reproduction steps

I can tell you what happened, haven't tried yet to reproduce it:

  • HAve a oauth monitor that works
  • shutdown the token provider IDP for 3/4 hours (maybe just longer than the tocke expiration date)
  • start again the token provider
  • the monitors shouldn't recover
  • pause / un-pause the monitor

πŸ‘€ Expected behavior

The monitors should recover automatically and try to fetch a new token

πŸ˜“ Actual Behavior

Monitors do not recover. They staid "OFFLINE" for 2 days until we pause / un-pause them

🐻 Uptime-Kuma Version

1.23.13-debian

πŸ’» Operating System and Arch

Docker

🌐 Browser

Chrome

πŸ–₯️ Deployment Environment

  • Runtime: Docker
  • Database: N/A
  • Filesystem used to store the database on: local storage
  • number of monitors: ~80

πŸ“ Relevant log output

2024-08-26T09:01:35+02:00 [MONITOR] WARN: Monitor #312 'oauth-test': Failing: Request failed with status code 401 | Interval: 120 seconds | Type: http | Down Count: 0 | Resend Interval: 0

2024-08-26T09:14:57+02:00 [RATE-LIMIT] INFO: remaining requests: 60
2024-08-26T09:15:18+02:00 [RATE-LIMIT] INFO: remaining requests: 60
2024-08-26T09:15:35+02:00 [MONITOR] WARN: Monitor #312 'oauth-test': Failing: Request failed with status code 401 | Interval: 120 seconds | Type: http | Down Count: 0 | Resend Interval: 0
2024-08-26T09:19:36+02:00 [MONITOR] WARN: Monitor #312 'oauth-test': Failing: Request failed with status code 401 | Interval: 120 seconds | Type: http | Down Count: 0 | Resend Interval: 0
2024-08-26T09:19:57+02:00 [RATE-LIMIT] INFO: remaining requests: 60
2024-08-26T09:20:18+02:00 [RATE-LIMIT] INFO: remaining requests: 60
2024-08-26T09:21:07+02:00 [MANAGE] INFO: Pause Monitor: 312 User ID: 1
2024-08-26T09:21:10+02:00 [MANAGE] INFO: Resume Monitor: 312 User ID: 1
@fabry006 fabry006 added the bug Something isn't working label Aug 26, 2024
@CommanderStorm
Copy link
Collaborator

Since #3903 we now refresh oauth credentials on 401.
See #4500 for further context on what needs to get done to get said PR released. (literally any help from going through PRs and writing changelogs to writing testcases to writing the needded migration helps)

@CommanderStorm CommanderStorm added area:monitor Everything related to monitors type:enhance-existing feature wants to enhance existing monitor labels Aug 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:monitor Everything related to monitors bug Something isn't working type:enhance-existing feature wants to enhance existing monitor
Projects
None yet
Development

No branches or pull requests

4 participants
@CommanderStorm @fabry006 and others