[Bug][github plugin] Unexpected "Bad Credentials" when accessing github connection proxy

[pimguilherme]

Search before asking

• I had searched in the issues and found no similar issues.

What happened

Hi! Hope all is well over there :)

I am not sure why this happens and can't tell the exact behavior, but we're seeing unexpected "Bad credentials" errors when accessing a specific Github plugin endpoint, like this: https://devlake/api/plugins/github/connections/3/proxy/rest/repos/my-org/my-repo

We use this endpoint when orchestrating the creationg of blueprints/projects from an API of ours (to abstract that away from the end user).

This seemed to work fine in version 0.17

This is the response we get

{
  "message": "Bad credentials",
  "documentation_url": "https://docs.github.com/rest"
}

Other Github connections seem unnaffected, and the authentication method we use for all of them is the same - github app. And it's also the same github app for all of them (which are just different orgs)

Apparently if I re-save the connection settings with the same token, things start working again.. I'm wondering if there's some sort of expired token being used here.. like it doesn't refresh it for some reason? Would you be able to help us understand this please?

one thing to note.. we've noticed our instance crashes quite often (about 3-4 times a day, haven investigated it very much, but apparently with OOM).. maybe this contributes to an unknown state, so wanted to share it

as a workaround, we are looking to perhaps trigger the connection settings update every 30 minutes or so

Thanks!

What do you expect to happen

We would expect the token to refresh properly (assuming we are using an expired token), and then we would not get Bad Credentials

How to reproduce

Not sure yet..

Anything else

It seems to occur every day, quite frequently.. this is the rate for the last 3 days:

Version

v0.21.0-rc1@ac4f7fe

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[pimguilherme]

this functionality is affected as well (it is not able to list repositories)

[klesh]

Hi there,

Thanks for reaching out! Let's address the two issues you mentioned:

1. Expired Token:

Unfortunately, due to GitHub's design, tokens cannot be refreshed. The best course of action is to regenerate a new token with the appropriate permissions for your needs.

2. OOM (Out-of-Memory) Error:

To help diagnose the OOM error, could you please provide some additional information:

Crash Logs: Were there any logs indicating the crash, such as "panic" messages, around the time of the error?
Container Resources: How much memory and CPU have you allocated to the container running the process?
Pipeline Context: Did the OOM occur while running a pipeline? If so, were there any specific workflows or tasks in progress during the breakdown?

[pimguilherme]

Hi @klesh, thanks for reaching back :D

I think we can focus solely on problem 1 on this issue if that's okay!

Unfortunately, due to GitHub's design, tokens cannot be refreshed. The best course of action is to regenerate a new token with the appropriate permissions for your needs.

We are using a github app, so I think the tokens should be re-generated with the app's secret key, which does not expire, right? These tokens only live for 1 hour if I am not mistaken.. so we just wanted to make sure they are being refreshed.. from what it looks, we might have caught a situation where it's not 😥

thanks!

[klesh]

@pimguilherme
I apologize for the misunderstanding. I wasn't aware of the GitHub App authorization process and initially thought you were referring to a Personal Access Token (PAT).

I've reviewed the code, and it appears the access token is regenerated at the start of each collection. This logic hasn't changed since version 0.17.

You mentioned the token expiring after an hour, which could be the issue. It's possible that previously collections completed within that timeframe. However, with growing data volumes, that might no longer be the case.

[pimguilherme]

Hi @klesh, thanks for the feedback! I apologize for the delay

We do have issues with large repositories that fail because they need to access github after the token has already expired :(

And I am not sure this is the same mechanics, but we also have problems loading data from the Github connection proxy (ie https://devlake/api/plugins/github/connections/3/proxy/rest/repos/my-org/my-repo), how does that token get refreshed?

Thanks again!!

[klesh]

@pimguilherme Yup, you are correct, the token for Github App should be refreshed when expires automatically.

The proxy endpoint is the perfect proof of the problem because the token is requested once when ApiClient gets instantiated and the proxy helper caches it so the token expires one hour later.

It might take a while to fix.

[klesh]

Do you have the status code of the Bad credentials response?