CI: failure of coverage upload doesn't change job status

[bsipocz]

Recently the coverage status upload ran into a github rate limit, but this didn't change the status of the job from green to red, instead generated false coverage % that sent us on a bit of a wild goose chase. Ideas how to improve CI:

• fail the job status if the upload fails, so it will be much easier to spot the issue and restart the job
• add in a sleep and retry logic in case of upload failure

[bsipocz]

There is a long standing upstream feature request for this here: codecov/codecov-action#926

It also links to many implemented workarounds, so I strongly suggest to adopt one of those instead (e.g. wrapping it all in a retry action) of waiting for an upstream, out-of-the box fix.

This should be an easy starter issue for a newcomer.

[pllim]

cc @rosteen since he was just struggling with this same problem over at jdaviz.

[pllim]

Thanks for the info, @bsipocz ! I will investigate.

[pllim]

Looking at the linked issue, I see two possible ways forward. Example implementations of each:

1. upload coverage in separate step nextcloud/appstore#1294 -- This does not require third-party Action. I think maintainer re-triggers only the upload part when upload fails. No automatic retry but Actions are guaranteed to be supported long time and no need to re-run the test suite.
2. Use the retry-action action to retry the codecov uploader action tagatac/bagoup#57 -- This uses a third-party Action. Looks super convenient but there is no guarantee upload would be successful within the attempts (which means you have to rerun everything again if that fails anyway) and not sure how long this Action is going to be maintained going forward.

I am leaning towards Option 1 for both astropy and jdaviz. What do you think, @bsipocz and @rosteen ?

[bsipocz]

The first one is fine, I think basically anything is fine that avoids rerunning the tests because of an upload error.

[pllim]

Huh, I think for astropy, I actually have to patch the OpenAstronomy workflow (OpenAstronomy/github-actions-workflows#199), but not for jdaviz (spacetelescope/jdaviz#2865).

[pllim]

@astrofrog reverted my attempt to address this at OpenAstronomy/github-actions-workflows#206

Also it never quite work because of a token failure that I cannot understand: #16535

So, for now, just check the logs diligently if you want to know if the upload really happened or not.