Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

confluence swallows any error that occurs during pagination #2394

Open
3 tasks
Tracked by #2393
seanstory opened this issue Apr 16, 2024 · 0 comments · May be fixed by #2610
Open
3 tasks
Tracked by #2393

confluence swallows any error that occurs during pagination #2394

seanstory opened this issue Apr 16, 2024 · 0 comments · May be fixed by #2610

Comments

@seanstory
Copy link
Member

seanstory commented Apr 16, 2024

Part of #2393

Description

The confluence connector currently swallows errors during its paginated_api_call() function. See:

except Exception as exception:
self._logger.warning(
f"Skipping data for type {url_name} from {url}. Exception: {exception}."
)
break

It would be better for these to be re-raised if they can't be handled, and for callers to be able to determine how significant the error is. Otherwise, critical calls (like the first attempt to list all pages) could fail, leading to a mostly empty sync that thinks it's successful.

Acceptance Criteria

  • paginated_api_call catches some types of errors, and retries/handles the ones it can
  • if paginated_api_call cannot handle an error, it re-raises it
  • calling code of paginated_api_call handles errors as makes sense for the use case. We balance the need to "fail fast" with the need to not crash a large sync because of a small data issue.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants