Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve tests in python_examples.py. #30115

Merged
merged 13 commits into from
Jun 3, 2024
Merged

Conversation

Vector73
Copy link
Collaborator

@Vector73 Vector73 commented May 16, 2024

Adds assert statements to check if the request succeeded before
validating against the openapi schema.

Improves the robustness of tests in python_examples.py.
Fixes:

Screenshots and screen captures:

Self-review checklist
  • Self-reviewed the changes for clarity and maintainability
    (variable names, code reuse, readability, etc.).

Communicate decisions, questions, and potential concerns.

  • Explains differences from previous plans (e.g., issue description).
  • Highlights technical choices and bugs encountered.
  • Calls out remaining decisions and concerns.
  • Automated tests verify logic where appropriate.

Individual commits are ready for review (see commit discipline).

  • Each commit is a coherent idea.
  • Commit message(s) explain reasoning and motivation for changes.

Completed manual review and testing of the following:

  • Visual appearance of the changes.
  • Responsiveness and internationalization.
  • Strings and tooltips.
  • End-to-end functionality of buttons, interactions and flows.
  • Corner cases, error conditions, and easily imagined bugs.

@zulipbot zulipbot added size: XL and removed size: M labels May 18, 2024
@Vector73 Vector73 changed the title [WIP] Improve tests in python_examples.py. Improve tests in python_examples.py. May 18, 2024
@Vector73 Vector73 marked this pull request as ready for review May 18, 2024 11:56
@Vector73 Vector73 requested a review from timabbott May 18, 2024 11:57
Copy link
Collaborator

@laurynmm laurynmm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Vector73 - Thanks for making these improvements. I had a few thoughts. Let me know if you have any questions about my comments.

Also, see my separate comment related to Greg's note. I think it'd be good to check both the success and error tests, so maybe instead of adding all of these asserts, we add a helper function to do that check.

zerver/openapi/python_examples.py Outdated Show resolved Hide resolved
zerver/openapi/python_examples.py Outdated Show resolved Hide resolved
zerver/openapi/python_examples.py Outdated Show resolved Hide resolved
zerver/openapi/python_examples.py Outdated Show resolved Hide resolved
zerver/openapi/python_examples.py Outdated Show resolved Hide resolved
zerver/openapi/python_examples.py Outdated Show resolved Hide resolved
@Vector73
Copy link
Collaborator Author

Vector73 commented May 22, 2024

I have addressed the feedbacks by @laurynmm. I have also fixed some incorrect validate_against_openapi_schema(..., 400) which I found while verifying the "error" response.

Also, there are 2 functions - test_user_account_deactivated and test_realm_deactivated - which don't seem to work as expected because result["result]="success" but the request (messages/render) is supposed to fail (REALM_DEACTIVATED or USER_DEACTIVATED error) with 401 status code (the current code though checks for 403 status code which has been changed to 401 for these errors).

Copy link
Collaborator

@laurynmm laurynmm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Vector73 - Here's another round of comments for these changes.

I'll note that this would be easier to review if you possible split this into more than one commit. And doing that would also possibly help you self-review for some of the small inconsistencies I've noticed in this last round of review.

So, having a commit for updating the linkifier endpoints to pass the filter_id parameter for example. And a separate commit for the helper function for getting subscription IDs. And so on.

Looking at how I split things out in #30135 might be helpful with that.


For those two error checks that are returning success:

  • I noted that if I comment out all the other tests in tools/api-tests and just run the deactivate realm test, then I do get the error response ... though the error code isn't the one documented and as you noted it's a 401 error now :/
  • I feel like you can work on figuring out those in a separate follow-up PR since they're already not working

For error checks here in general, I'm wondering if we:

  • Remove the calls to validate_against_openapi_schema since that's not currently validating the error response anyway.
  • Add a check for the expected code in the error result.

I'll start a discussion in CZO and/or file an issue for validating error responses, but unfortunately it's a bit more complicated than adding error documentation to all the documented endpoints.

zerver/openapi/python_examples.py Show resolved Hide resolved
zerver/openapi/python_examples.py Show resolved Hide resolved
zerver/openapi/python_examples.py Outdated Show resolved Hide resolved
zerver/openapi/python_examples.py Show resolved Hide resolved
zerver/openapi/python_examples.py Show resolved Hide resolved
zerver/openapi/python_examples.py Show resolved Hide resolved
zerver/openapi/python_examples.py Show resolved Hide resolved
zerver/openapi/python_examples.py Outdated Show resolved Hide resolved
zerver/openapi/python_examples.py Show resolved Hide resolved
zerver/openapi/python_examples.py Outdated Show resolved Hide resolved
@Vector73
Copy link
Collaborator Author

I have addressed the recent feedback by @laurynmm and split the changes into multiple commits.

Copy link
Collaborator

@laurynmm laurynmm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Vector73 - Thanks for splitting up your changes! It was much easier to review and inspired me to make a few more updates to these tests/this file.

I've pushed my updates here and it would be great for you to review those changes!

Copy link
Collaborator Author

@Vector73 Vector73 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @laurynmm for this clean-up. It looks great to me other than the comments below.

zerver/openapi/python_examples.py Outdated Show resolved Hide resolved
zerver/openapi/python_examples.py Outdated Show resolved Hide resolved
Creates a helper function `validate_message` to remove the repeated code
for verifying the sent messages.
Creates a helper function `get_subscribed_stream_ids` to fetch
subscribed streams' ids of the client.

Updates various functions to remove hardcoded values in request
body and used available API calls to fetch them.
Updates `*_realm_filter` functions to use `filter_id` from function
argument rather than a hardcoded value.
@laurynmm
Copy link
Collaborator

laurynmm commented Jun 3, 2024

@Vector73 - Thanks for making those updates. I think this looks good for now, so marking for integration review.

Follow-ups that remain are the error testing clean-ups:

  • Remove validate_against_openapi_schema for the error tests since that's not currently testing anything. Probably replace those with an assertion/check of the error code (often defaults to "BAD_REQUEST") that we expect to be returned for the error.
  • Fixing the two error tests in test-api that are not returning errors in the test suite: test_user_account_deactivated and test_realm_deactivated.

@laurynmm laurynmm added the integration review Added by maintainers when a PR may be ready for integration. label Jun 3, 2024
@timabbott timabbott merged commit c514d39 into zulip:main Jun 3, 2024
7 checks passed
@timabbott
Copy link
Sponsor Member

Merged, thanks for all the work on this @Vector73 and @laurynmm!

I really appreciate how much easier to maintain this now looks to be.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
integration review Added by maintainers when a PR may be ready for integration. size: XL
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants