Rspec 'integration' tests for optional use by contributors and ci #145
Conversation
|
@mattlindsey Isn't RSpec enough for testing library code? Why do we need |
|
@alchaplinsky Yeah. I'll redo this proposal to only include the rspec file, exclude the 'integrations' folder when running rspec, and add a short note in the README suggesting contributors utilize the specs directly. |
|
@alchaplinsky. Is this better? I think they're useful. |
|
@mattlindsey I think integration tests for concept like Agent is a good idea. But we need to mock requests to external APIs. I don't think that anyone would want to pay for tokens used during test runs. |
|
Personally I wanted these to aid in development, but I can just keep them for myself. Existing tests already do the mocking. |
mattlindsey
changed the title
|
@alchaplinsky Sorry, I meant to tag you on the last comment. |
|
Ok, got it. So this is a type of test that actually runs against real APIs and makes sure that code works with real integrations. Thus we skip running them every time when we run a test suite and we can enable them by setting the Otherwise, it is not entirely clear what |
@alchaplinsky Yeah. That's better. Done |
|
Closing this to avoid too many open PRs but leaving issue #144 open. |
| @@ -13,6 +13,9 @@ | |||
| # Disable RSpec exposing methods globally on `Module` and `main` | |||
| config.disable_monkey_patching! | |||
|
|
|||
| # Only run integration tests when enabled | |||
| config.filter_run_excluding type: :integration unless ENV["INTEGRATION_TESTS_ENABLED"] | |||
There was a problem hiding this comment.
We could probably just filter out all integration tests by default is ENV["CI"] == true
| ) | ||
| result = agent.run(question: question) | ||
|
|
||
| expect(result).to include("distance between NYC and DC") |
There was a problem hiding this comment.
One way that people do this is indexing into a vectorsearch DB all of the variations of possible acceptable answers and then run a vector search agains them to see if they "match".
|
I'm reopening this PR because I think it's a good idea to have these optional 'integration' tests, especially for use when developing langchainrb itself. It's quicker and less error prone than using @andreibondarev I can update this PR to test |
This isn't done, but I'm looking for feedback on using either cucumber or rspec for integration tests, which would aid in BDD development and be run before releases. If rspec is chosen, we can configure it to skip the 'integration' folder unless requested.
This PR includes both options for comparison.
For #144
One difficulty here is I am getting several different answers for the tests, so what's the acceptance criteria? Just absence of errors?
5,845 soccer fields
204 soccer fields
4,125 soccer fields