Anthropic streaming support

[ssonal]

Streaming is now live on Anthropic. This change introduces json parsing support for streamed chunks.

https://docs.anthropic.com/en/api/messages-streaming

---

🚀 This description was created by Ellipsis for commit a846c45

Summary:

This PR introduces streaming support for Anthropic in the instructor library, updating methods and adding tests to ensure functionality.

Key points:

• Removed Anthropic restrictions in instructor/client.py for streaming support.
• Updated JSON extraction in instructor/dsl/iterable.py and instructor/dsl/partial.py for new Anthropic modes.
• Added new tests in tests/llm/test_anthropic/ to validate streaming functionality.
• Ensured all new test files are included in the test suite.

---

Generated with ❤️ by ellipsis.dev

[sweep-ai]

Sweep: PR Review

instructor/client.py

The changes remove assertions that prevent the create_partial and create_iterable methods from executing when the provider is Provider.ANTHROPIC.

Sweep Found These Issues

• Removing the assertion in create_partial may cause the method to fail or behave unexpectedly if Provider.ANTHROPIC does not support partial responses.

instructor/instructor/client.py

Line 128 in a846c45

strict: bool = True,


View Diff
• Removing the assertion in create_iterable may cause the method to fail or behave unexpectedly if Provider.ANTHROPIC does not support iterable responses.

instructor/instructor/client.py

Line 311 in a846c45

) -> AsyncGenerator[T, None]:


View Diff

---

instructor/dsl/iterable.py

The changes introduce new conditions to handle Mode.ANTHROPIC_JSON and Mode.ANTHROPIC_TOOLS in both extract_json and extract_json_async methods, ensuring that specific JSON chunks are yielded based on the mode.

Sweep Found These Issues

• The line yield chunk.model_extra.get("delta", "").get("partial_json", "") may raise an AttributeError if chunk.model_extra is None or does not contain the expected structure.

instructor/instructor/dsl/iterable.py

Lines 114 to 115 in a846c45

	yield chunk.model_extra.get("delta", "").get("partial_json", "")
	elif chunk.choices:


View Diff

---

instructor/dsl/partial.py

The changes introduce new conditions to handle Mode.ANTHROPIC_JSON and Mode.ANTHROPIC_TOOLS in both synchronous and asynchronous JSON extraction functions, improving flexibility in dealing with different streaming response formats.

Sweep Found These Issues

• The condition for Mode.ANTHROPIC_TOOLS in the asynchronous function does not check if chunk.model_extra is None, which could lead to an AttributeError.

instructor/instructor/dsl/partial.py

Line 185 in a846c45

yield chunk.model_extra.get("delta", "").get("partial_json", "")


View Diff

---

tests/llm/test_anthropic/conftest.py

The changes introduce pytest fixtures for Anthropic and AsyncAnthropic clients with conditional configuration based on the BRAINTRUST_API_KEY environment variable and the availability of the braintrust module.

---

tests/llm/test_anthropic/test_stream.py

The changes add four new test functions to validate synchronous and asynchronous creation of models returning either an iterable or a partial UserExtract object, using various parameter combinations.

Sweep Found These Issues

• The test_iterable_model_async function does not handle the case where the model is not iterable when stream is False, which could lead to a runtime error.

instructor/tests/llm/test_anthropic/test_stream.py

Lines 33 to 52 in a846c45

	@pytest.mark.parametrize("model, mode, stream", product(models, modes, [True, False]))
	@pytest.mark.asyncio
	async def test_iterable_model_async(model, mode, stream, aclient):
	aclient = instructor.from_anthropic(aclient, mode=mode)
	model = await aclient.messages.create(
	model=model,
	response_model=Iterable[UserExtract],
	max_retries=2,
	stream=stream,
	max_tokens=1024,
	messages=[
	{"role": "user", "content": "Make two up people"},
	],
	)
	if stream:
	async for m in model:
	assert isinstance(m, UserExtract)
	else:
	for m in model:
	assert isinstance(m, UserExtract)


View Diff
• The test_partial_model_async function assumes that the model is always iterable, which may not be the case and could lead to a runtime error.

instructor/tests/llm/test_anthropic/test_stream.py

Lines 72 to 87 in a846c45

	@pytest.mark.parametrize("model,mode", product(models, modes))
	@pytest.mark.asyncio
	async def test_partial_model_async(model, mode, aclient):
	aclient = instructor.from_anthropic(aclient, mode=mode)
	model = await aclient.messages.create(
	model=model,
	response_model=Partial[UserExtract],
	max_retries=2,
	stream=True,
	max_tokens=1024,
	messages=[
	{"role": "user", "content": "Jason Liu is 12 years old"},
	],
	)
	async for m in model:
	assert isinstance(m, UserExtract)


View Diff

---

tests/llm/test_anthropic/util.py

Introduced a new utility file for testing with model and mode configurations.

Potential Issues

Sweep isn't 100% sure if the following are issues or not but they may be worth taking a look at.

• The code assumes that the instructor module and instructor.Mode.ANTHROPIC_TOOLS are available and correctly imported, which could cause an ImportError if they are not.

instructor/tests/llm/test_anthropic/util.py

Lines 1 to 5 in a846c45

	import instructor
	models = ["claude-3-haiku-20240307"]
	modes = [
	instructor.Mode.ANTHROPIC_TOOLS,


View Diff

---

[ellipsis-dev]

❌ Changes requested. Reviewed everything up to a846c45 in 47 seconds

More details

• Looked at 242 lines of code in 6 files
• Skipped 0 files when reviewing.
• Skipped posting 0 drafted comments based on config settings.

Workflow ID: wflow_dnsiYwLSpx2waGgY

---

Want Ellipsis to fix these issues? Tag @ellipsis-dev in a comment. You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.

[ellipsis-dev]

The PR introduces significant functionality changes by supporting streaming for Anthropic. Please ensure that the documentation is updated to reflect these changes, and consider adding relevant entries to the mkdocs.yml if new documentation files are created.

[jxnl]

Suggested change

	yield chunk.model_extra.get("delta", "").get("partial_json", "")
	yield chunk.model_extra.get("delta", {}).get("partial_json", "")

should it be {} since "".get would error?

[jxnl]

can you try and update the documentation on anthropic docs?

@ellipsis-dev

[ellipsis-dev]

@jxnl, I have addressed your comments in pull request #689