bug: Langfuse craps out when there is an error upstream

[ajram23]

Describe the bug

If there is an upstream error with the service like OpenAI or grok, Langfuse says invalid JSON error

Langfuse craps out when there is an error with groq.RateLimitError: Error code: 429 - {'error': {'message': 'Rate limit reached for model llama3-70b-8192 in organization org_BLAH on tokens per minute (TPM): Limit 7000, Used 1607, Requested ~47333. Please try again in 5m59.484999999s. Visit https://console.groq.com/docs/rate-limits for more information.', 'type': 'tokens', 'code': 'rate_limit_exceeded'}}

Giving up execute_task_with_backoff(...) after 3 tries (langfuse.request.APIError: Invalid JSON (400): None)
Giving up execute_task_with_backoff(...) after 3 tries (langfuse.request.APIError: Invalid JSON (400): None)
Giving up execute_task_with_backoff(...) after 3 tries (langfuse.request.APIError: Invalid JSON (400): None)

Another example groq.BadRequestError: Error code: 400 - {'error': {'message': 'messages.1.content: user message cannot be an null or empty', 'type': 'invalid_request_error'}}

Giving up execute_task_with_backoff(...) after 3 tries (langfuse.request.APIError: Invalid JSON (400): None)
Giving up execute_task_with_backoff(...) after 3 tries (langfuse.request.APIError: Invalid JSON (400): None)
Giving up execute_task_with_backoff(...) after 3 tries (langfuse.request.APIError: Invalid JSON (400): None)

To reproduce

See above

SDK and container versions

No response

Additional information

No response

Are you interested to contribute a fix for this bug?

No

[marcklingen]

do you use groq via the openai integration and host [https://api.groq.com/openai/v1](https://api.groq.com/openai/v1)?

[ajram23]

Not sure but I have seen this happen with OpenAI as well.

[tommyhe6]

Can confirm that I receive the same error

Giving up execute_task_with_backoff(...) after 3 tries (langfuse.request.APIError: Invalid JSON (400): None)

when OpenAI errors out too, due to rate limiting.

[marcklingen]

thanks @tommyhe6 @ajram23! we'll have a look

[maxdeichmann]

@tommyhe6 @ajram23 does this happen in the langchain integration?

[ajram23]

@maxdeichmann yes.

[tommyhe6]

@maxdeichmann I'm not using the langchain integration. We have our own function that calls an LLM API that's wrapped in @observe(as_type="generation"). And our outer functions are wrapped in @observe(). Luckily it doesn't seem to affect performance, but the log is pretty annoying since it makes debugging harder and clutters our Sentry.

[adithya-dinesh]

@marcklingen
Similar error we are getting on using langfuse with python
Sharing trace

`Giving up execute_task_with_backoff(...) after 3 tries (langfuse.request.APIError: Invalid JSON (400): None)
error uploading: Invalid JSON (400): None
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/langfuse/request.py", line 104, in _process_response
payload = res.json()
^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/httpx/_models.py", line 756, in json
return jsonlib.loads(self.text, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/json/init.py", line 346, in loads
return _default_decoder.decode(s)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/langfuse/task_manager.py", line 139, in upload
self._upload_batch(batch)
File "/usr/local/lib/python3.11/site-packages/langfuse/task_manager.py", line 166, in _upload_batch
execute_task_with_backoff(batch)
File "/usr/local/lib/python3.11/site-packages/backoff/_sync.py", line 105, in retry
ret = target(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/langfuse/task_manager.py", line 164, in execute_task_with_backoff
return self._client.batch_post(batch=batch, metadata=metadata)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/langfuse/request.py", line 57, in batch_post
return self._process_response(
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/langfuse/request.py", line 108, in _process_response
raise APIError(res.status_code, res.text)
langfuse.request.APIError: Invalid JSON (400): None`