[DESKTOP]: When navigating away from streaming chat - the chat should be retained

[atljoseph]

How are you running AnythingLLM?

AnythingLLM desktop app

What happened?

Open a new workspace.
Start a new chat.
Do not wait on the model’s response to complete.
Visit the settings page for either the workspace or API.
Come back to the thread you were on.
The last prompt sent is saved in the chat history.
The response from the model was never saved.
I know it was doing something because I saw the initial part of the response.
I know that it is still doing something because nvtop still shows my gpus cranking on something… They only do that when model is responding.
Going to the settings page should not prevent the model’s response from being saved into the chat history in the backend.
When I get back to the thread, it should just load what the backend has in its data store.

This issue makes me doubt the efficacy of this as a self-hostable solution at an enterprise scale, as it makes me doubt that it can handle high concurrency rates.

Please educate me, and/or fix this issue.

Thank you

Are there known steps to reproduce?

This is listed above in the description.

[timothycarambat]

Not saving a response by closing the connection client-side during streaming would have zero impact, or indication, of if the system can handle concurrency. So I'm unsure how you draw that conclusion. That being said, depending on the LLM provider you are using, most don't even have a way to "force-stop" a streaming response. So while we can close the client connection, we cant tell your provider to stop working on the response - which is why you saw it still working despite no completion saved.

I did some testing for replication and this only occurs on the desktop app. On the docker version, the streaming will continue and will be saved. Even if you close the browser, so it works as one would expect.

However, i am a bit confused by your commentary on the issue. The Desktop app is for a single-player isolated use of AnythingLLM. There is no "user/chat concurrency" or anything to that effect. So while this may be not ideal I do also think you may be misunderstanding the intention of the desktop app and who it is for.

The Dockerized version supports multiple users, concurrent chats, and to the point of this issue - behaves as you want it. It sounds from the recent comments on other issues that are you testing AnythingLLM desktop for assessment of "self-hostable solution at an enterprise scale" when the desktop app is not the medium in which that service is delivered.

https://docs.useanything.com/desktop-vs-docker

However, agreed that the chat on the desktop not continuing to stream and save is indeed not ideal and should be resolved.

[atljoseph]

Have not tried the desktop app since this had an update. I did try to install the doctor service today and things broke right from the start after on boarding.

Can’t succinctly articulate the feeling I indicated above about concurrency, except that: Not retaining the streaming of a response in the background while someone edits a setting could cause them to go ask the question again when they come back and don’t see it any longer. Could lead to resource overusage in a given amount of time, or bumping a request limit. While that doesn’t matter on my computer, for me, it can matter at scale, in k8s, for many concurrent users.

I was never able to go back at any point and view the response that was going while the settings were being edited. It was like it never happened. I’d expect at least to be able to go back and view the response later after completion, but it vanished into thin air.

Was planning to see how that worked on the website/docker service today :/ but that won’t work either :/