UTF-8 streams might break in current stream implementation

[lazydogP]

I'm using this package in Node.js environment and call cohere.chatStream to generate long Chinese texts. However, The replacement character (�) appears in random places in 'stream-end' event. The following code may convert incomplete UTF-8 chunks, which are yielded in parts by stream, into strings.

cohere-typescript/src/core/streaming-fetcher/Stream.ts

Lines 29 to 32 in 40c146c

	// Buffer is present in Node.js environment
	if (typeof Buffer !== "undefined") {
	bufferChunk += Buffer.isBuffer(chunk) ? chunk : Buffer.from(chunk);
	}

While Latin Basic characters require only 1 byte in UTF-8, other characters, such as CJK characters, need more bytes to encode. This means there's a chance that a character could be split across chunks.

[billytrend-cohere]

Hey @lazydogP many thanks for this interesting find! We have repro'd it and will have a fix for you asap.

[billytrend-cohere]

@lazydogP we're planning to move to SSE to fix this issue for you! thanks for your patience

[danny-avila]

@lazydogP we're planning to move to SSE to fix this issue for you! thanks for your patience

Is this resolved yet?

[danny-avila]

Still happening on the latest version, it seems we can't rely on the stream-end event to avoid the issue:

danny-avila/LibreChat@fe93f3a