Accessing Search Candidates after Content Retrieval

[robmartin-scibite]

We are assessing if we can use LangChain4J for our current RAG use case. We currently retrieve candidates from our data store, extract relevant information and send to an LLM to generate an answer. We then need to cross reference the LLM answer with the original search candidates and return both answer and candidates to the end user. Im struggling to achieve this in LangChain4J as RetrievalAugmentor returns only a new UserMessage so the search results are no longer in scope for any down stream processes in the chain. Any help would be greatly appreciated.

[robmartin-scibite]

Just come across #199. Perhaps a listener could be used to capture the candidate results during the retrieval phase and then access them later on in the process?

[langchain4j]

Hi @robmartin-scibite we are working on this feature right now, it should be released quite soon.

Before this feature is released, you could temporarily use the hack from this discussion: #660

[robmartin-scibite]

Thanks for the prompt reply! The new feature looks great. The only issue I can see regarding our use case is that is appears WithSources stores the Content sent to the LLM rather than the original data retrieved from a data store. In our case we dont send all the data to the LLM, but we do enrich the results that comes back from the LLM with the candidate data.

[langchain4j]

Could you please elaborate? WithSources (or whatever it will be called) will contain a list of Content that has been retrieved using ContentRetreiver(s) and then aggregated/reranked. So it will contain all the Content that the LLM has "seen". Do you mean you want to also get access to all the Contents that were retreived, but filtered out and not sent to the LLM?

[robmartin-scibite]

Yeh sure, when we retrieve candidates documents, we retrieve the text plus a lot of metadata, for example authors, publish dates, etc. This can be a considerable amount of metadata and not relevant for the LLM to answer the question, so we send only the text data and not the meta data. After the LLM has returned the answer we want to return all the candidate data back to the user, both the textual data used for the LLM call and all the additional metadata. Would the solution be to extend TextContent so a new class can store all the additional data, and only the result of text() get sent to the LLM. I can then cast back to my extended type from WithSources

[langchain4j]

If you are talking about metadata stored in TextSegment.metadata(), it will be returned to you inside WithSources.contents().get(x).textSegment().metadata().

It is true that DefaultContentInjector by default does not include metadata into the prompt, but it does not mean that this information is not accessible to the developer.

So, all data from TextSegment (both text and metadata) will be accessible for you from WithSources. Does this fit your use case?

[robmartin-scibite]

oh yes, that makes sense. I look forward to experimenting with the new feature. Thanks for your help!