[FEATURE] Support for chunking strategies

[orpiske]

Is your feature request related to a problem? Please describe.

One of the lessons we learned from a project we worked recently was that there doesn't seem to be great/widespread support for Chunking in Java. We were particularly looking for support for different chunking strategies. That could have helped us maximize our ability to store, retrieve and match data in our VectorDB.

Describe the solution you'd like

We would like to discuss with the Langchain4j community whether they having support for chunking feasible within this project and aligned with the project goals and feature set.

Describe alternatives you've considered

Among other things, we have considered creating a chunking library as a separate project, but we believe that adding a chunking library as part of Langchain4j would result in a better developer experience and would also allow the project to, more easily, implement chunking strategies that would involve LLM-based chunking.

Additional context

If the community believes that this is in line with the project, we are motivated to contribute and help maintain this feature.

[langchain4j]

HI @orpiske this sounds great! We are looking forward to improve this in the LC4J, so any contributions are welcome!
Apart from existing document splitters we plan to add markdown splitter and semantic splitter in the near future.

What chunking strategies do you have in mind?

[orpiske]

It's great news that you have plans for Markdown (and/or AsciiDoc) and semantic splitter! Those would have been very useful for our project.

In general, specialized chunkers/splitters (and/or an interface for implementing those) could be particularly helpful (i.e.: so we could deal with YAML, XML, etc) for a subset of our data.

I also think that it API/services/LLM-based chucking, where we defer the chunking to an external service, could be useful.

[langchain4j]

The interface you are looking for is DocumentSplitter, please take a look at it and it's existing implementations

[orpiske]

The interface you are looking for is DocumentSplitter, please take a look at it and it's existing implementations

Noted, thanks!