Split package problem when using java modules

[hendrikebbers]

langchain4j can not be used in a Java project that is working on the module path. By doing so the following error happens:

error: the unnamed module reads package dev.langchain4j.retriever from both langchain4j and langchain4j.core
That is based on having the same package dev.langchain4j.retriever (with content) in both modules langchain4j and langchain4j.core.

A project like langchain4j should be 100% compatible with the Java module system since that is used by more and more libs over time.

I did a lot of work in that area and happy to help if you want to work on that topic.

[dosubot]

To address the split package issue you're encountering with langchain4j and langchain4j.core in a Java module system, here are some targeted solutions:

1. Refactor the Package Structure: Adjust the package names in either langchain4j or langchain4j.core to eliminate the overlap. This is the most direct way to resolve the conflict.

2. Use Services and Service Loaders: For shared interfaces or classes, leverage Java's ServiceLoader to allow one module to use implementations defined in another without sharing a package name.

3. Merge Modules: If feasible and the modules are closely related, consider combining them into a single module to simplify the structure.

4. Use Automatic Modules: As a temporary workaround, you could place the conflicting JARs on the classpath, treating them as automatic modules, though this bypasses some module system benefits.

Given your project's Maven structure, it might be most straightforward to refactor the langchain4j module, especially if langchain4j-core serves as a foundational library for other modules or projects.

Additionally, there's an open issue related to Java module system support ([FEATURE] Support Java9+ module system) and a closed issue that dealt with a similar problem by moving EmbeddingStoreIngestor from langchain4j to langchain4j-core in version 0.25.0. These might offer further insights or avenues for resolution.

If you need more detailed advice or further clarification, don't hesitate to ask.

Details

Sources

• [FEATURE] Support Java9+ module system
• make it module friendly [BUG]
• langchain4j/pom.xml
• langchain4j-core/src/test/java/dev/langchain4j/rag/DefaultRetrievalAugmentorTest.java
• langchain4j-core/src/main/java/dev/langchain4j/spi/ServiceHelper.java
• langchain4j-core/src/test/java/dev/langchain4j/spi/ServiceHelperTest.java
• langchain4j-neo4j/pom.xml
• langchain4j-core/src/main/java/dev/langchain4j/rag/content/retriever/ContentRetriever.java
• langchain4j-core/src/main/java/dev/langchain4j/rag/content/retriever/WebSearchContentRetriever.java

About Dosu
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.

[koppor]

In our project, we used the hints by Nicolai Parlog to have a workaround:

+        '--add-exports=langchain4j/dev.langchain4j.model.chat=org.jabref',
+        '--add-exports=langchain4j/dev.langchain4j.model.chat=org.jabref.merged.module',
...
+modularity.patchModule("langchain4j", "langchain4j-core-0.28.0.jar")
+modularity.patchModule("langchain4j", "langchain4j-embeddings-0.28.0.jar")
+modularity.patchModule("langchain4j", "langchain4j-embeddings-all-minilm-l6-v2-0.28.0.jar")

[langchain4j]

Hi @hendrikebbers, thanks a lot for reporting!

Since dev.langchain4j.retriever.Retriever is deprecated since 0.26, we could get rid of it in the next (0.31) release.
Do you see any other way around it?
Thanks!

[koppor]

There are many more issues with "split pacakges":

[koppor]

• dev.langchain4j.agent.tool from both langchain4j and langchain4j.core
• dev.langchain4j.classification from both langchain4j and langchain4j.core
• dev.langchain4j.chain from both langchain4j and langchain4j.core
• dev.langchain4j.code from both langchain4j and langchain4j.core
• dev.langchain4j.model.embedding from both langchain4j.embeddings.all.minilm.l6.v2 and langchain4j.embeddings
• dev.langchain4j.model.output from both langchain4j and langchain4j.core

[langchain4j]

Oh :(

[hendrikebbers]

I assume we have two problems here:

• langchain4j and langchain4j.core have a lot of packages in common. Here it would be great to introduce a custom package for one of them. Example: all content of langchain4j will be under the package dev.langchain4.impl.* in future. By doing so you would have the interface dev.langchain4.impl.retriever.Retriever in langchain4j and the classdev.langchain4.retriever.EmbeddingStoreRetriever in langchain4j.core. Is my assumption (api in langchain4j.core and implementations inlangchain4j) correct?
• langchain4j.embeddings shares packages with core (dev.langchain4j.model.embedding). Maybe all content in the embeddings jars can be moved to dev.langchain4j.embedding.*?

[konczdev]

This is also a problem in osgi environment. The preferred way to wire bundles are import/export packages but this is not working with split packages. The solution is require bundle in the manifest, but it is tighter coupling. I think either langchain and langchain-core should be merged or the package names should be changed. As I see the most recent issues, there will be breaking changes in the near future, so maybe this is the time when it is possible to done.

[hendrikebbers]

@langchain4j if you want / need help in that refactoring or just for review please ping me. Happy to help

[langchain4j]

Hi @hendrikebbers, langchain4j-core now has APIs and some implementaitons useful for many use cases. I was considering extracting API into langchain4j-api or something like that. So it might look like this: langchain4j -> langchain4j-core -> langchain4j-api.
I need a bit more time to analze where breaking changes are acceptable and where it would be nicer to avoid it.

[langchain4j]

@konczdev could you please elaborate on breaking changes in the near future? We usually try to avoid breaking changes as much as possible.

[konczdev]

A few examples: #1042 #1044 #1048 #1049 #1050

[konczdev]

Hi @hendrikebbers, langchain4j-core now has APIs and some implementaitons useful for many use cases. I was considering extracting API into langchain4j-api or something like that. So it might look like this: langchain4j -> langchain4j-core -> langchain4j-api. I need a bit more time to analze where breaking changes are acceptable and where it would be nicer to avoid it.

I think it's already confusing what the role of the langchain4j and langchain4j-core modules are. Whatever wil be the solution, each module should have its own package namespace and the api and implementation modules should not share the same package.

[langchain4j]

@konczdev #1042 and #1044 are not supposed to be breaking changes. #1048 and #1049 are, but this functionality is not widely used AFAIK, so it is OKish. #1050 is not decided yet how to proceed.

But I agree that there should be no split-packages. Let's see how to fix this problem with as less breaking changes as possible.

[konczdev]

@langchain4j For me those are breaking changes too because I have to wrap non-OSGi dependencies into OSGi bundles. Maybe after the resolution of this issue I will make some contribution to generate the proper OSGi manifests at compile time.

[konczdev]

@langchain4j Don't get me wrong, I didn't mean to criticize anything :) langchain4j is a wonderful project, I follow its development day by day.

[langchain4j]

@konczdev all good, constructive criticism is always welcome! And you are providing very valuable feedback, so thank you! I will have a bit more time next week to look into this problem more closely, so we can continue this discussion then, ok?

My only concern is to avoid introducing unnecessary breaking changes as much as possible, but of course some breaking changes will have to be done sooner or later, so let's do that sooner. I just need a bit more time to analyze it.

[hendrikebbers]

Sounds reasonable! It is better to have a set of breaking changes in 1 release than to have new breaking changes in every release.