Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support loading external files in OpenAPI #8067

Open
mwaterous opened this issue Dec 17, 2020 · 10 comments · May be fixed by #11076
Open

Support loading external files in OpenAPI #8067

mwaterous opened this issue Dec 17, 2020 · 10 comments · May be fixed by #11076

Comments

@mwaterous
Copy link

Regarding the updates made in 0.74.0: https://gohugo.io/news/0.74.0-relnotes/

The current implementation doesn't appear to allow isExternalRefsAllowed to be configured anywhere and it is false by default. This causes any attempt to unmarshal an openapi spec using external references to fail: https://github.com/getkin/kin-openapi/blob/9b9280d707773bb764ec07ac7363248d8240343d/openapi3/swagger_loader.go#L67

This could be a really fantastic feature for some of us building documentation sites in Hugo based on the OpenAPI spec, but due to the above, it's at present only useful for smaller and non-complex openapi specifications.

Could this either be set to true by default when initializing the kin-openapi library, or could this be designed so that we can set it to true in config.toml or somewhere else that makes sense?

@illeatmyhat
Copy link

My OpenAPI YAMLs are getting pretty big, would be appreciated

@bep bep added this to the v0.83 milestone Apr 6, 2021
@bep
Copy link
Member

bep commented Apr 6, 2021

Question: I assume that you by external (mostly) mean files on the same file system (and not remote files)?

@illeatmyhat
Copy link

As for me, yes on the same file system.
Specifically, I organized all of my YAMLs under the assets/ directory since that's what works with resources.Get

@mwaterous
Copy link
Author

Same for me, all on the same file system.

@bep
Copy link
Member

bep commented Apr 6, 2021

OK,

I hadded the initial OpenAPi support in Hugo, and the project I used it in has the API source mounted as a Hugo Module. So, this is what we need to do:

  1. Set SwaggerLoader.IsExternalRefsAllowed to true
  2. Implement a SwaggerLoader.ReadFromURIFunc that resolves the files in the Assets (only) filesystem (which allows them to live in modules).

@illeatmyhat
Copy link

illeatmyhat commented Apr 7, 2021

I was able to hack something together that works for my specific use-case, but isn't production ready.

There are a number of issues

  1. Relative $refs (e.g. $ref: ./definitions/foobar.yaml#/components/schema/Foobar need to be resolved relative to the current YAML file being parsed, and it isn't clear how we're supposed to figure out the location of the current YAML file with SwaggerLoader.ReadFromURIFunc(). Without this, we will not be able to recursively resolve $refs.
    Turns out ReadFromURIFunc will automatically provide the correct relative URL from the root file.

  2. It isn't clear how we're supposed to determine whether a file is in the Assets filesystem, and then receive a filepath in that filesystem. (e.g. go from ./definitions/foobar.yaml to /workspaces/my-docs/assets/api/definitions/foobar.yaml)
    I attempted to use SourceFilesystem.MakePathRelative() and SourceFilesystem.Path(), and neither worked for me.

loader := kopenapi3.NewSwaggerLoader()
loader.IsExternalRefsAllowed = true
loader.ReadFromURIFunc = func(loader *kopenapi3.SwaggerLoader, url *url.URL) ([]byte, error) {
	if url.Scheme != "" || url.Host != "" || url.RawQuery != "" {
		return nil, errors.Errorf("unsupported URI: %q", url.String())
	}

	fullpath := filepath.Join(filepath.Dir(key), url.Path)
	assetpath := ns.deps.SourceFilesystems.Assets.RealDirs(fullpath)[0]
	return ioutil.ReadFile(assetpath)
}
err = loader.ResolveRefsIn(s, nil)

@bep bep changed the title OpenAPI support should allow IsExternalRefsAllowed to be configurable, or set to true by default Support loading external files in OpenAPI Apr 7, 2021
@bep
Copy link
Member

bep commented Apr 7, 2021

@illeatmyhat It's hard for me to debug your example/code without additional context, but I'll try:

  • I think the closest existing example would be somthing like this:
    resolveImport := func(args api.OnResolveArgs) (api.OnResolveResult, error) {
  • You by definition needs to mount all your API files inside /assets, so if it's not found there, it does not exist.
  • Then, in my head, there are two cases:
    • Either the file includes are root relative (e.g. definitions/foobar.yaml)
    • Or they are relative to the current file (e.g. ./baz.yaml)

There may be discussions about the details above, but the first is simple (use fs.Stat directly), for the second you need to determine the relative directory via MakePathRelative.

@bep bep modified the milestones: v0.83, v0.84 May 1, 2021
@bep bep modified the milestones: v0.84, v0.85 Jun 18, 2021
@bep bep modified the milestones: v0.85, v0.86 Jul 5, 2021
@bep bep modified the milestones: v0.86, v0.87, v0.88 Jul 26, 2021
@bep bep modified the milestones: v0.88, v0.89 Sep 2, 2021
@bep bep removed this from the v0.89 milestone Nov 2, 2021
morysh added a commit to morysh/hugo that referenced this issue Jun 8, 2023
Set IsExternalRefsAllowed to true and provide a ReadFromURIFunc.

Fixes gohugoio#8067
@morysh morysh linked a pull request Jun 8, 2023 that will close this issue
morysh added a commit to morysh/hugo that referenced this issue Jun 9, 2023
Set IsExternalRefsAllowed to true and provide a ReadFromURIFunc.

Fixes gohugoio#8067
morysh added a commit to morysh/hugo that referenced this issue Jun 9, 2023
Set IsExternalRefsAllowed to true and provide a ReadFromURIFunc.

Fixes gohugoio#8067
@bep bep modified the milestones: v0.113.0, v0.115.0 Jun 13, 2023
@bep bep modified the milestones: v0.115.0, v0.116.0 Jun 30, 2023
@bep bep modified the milestones: v0.116.0, v0.117.0 Aug 1, 2023
@bep bep modified the milestones: v0.117.0, v0.118.0 Aug 30, 2023
@bep bep modified the milestones: v0.118.0, v0.119.0 Sep 15, 2023
@bep bep modified the milestones: v0.119.0, v0.120.0 Oct 4, 2023
@bep bep modified the milestones: v0.120.0, v0.121.0 Oct 31, 2023
@bep bep modified the milestones: v0.121.0, v0.122.0 Dec 6, 2023
@bep bep modified the milestones: v0.122.0, v0.123.0, v0.124.0 Jan 27, 2024
@bep bep modified the milestones: v0.124.0, v0.125.0 Mar 4, 2024
@bep bep modified the milestones: v0.125.0, v0.126.0 Apr 23, 2024
@bep bep modified the milestones: v0.126.0, v0.127.0 May 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants