Import modules/objects directly from v1 namespace

[exs-dwoodward]

Change Summary

Following on from this PR about adding a v1.py namespace to pydantic to better follow the v2 syntax of importing v1 objects, imports such as:

from pydantic.fields import ModelField
from pydantic.generics import GenericModel

or:

from pydantic import fields
from pydantic import generics

cannot be directly swapped for:

from pydantic.v1.fields import ModelField
from pydantic.v1.generics import GenericModel

or:

from pydantic.v1 import fields
from pydantic.v1 import generics

imports. If this was the case, this would make unpinning from v1 --> v2 much simpler as a simple find and replace would be all that is needed to convert from pydantic --> from pydantic.v1 instead (ofc still using v1 types and models).

One caveat with this approach is if this happens:

from pydantic.fields import ModelField
from pydantic.v1.fields import ModelField as ModelFieldFromV1

ModelField is ModelFieldFromV1 # False

but ideally the import syntax should be changed across a codebase rather than only in some places. Note that anything in the __init__.py pydantic namespace is still imported wholesale:

from pydantic import dataclasses
from pydantic.v1 import v1_dataclasses

dataclasses is v1_dataclasses # True

since these modules are imported directly from the underlying submodules. This also means that for the former import block, the ModelField in this case is imported twice.

Additionally, things like Pylance can't directly resolve the import path for pydantic.v1.fields imports using this syntax, but ideally that will be very transient since as soon as the find and replace is executed, any pydantic<2 pins should be able to be removed wholesale and the import will be resolved in pydantic v2.

Checklist

• The pull request title is a good summary of the changes - it will be used in the changelog
• Unit tests for the changes exist
• Tests pass on CI
• Documentation reflects the changes where applicable
• My PR is ready to review, please add a comment including the phrase "please review" to assign reviewers

Selected reviewer: @sydney-runkle

skip change file check

Fix #9357

Selected Reviewer: @PrettyWood

[exs-dwoodward]

This PR is ready for reviewing, please let me know if there are any other steps I can help out with here! Please review :)

[exs-dwoodward]

@PrettyWood is there anything else I need to do on my end on this PR? This is my first PR into pydantic so apologies if I've missed an automation somewhere!

[sydney-runkle]

@exs-dwoodward,

Great, thanks so much for your work! Apologies for the review delay - was focusing on FastUI for a bit. You've done everything perfectly here!

[sydney-runkle]

Great, thanks! We can release this with 1.10.16!

[sydney-runkle]

Working on getting the pipeline all green, will merge when that's done!

[sydney-runkle]

@exs-dwoodward,

Hmph, looks like tests are failing in some cases:

_______________________ test_can_import_modules_from_v1 ________________________
  
      def test_can_import_modules_from_v1() -> None:
          """That imports from any module in pydantic can be imported through
          ``pydantic.v1.<module>``"""
          for module_fname in os.listdir(pydantic.__path__[0]):
              if module_fname.startswith('_') or not module_fname.endswith('.py') or module_fname == 'v1.py':
                  continue
              module_name = module_fname[:-3]
      
  >           _ = importlib.import_module(f'pydantic.v1.{module_name}')
  
  /Users/runner/work/pydantic/pydantic/tests/test_v1.py:27: 
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
  /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/importlib/__init__.py:127: in import_module
      return _bootstrap._gcd_import(name[level:], package, level)
  <frozen importlib._bootstrap>:1006: in _gcd_import
      ???
  <frozen importlib._bootstrap>:983: in _find_and_load
      ???
  <frozen importlib._bootstrap>:967: in _find_and_load_unlocked
      ???
  <frozen importlib._bootstrap>:677: in _load_unlocked
      ???
  <frozen importlib._bootstrap_external>:728: in exec_module
      ???
  <frozen importlib._bootstrap>:219: in _call_with_frames_removed
      ???
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
  
      import sys
      from configparser import ConfigParser
      from typing import Any, Callable, Dict, List, Optional, Set, Tuple, Type as TypingType, Union
      
  >   from mypy.errorcodes import ErrorCode
  E   ModuleNotFoundError: No module named 'mypy'
  
  /Users/runner/work/pydantic/pydantic/pydantic/mypy.py:5: ModuleNotFoundError
  =========================== short test summary info ============================
  FAILED ../../../../../../../../../Users/runner/work/pydantic/pydantic/tests/test_v1.py::test_can_import_modules_from_v1

[sydney-runkle]

And another case:

https://github.com/pydantic/pydantic/actions/runs/8900758459/job/24443047366?pr=9162

[exs-dwoodward]

Thanks @sydney-runkle for looking through it!

hmm weird, running locally all my tests pass, though perhaps I should exclude the mypy module from being imported directly in the import test, given that mypy is an optional install.

I get the feeling that the ImportWarning's on the compiled linux tests seem to be due to the package spec not matching. I wonder if this is due to the relative imports used in pydantic (though I wouldn't want to make a massive change here). An alternative could be to employ a module level __getattr__ that imports the module asked for, but this feels quite heavy handed - wdyt?

[exs-dwoodward]

oh actually nevermind the __getattr__ can be pretty simple:

import pydantic

def __getattr__(name: str) -> Any:
     return getattr(pydantic, name)

seems to do the trick.

[sydney-runkle]

@exs-dwoodward,

Hmph, looks like we're still getting some test failures. I'd be ok with a more minimal approach here - not testing mypy or dotenv. Also, could you please fix the linting issue?

Thanks!

[exs-dwoodward]

I'm not entirely sure there is a nice fix for the linux compiled tests to be honest - not one that I can think of at least :( (that doesn't involve converting all the relative imports in pydantic to absolute) - given it only raises an ImportWarning - and that this 1.10.16 release would only be a temporary bound so that people can easily swap to pydantic 2 type imports, how worried are we about these warning failures? They only seem to crop up after the v1 tests are being run, so I'm wondering if swapping the test order would just magically make them disappear anyway.

[sydney-runkle]

Also, I think we can revert the commit that added the absolute imports 👍

[exs-dwoodward]

not too sure what changes need to be made to ci.yml - is there a commit on main you could point me towards? Also I'm happy to revert the absolute imports commit if its tolerated in the CI - if not I think its best to keep it.

[sydney-runkle]

@exs-dwoodward,

Try using the ci.yml file from main so that there's no diff and reverting the import changes...

[exs-dwoodward]

@exs-dwoodward,

Try using the ci.yml file from main so that there's no diff

I believe thats what I've done here (the diff comes from the fact this PR points to pydantic:1.10.X-fixes not main)?

[sydney-runkle]

Ah yes sorry, let's pull whatever is on the 1.10.X fixes branch then

[sydney-runkle]

Ahhh ok I forgot that this was the PR where I made the CI changes - I'll play around with it later today and try to fix things!

Could you please revert the relative import changes?

[exs-dwoodward]

I would but I'm afraid that doing so might cause the linux-compiled tests to fail 😅 whereas swapping the relative --> absolute imports causes all the tests to pass (apart from the Windows ones)

[sydney-runkle]

Some context for my CI change, btw: https://github.com/pydantic/pydantic/pull/9305/files

I would but I'm afraid that doing so might cause the linux-compiled tests to fail 😅

Hmm I see what you mean... I'll take a closer look.

[sydney-runkle]

Also just as a side note - we can get this PR into v1.10.X-fixes, but this will probably be the last pydantic 1.10.X release, as we put a June 1st 2024 end date on maintenance 🚀.

[sydney-runkle]

I pinged @hramezani re these test failures, let's see what he thinks :).

[sydney-runkle]

@exs-dwoodward,

Hmm, so I got most of the tests passing. Let's revert those import changes and see if we can get the linux build tests passing :).

[sydney-runkle]

To your point, with the relative imports and the cibuildwheel pin: https://github.com/pydantic/pydantic/actions/runs/9351505506

[exs-dwoodward]

Yeah - looks like it might need the absolute imports - are you happy for me to revert the revert and use absolute imports?

[sydney-runkle]

@exs-dwoodward I'd like to get confirmation / explanation from @davidhewitt or @hramezani first, but I'll send an update soon!

[davidhewitt]

I don't have an immediate answer for what Cython might be doing to the code to break things when compiled; is it possible that the new __getattr__ implementation has the effect of making the wrong __package__ and __spec__ in that module?

[exs-dwoodward]

I believe that is exactly the issue - which appears to be 'fixed' by using absolute rather than relative imports.

If all the tests pass with absolute rather than relative imports - is there a particular reason other than caution to stick with relative?

[davidhewitt]

I believe that is exactly the issue - which appears to be 'fixed' by using absolute rather than relative imports.

I guess my worry is that if the relative imports broke things in a way we don't understand and the absolute imports will fixes them, is that just because stuff is still broken but we don't happen to be triggering the unhappy case?

What happens if we modify __getattr__ to exclude everything which starts with __ (maybe except for __all__ and __doc__)?

[exs-dwoodward]

Thats my only worry as I do not know exactly what cases pydantics CI tests cover - I can give that a go, see if it helps!

[exs-dwoodward]

I'm not too certain it will make a difference - since the modules / subpackages will still be importable, and it looks like their __package__ != __spec__.parent I believe, so this would stay the same. I'll try a commit anyways!

[davidhewitt]

Well the new error message seems interesting 👀

In PyO3 I think we've worked around similar errors by adding to sys.modules manually. E.g. sys.modules['pydantic.v1.fields'] = pydantic.fields.

Is this a good idea? I have no idea 😂

[davidhewitt]

Yeah so it looks like this hasn't really helped and is creating a lot of pain. Is the conclusion that the absolute imports was the only working option then? Maybe we just go with that and hope for the best...

[exs-dwoodward]

Okay nice - looks like the CI is all passing with the absolute imports in place now! 🚀

[sydney-runkle]

@exs-dwoodward,

Great work! We'll include this in the next v1.10.X patch release, then be done with V1 changes :). Thanks for all of your help (and patience) here!

Ping me anytime if you're looking for issues to contribute to, we really appreciate your work!