Upgrade to Pydantic V2 #650

birdperson1970 · 2023-12-03T09:40:23Z

Upgraded numerous modules comments will be inline to descripe all of the changes and decisions made

netlify · 2023-12-03T09:40:27Z

✅ Deploy Preview for continuedev canceled.

Name	Link
🔨 Latest commit	`69ef211`
🔍 Latest deploy log	https://app.netlify.com/sites/continuedev/deploys/657456149ef3e20008fd038d

birdperson1970 · 2023-12-03T09:42:41Z

server/continuedev/core/autopilot.py

@@ -3,7 +3,8 @@
 from typing import List, Optional, cast

 from aiohttp import ClientPayloadError
-from openai import error as openai_errors
+import openai


uplifted openai v0.27.5 to v1.3.6 which was a breaking change.
As mentioned we'll need to uplift the entire OpenAI framework but it's a lot easier to implement

birdperson1970 · 2023-12-03T09:48:52Z

server/continuedev/core/config.py

    params: Optional[Dict] = {}

    # Allow step class for the migration
-    @validator("step", pre=True, always=True)
+    @field_validator("step")


Pydantic Docs:
validator has been deprecated, and should be replaced with field_validator
The new @field_validator decorator does not have the each_item keyword argument; validators you want to apply to items within a generic container should be added by annotating the type argument. See validators in Annotated metadata for details. This looks like List[Annotated[int, Field(ge=0)]]

https://docs.pydantic.dev/latest/migration/#validator-and-root_validator-are-deprecated

birdperson1970 · 2023-12-03T09:55:27Z

server/continuedev/core/context.py

-    class Config:
-        arbitrary_types_allowed = True
-        exclude = {"ide", "delete_documents", "update_documents"}
+    model_config = ConfigDict(arbitrary_types_allowed=True, exclude={"ide", "delete_documents", "update_documents"})


This was updated by Pydantic's bump-pydantic server run

Pydantic Doc:
The Pydantic V1 behavior to create a class called Config in the namespace of the parent BaseModel subclass is now deprecated.
https://docs.pydantic.dev/latest/migration/#changes-to-config

birdperson1970 · 2023-12-03T09:56:07Z

server/continuedev/core/config.py

@@ -257,10 +258,22 @@ class SerializedContinueConfig(BaseModel):
    @staticmethod
    @contextmanager
    def edit_config():
-        config = SerializedContinueConfig.parse_file(CONFIG_JSON_PATH)
+        # Read the JSON file and parse it into a dictionary


Pydantic Docs:
In particular, parse_raw and parse_file are now deprecated. In Pydantic V2, model_validate_json works like parse_raw. Otherwise, you should load the data and then pass it to model_validate.

https://docs.pydantic.dev/latest/migration/#changes-to-pydanticbasemodel

birdperson1970 · 2023-12-03T09:59:26Z

server/continuedev/core/main.py

-
-    class Config:
-        smart_union = True
+    # TODO[pydantic]: The following keys were removed: `smart_union`.


Says smart_union is deprecated but no suggestion on how to migrate it

https://docs.pydantic.dev/latest/migration/#changes-to-config

birdperson1970 · 2023-12-03T10:00:50Z

server/continuedev/core/main.py

-
-    class Config:
-        copy_on_model_validation = False
+    # TODO[pydantic]: The following keys were removed: `copy_on_model_validation`.


copy_on_model_validation deprecated but no replacment

https://docs.pydantic.dev/latest/migration/#changes-to-config

birdperson1970 · 2023-12-03T10:03:35Z

server/continuedev/core/models.py

    )
-    def roles_not_none(cls, v, values):
+    def roles_not_none(cls, v, val_info):


value: ModelField is now ValidationInfo. I renamed the argument from val_info to make it clear it is not value.

birdperson1970 · 2023-12-03T10:04:21Z

server/continuedev/core/models.py

        if v is None:
-            return values["default"]
+            return cls.model_fields[val_info.field_name].default


There is no default in the val_info so it now needs to be fetched from the class.model_fields...

birdperson1970 · 2023-12-03T10:05:40Z

server/continuedev/headless/headless_ide.py

@@ -31,11 +31,13 @@ class LocalIdeProtocol(AbstractIdeProtocolServer):
    workspace_directory: str = os.getcwd()
    unique_id: str = get_mac_address()

-    filesystem: FileSystem = RealFileSystem()
+    filesystem: FileSystem


Its now fussy about not parsing in an arguement so this needs to be instantiated in the init

birdperson1970 · 2023-12-03T10:07:09Z

server/continuedev/libs/llm/base.py

-    @validator("prompt_templates", pre=True, always=True)
-    def set_prompt_templates(cls, prompt_templates, values):
-        return prompt_templates or autodetect_prompt_templates(values["model"])
+        # TODO Pydantic V2 no longer supports fields it should be done inline


Bit Confused here becuase the fields exist in the parent and child Config class which has been deprecated. Happy to discuss about what you want to do here.

birdperson1970 · 2023-12-03T10:10:36Z

server/continuedev/models/llm.py

-        "*",
-        pre=True,
-        always=True,
+    @field_validator(


Not sure how to handle this. Each fields now needs validate_default=True and to be Annotated. But I'm unsure which fields they are

birdperson1970 · 2023-12-03T10:11:38Z

server/continuedev/plugins/context_providers/diff.py

-    display_title = "Diff"
-    description = "Output of 'git diff' in current repo"
-    dynamic = True
+    title: str = "diff"


Pydantic2 is fussy and everything needs a type now

server/requirements.txt

birdperson1970 · 2023-12-03T10:16:44Z

server/requirements.txt

 boltons==23.0.0
-pydantic==1.10.8
+pydantic==2.0.2


This is what I came here for!

birdperson1970 · 2023-12-03T10:20:45Z

server/requirements.txt

-fastapi==0.95.2
-typer==0.9.0
-openai==0.27.5
+openai==1.3.6


I've re-ordered this file so it aligns with the pyproject.toml

birdperson1970 · 2023-12-06T22:41:52Z

server/continuedev/models/generate_json_schema.py

@@ -33,12 +33,12 @@
        ContextProviderDescription,
    ]
    + [SerializedContinueConfig, ModelDescription]
-    + [ModelProvider, ModelName]
+    + [ModelNameWrapper]


Pydantic Doesn't support Literals Directly
pydantic/bump-pydantic#124
So we need to create a class wrapper which is then referenced in modelData.tx.
import { Provider } from "../schema/ModelDescription";
import { ModelName } from "../schema/ModelNameWrapper";

birdperson1970 · 2023-12-09T08:56:07Z

server/continuedev/core/main.py

+    # NOTE [pydantic]: with Pydantic Unions don't use relative imports (i.e. `from .main import DeltaStep`) otherwise it will fail to match throwing:
+    #   Input should be a valid dictionary or instance of DeltaStep ....
+    # The string class names will differ e.g.:
+    # <class 'continuedev.core.main.DeltaStep'> != <class 'server.continuedev.core.main.DeltaStep'>
    update: "UpdateStep"


# NOTE [pydantic]: with Pydantic Unions don't use relative imports (i.e. `from .main import DeltaStep`) otherwise it will fail to match throwing: # Input should be a valid dictionary or instance of DeltaStep .... # The string class names will differ e.g.: # <class 'continuedev.core.main.DeltaStep'> != <class 'server.continuedev.core.main.DeltaStep'>

birdperson1970 added 6 commits December 2, 2023 09:53

Som broken But I need github perm links

62422f0

clean compile

6ea256a

can't find llm

5af454d

Merge remote-tracking branch 'origin/main' into pydantic_2_upgrade

914137b

Wroking

2d137bb

final tweaks for review

7b8528e

birdperson1970 commented Dec 3, 2023

View reviewed changes

server/requirements.txt Outdated Show resolved Hide resolved

birdperson1970 commented Dec 3, 2023

View reviewed changes

birdperson1970 added 5 commits December 3, 2023 21:31

fixed python-lsp-server[websockets]

15aa9f7

pulling back the version for WSL compatability

2f3bad0

module version problem reverting

cffbf2a

migrated schema generation to Pydantic V2

9a1002d

fixed chroma db version issue

82c0333

birdperson1970 commented Dec 6, 2023

View reviewed changes

fixed aweful Union serialization bug

ae3610f

birdperson1970 commented Dec 9, 2023

View reviewed changes

fixed templates to validation

69ef211

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade to Pydantic V2 #650

Upgrade to Pydantic V2 #650

birdperson1970 commented Dec 3, 2023

netlify bot commented Dec 3, 2023 •

edited

birdperson1970 Dec 3, 2023 •

edited

birdperson1970 Dec 3, 2023

birdperson1970 Dec 3, 2023

birdperson1970 Dec 3, 2023

birdperson1970 Dec 3, 2023

birdperson1970 Dec 3, 2023

birdperson1970 Dec 3, 2023

birdperson1970 Dec 3, 2023

birdperson1970 Dec 3, 2023

birdperson1970 Dec 3, 2023

birdperson1970 Dec 3, 2023

birdperson1970 Dec 3, 2023 •

edited

birdperson1970 Dec 3, 2023 •

edited

birdperson1970 Dec 3, 2023

birdperson1970 Dec 6, 2023

birdperson1970 Dec 9, 2023

Upgrade to Pydantic V2 #650

Are you sure you want to change the base?

Upgrade to Pydantic V2 #650

Conversation

birdperson1970 commented Dec 3, 2023

netlify bot commented Dec 3, 2023 • edited

✅ Deploy Preview for continuedev canceled.

birdperson1970 Dec 3, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

birdperson1970 Dec 3, 2023 • edited

Choose a reason for hiding this comment

birdperson1970 Dec 3, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

netlify bot commented Dec 3, 2023 •

edited

birdperson1970 Dec 3, 2023 •

edited

birdperson1970 Dec 3, 2023 •

edited

birdperson1970 Dec 3, 2023 •

edited