Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

documentation bug at transform->mapping: https://pgsync.com/tutorial/transform-nodes/ #538

Open
MeRuslan opened this issue May 6, 2024 · 3 comments
Open

documentation bug at transform->mapping: https://pgsync.com/tutorial/transform-nodes/ #538

MeRuslan opened this issue May 6, 2024 · 3 comments

Comments

@MeRuslan
Copy link
Contributor

MeRuslan commented May 6, 2024 •
edited

PGSync version:
3.1.0

Postgres version:
PostgreSQL 13.12

Elasticsearch/OpenSearch version:
elasticsearch:8.10.2

Redis version:
NA

Python version:
3.9

Problem Description:
Minimal example schema:

[
    {
        "database": "postgres",
        "index": "user_index",
        "nodes": {
            "table": "user",
            "columns": [
                "id"
            ],
            "transform": {
                "mapping": {
                    "id":  "long"
                }
            }
        }
    }
]

Works well without transform->mapping

Error Message (if any):

2024-05-06 18:51:06 .env found!
2024-05-06 18:51:06 localhost:9432
2024-05-06 18:51:06 localhost:9201
2024-05-06 18:51:06 localhost:6389
2024-05-06 18:51:08  - public.user
2024-05-06 18:51:09  - public.user
2024-05-06 18:51:09  0:00:00.252404 (0.25 sec)
2024-05-06 18:51:06 wait-for-it.sh: waiting 30 seconds for localhost:9432
2024-05-06 18:51:06 wait-for-it.sh: localhost:9432 is available after 0 seconds
2024-05-06 18:51:06 wait-for-it.sh: waiting 30 seconds for localhost:9201
2024-05-06 18:51:06 wait-for-it.sh: localhost:9201 is available after 0 seconds
2024-05-06 18:51:06 wait-for-it.sh: waiting 30 seconds for localhost:6389
2024-05-06 18:51:06 wait-for-it.sh: localhost:6389 is available after 0 seconds
2024-05-06 18:51:08 Traceback (most recent call last):
2024-05-06 18:51:08   File "/app/.venv/bin/bootstrap", line 75, in <module>
2024-05-06 18:51:08     main()
2024-05-06 18:51:08   File "/app/.venv/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
2024-05-06 18:51:08     return self.main(*args, **kwargs)
2024-05-06 18:51:08            ^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-05-06 18:51:08   File "/app/.venv/lib/python3.11/site-packages/click/core.py", line 1078, in main
2024-05-06 18:51:08     rv = self.invoke(ctx)
2024-05-06 18:51:08          ^^^^^^^^^^^^^^^^
2024-05-06 18:51:08   File "/app/.venv/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
2024-05-06 18:51:08     return ctx.invoke(self.callback, **ctx.params)
2024-05-06 18:51:08            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-05-06 18:51:08   File "/app/.venv/lib/python3.11/site-packages/click/core.py", line 783, in invoke
2024-05-06 18:51:08     return __callback(*args, **kwargs)
2024-05-06 18:51:08            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-05-06 18:51:08   File "/app/.venv/bin/bootstrap", line 60, in main
2024-05-06 18:51:08     sync: Sync = Sync(
2024-05-06 18:51:08                  ^^^^^
2024-05-06 18:51:08   File "/app/.venv/lib/python3.11/site-packages/pgsync/singleton.py", line 36, in __call__
2024-05-06 18:51:08     cls._instances[key] = super(Singleton, cls).__call__(
2024-05-06 18:51:08                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-05-06 18:51:08   File "/app/.venv/lib/python3.11/site-packages/pgsync/sync.py", line 105, in __init__
2024-05-06 18:51:08     self.create_setting()
2024-05-06 18:51:08   File "/app/.venv/lib/python3.11/site-packages/pgsync/sync.py", line 252, in create_setting
2024-05-06 18:51:08     self.search_client._create_setting(
2024-05-06 18:51:08   File "/app/.venv/lib/python3.11/site-packages/pgsync/search_client.py", line 277, in _create_setting
2024-05-06 18:51:08     mapping = self._build_mapping(tree, routing)
2024-05-06 18:51:08               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-05-06 18:51:08   File "/app/.venv/lib/python3.11/site-packages/pgsync/search_client.py", line 309, in _build_mapping
2024-05-06 18:51:08     column_type: str = mapping[column]["type"]
2024-05-06 18:51:08                        ~~~~~~~~~~~~~~~^^^^^^^^
2024-05-06 18:51:08 TypeError: string indices must be integers, not 'str'
2024-05-06 18:51:09 Traceback (most recent call last):
2024-05-06 18:51:09   File "/app/.venv/bin/pgsync", line 7, in <module>
2024-05-06 18:51:09     sync.main()
2024-05-06 18:51:09   File "/app/.venv/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
2024-05-06 18:51:09     return self.main(*args, **kwargs)
2024-05-06 18:51:09            ^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-05-06 18:51:09   File "/app/.venv/lib/python3.11/site-packages/click/core.py", line 1078, in main
2024-05-06 18:51:09     rv = self.invoke(ctx)
2024-05-06 18:51:09          ^^^^^^^^^^^^^^^^
2024-05-06 18:51:09   File "/app/.venv/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
2024-05-06 18:51:09     return ctx.invoke(self.callback, **ctx.params)
2024-05-06 18:51:09            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-05-06 18:51:09   File "/app/.venv/lib/python3.11/site-packages/click/core.py", line 783, in invoke
2024-05-06 18:51:09     return __callback(*args, **kwargs)
2024-05-06 18:51:09            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-05-06 18:51:09   File "/app/.venv/lib/python3.11/site-packages/pgsync/sync.py", line 1480, in main
2024-05-06 18:51:09     sync: Sync = Sync(
2024-05-06 18:51:09                  ^^^^^
2024-05-06 18:51:09   File "/app/.venv/lib/python3.11/site-packages/pgsync/singleton.py", line 36, in __call__
2024-05-06 18:51:09     cls._instances[key] = super(Singleton, cls).__call__(
2024-05-06 18:51:09                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-05-06 18:51:09   File "/app/.venv/lib/python3.11/site-packages/pgsync/sync.py", line 105, in __init__
2024-05-06 18:51:09     self.create_setting()
2024-05-06 18:51:09   File "/app/.venv/lib/python3.11/site-packages/pgsync/sync.py", line 252, in create_setting
2024-05-06 18:51:09     self.search_client._create_setting(
2024-05-06 18:51:09   File "/app/.venv/lib/python3.11/site-packages/pgsync/search_client.py", line 277, in _create_setting
2024-05-06 18:51:09     mapping = self._build_mapping(tree, routing)
2024-05-06 18:51:09               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-05-06 18:51:09   File "/app/.venv/lib/python3.11/site-packages/pgsync/search_client.py", line 309, in _build_mapping
2024-05-06 18:51:09     column_type: str = mapping[column]["type"]
2024-05-06 18:51:09                        ~~~~~~~~~~~~~~~^^^^^^^^
2024-05-06 18:51:09 TypeError: string indices must be integers, not 'str'
@MeRuslan
Copy link
Contributor Author

MeRuslan commented May 6, 2024

am I doing something wrong?

@SkymaxPlay
Copy link

SkymaxPlay commented May 7, 2024 •
edited

Correct format:

"transform": {
    "mapping": {
        "id": {
            "type": "long"
        }
    }
}

@MeRuslan
Copy link
Contributor Author

MeRuslan commented May 7, 2024

@SkymaxPlay thanks!
There's a documentation issue then at https://pgsync.com/tutorial/transform-nodes/
image

@MeRuslan MeRuslan changed the title "mappings" keyword does not work properly for setting index type documentation bug at transform->mapping: https://pgsync.com/tutorial/transform-nodes/ May 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@MeRuslan @SkymaxPlay