[PostgreSQL] Add logs #2529
[PostgreSQL] Add logs #2529
Conversation
connectors/sources/postgresql.py
Outdated
| msg = ( | ||
| "Fetching all tables as the configuration field 'tables' is set to '*'" | ||
| if not is_filtering | ||
| else "Fetching all tables as the advanced sync rules are enabled." | ||
| ) | ||
| self._logger.info(msg) |
There was a problem hiding this comment.
| msg = ( | |
| "Fetching all tables as the configuration field 'tables' is set to '*'" | |
| if not is_filtering | |
| else "Fetching all tables as the advanced sync rules are enabled." | |
| ) | |
| self._logger.info(msg) | |
| msg = "Fetching all tables" | |
| self._logger.info(msg) |
| return primary_keys | ||
|
|
||
| async def get_table_last_update_time(self, table): | ||
| self._logger.debug(f"Fetching last updated time for table: {table}") |
connectors/sources/postgresql.py
Outdated
| @@ -266,9 +274,12 @@ async def get_table_primary_key(self, table): | |||
| ) | |||
| ] | |||
|
|
|||
| self._logger.debug(f"Found primary keys for '{table}' table") | |||
There was a problem hiding this comment.
let's log what the primary keys were
connectors/sources/postgresql.py
Outdated
| @@ -300,6 +311,7 @@ async def data_streamer( | |||
| list: It will first yield the column names, then data in each row | |||
| """ | |||
| if query is None and row_count is not None and order_by_columns is not None: | |||
| self._logger.debug(f"Streaming records from database using query: {query}") | |||
There was a problem hiding this comment.
This seems wrong. query is None was just asserted above this
There was a problem hiding this comment.
I think the logs are swapped for if-else conditions.
connectors/sources/postgresql.py
Outdated
| @@ -328,6 +340,7 @@ async def data_streamer( | |||
| if row_count <= offset: | |||
| return | |||
| else: | |||
| self._logger.debug(f"Streaming records from database for table: {table}") | |||
There was a problem hiding this comment.
In the sample log file, table is None. That seems like an error.
[FMWK][18:58:23][DEBUG] [Connector id: 3S2VrY4BAqKLTyj6P7_3, index name: postgresql, Sync job id: KpLkvY4BBpk9ZgBA2rJn] Streaming records from database for table: None
|
Copying from another PR - that's applicable to all SQL connectors: The most useful connector logs are - which queries were actually executed by the connectors. Can you add debug logs that state which exact queries were executed? A good addition to it would be to also debug log the number of records returned by the query. |
@artem-shelkovnikov This is already present here connectors/connectors/sources/postgresql.py Line 348 in 43ee46c Also, i've added a log for the number of records fetched from table/query. |
Part Of #2299
Adding more logs in PostgreSQL connector.
Log file: https://drive.google.com/file/d/12Pkq94J3ymnVGhXBtNc8S61xLqzDFCSK/view?usp=drive_link
Checklists
Pre-Review Checklist
config.yml.example)v7.13.2,v7.14.0,v8.0.0)