Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[MSSQL] Add logs #2527

Merged
merged 8 commits into from
May 28, 2024
Merged

[MSSQL] Add logs #2527

merged 8 commits into from
May 28, 2024

Conversation

moxarth-elastic
Copy link
Collaborator

@moxarth-elastic moxarth-elastic commented May 8, 2024

Part Of #2299

Adding more logs in MSSQL connector.

Log File: https://drive.google.com/file/d/1Rnm-tMY-ECYv1JZpj3oP9SvMljZZRxjZ/view?usp=drive_link

Checklists

Pre-Review Checklist

  • this PR does NOT contain credentials of any kind, such as API keys or username/passwords (double check config.yml.example)
  • this PR has a meaningful title
  • this PR links to all relevant github issues that it fixes or partially addresses
  • if there is no GH issue, please create it. Each PR should have a link to an issue
  • this PR has a thorough description
  • Tested the changes locally
  • Added a label for each target release version (example: v7.13.2, v7.14.0, v8.0.0)

@moxarth-elastic moxarth-elastic requested a review from a team as a code owner May 8, 2024 12:19
@moxarth-elastic moxarth-elastic changed the title Add logs in MSSQL [MSSQL] Add logs May 8, 2024
@moxarth-elastic moxarth-elastic mentioned this pull request May 8, 2024
6 tasks
@moxarth-elastic
Copy link
Collaborator Author

buildkite test this

Copy link
Member

@seanstory seanstory left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Few nits, but mostly looks good.

Comment on lines 258 to 262
msg = (
"Fetching all tables as the configuration field 'tables' is set to '*'"
if not is_filtering
else "Fetching all tables as the advanced sync rules are enabled."
)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
msg = (
"Fetching all tables as the configuration field 'tables' is set to '*'"
if not is_filtering
else "Fetching all tables as the advanced sync rules are enabled."
)
msg = "Fetching all tables"

@@ -496,6 +512,7 @@ def row2doc(self, row, doc_id, table, timestamp):
return row

async def get_primary_key(self, tables):
self._logger.debug(f"Extracting primary keys for tables: {tables}")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lets log what we find as primary keys, too

return primary_keys

async def get_table_last_update_time(self, table):
self._logger.debug(f"Fetching last updated time for table: {table}")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's log what time we found, too

@artem-shelkovnikov
Copy link
Member

Copying from another PR - that's applicable to all SQL connectors:

The most useful connector logs are - which queries were actually executed by the connectors. Can you add debug logs that state which exact queries were executed? A good addition to it would be to also debug log the number of records returned by the query.

@moxarth-elastic
Copy link
Collaborator Author

buildkite test this

connectors/sources/mssql.py Outdated Show resolved Hide resolved
connectors/sources/mssql.py Outdated Show resolved Hide resolved
connectors/sources/mssql.py Outdated Show resolved Hide resolved
connectors/sources/mssql.py Outdated Show resolved Hide resolved
connectors/sources/mssql.py Outdated Show resolved Hide resolved
connectors/sources/mssql.py Outdated Show resolved Hide resolved
connectors/sources/mssql.py Outdated Show resolved Hide resolved
connectors/sources/mssql.py Outdated Show resolved Hide resolved
moxarth-elastic and others added 3 commits May 16, 2024 11:46
Co-authored-by: Artem Shelkovnikov <artem.shelkovnikov@elastic.co>
@moxarth-elastic
Copy link
Collaborator Author

buildkite test this

@artem-shelkovnikov
Copy link
Member

@moxarth-elastic I'll be taking over this and some other PRs reviewed by me to finish the work

@artem-shelkovnikov artem-shelkovnikov enabled auto-merge (squash) May 28, 2024 16:37
@artem-shelkovnikov artem-shelkovnikov merged commit 502f666 into main May 28, 2024
2 checks passed
@artem-shelkovnikov artem-shelkovnikov deleted the ms-sql-logs branch May 28, 2024 16:55
Copy link

💔 Failed to create backport PR(s)

The backport operation could not be completed due to the following error:
There are no branches to backport to. Aborting.

The backport PRs will be merged automatically after passing CI.

To backport manually run:
backport --pr 2527 --autoMerge --autoMergeMethod squash

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants