Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

时间精度问题 #1699

Open
3 tasks done
huangfuProject opened this issue May 31, 2023 · 4 comments · May be fixed by #1702
Open
3 tasks done

时间精度问题 #1699

huangfuProject opened this issue May 31, 2023 · 4 comments · May be fixed by #1702
Labels
bug Something isn't working

Comments

@huangfuProject
Copy link

Search before asking

  • I had searched in the issues and found no similar issues.

What happened

目前项目上使用 时间戳当作增量字段【startLocation】，同步dm数据，当同步2009年9月9号之前的数据的时候，项目报错：

   Unknown time unit:startLocation=xxxxxxxxxx

经查验源代码中有如下逻辑，代码位置
com.dtstack.chunjun.connector.jdbc.util.JdbcUtil#getMillis
代码如下：

    /**
     * 将边界位置时间转换成对应饿的毫秒时间
     *
     * @param startLocation 边界位置(起始/结束)
     * @return
     */
    public static long getMillis(long startLocation) {
        String timeStr = String.valueOf(startLocation);
        long millisSecond;
        if (timeStr.length() == SECOND_LENGTH) {
            millisSecond = startLocation * 1000;
        } else if (timeStr.length() == MILLIS_LENGTH) {
            millisSecond = startLocation;
        } else if (timeStr.length() == MICRO_LENGTH) {
            millisSecond = startLocation / 1000;
        } else if (timeStr.length() == NANOS_LENGTH) {
            millisSecond = startLocation / 1000000;
        } else {
            throw new IllegalArgumentException("Unknown time unit:startLocation=" + startLocation);
        }

        return millisSecond;
    }

经查验，上述代码对于时间单位的判断是基于时间戳长度来判断的

    /** 秒级时间戳的长度为10位 */
    private static final int SECOND_LENGTH = 10;
    /** 毫秒级时间戳的长度为13位 */
    private static final int MILLIS_LENGTH = 13;
    /** 微秒级时间戳的长度为16位 */
    private static final int MICRO_LENGTH = 16;
    /** 纳秒级时间戳的长度为19位 */
    private static final int NANOS_LENGTH = 19;

但是 2001-09-09 09:46:39 之前的毫秒级时间戳为 999999999999,长度为12位，导致在上述代码判断中，出现无法判断单位的情况！所以报错：

Unknown time unit:startLocation=999517080000

What you expected to happen

当我使用毫秒且时间为 2001-09-09 09:46:39 之前的数据，希望可用！

How to reproduce

rdb类型的数据，startLocation数值为2001-09-09 09:46:39之前的时间

Anything else

No response

Version

master

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct
@huangfuProject huangfuProject added the bug Something isn't working label May 31, 2023
@huangfuProject
Copy link
Author

补充配置文件

{
    "job": {
        "content": [
            {
                "reader": {
                    "parameter": {
                        "password": "xxxxxxxxxxxx",
                        "startLocation": "2001-09-03 19:38:00",
                        "increColumn": "CS_TSTP",
                        "useMaxFunc": false,
                        "column": [
                            {
                                "name": "AGE",
                                "type": "NUMBER"
                            },
                            {
                                "name": "BIRTH2",
                                "type": "DATE"
                            },
                            {
                                "name": "CESHI2",
                                "type": "VARCHAR2"
                            },
                            {
                                "name": "CESHIINT",
                                "type": "INT"
                            },
                            {
                                "format": "yyyy-MM-dd HH:mm:ss",
                                "name": "CS_TSTP",
                                "type": "TIMESTAMP"
                            },
                            {
                                "name": "HIGH",
                                "type": "NUMBER"
                            },
                            {
                                "name": "ID",
                                "type": "NUMBER"
                            },
                            {
                                "name": "MOMOU",
                                "type": "VARCHAR2"
                            },
                            {
                                "name": "NAME",
                                "type": "VARCHAR2"
                            },
                            {
                                "name": "WEIGHT",
                                "type": "NUMBER"
                            }
                        ],
                        "increment": true,
                        "connection": [
                            {
                                "schema": "HUANGFU",
                                "jdbcUrl": [
                                    "jdbc:dm://xxxxxxxxx:xxxxx"
                                ],
                                "table": [
                                    "PERSON_INFO"
                                ]
                            }
                        ],
                        "where": "",
                        "polling": true,
                        "increColumnType": "TIMESTAMP",
                        "username": "HUANGFU"
                    },
                    "name": "dmreader"
                },
                "writer": {
                    "parameter": {
                        "tableFields": [
                            "AGE",
                            "BIRTH2",
                            "CESHI2",
                            "CESHIINT",
                            "CS_TSTP",
                            "HIGH",
                            "ID",
                            "MOMOU",
                            "NAME",
                            "WEIGHT"
                        ],
                        "producerSettings": {
                            "retries": "3",
                            "request.timeout.ms": "60000",
                            "batch.size": "16384",
                            "acks": "0",
                            "bootstrap.servers": "xxxx:9092,xxxx:9092,xxxxx:9092",
                            "buffer.memory": "33554432"
                        },
                        "topic": "dm_tp_w_0530"
                    },
                    "name": "kafkawriter"
                }
            }
        ],
        "setting": {
            "speed": {
                "bytes": 1024
            }
        }
    }
}

@FlechazoW FlechazoW linked a pull request Jun 11, 2023 that will close this issue
针对时间精度，默认为毫秒 #1702
Open
@FlechazoW
Copy link
Member

issue 描述中的startLocaltion 不是标准的时间戳格式，如果转成正常的时间戳格式呢？

@FlechazoW
Copy link
Member

抱歉，最近比较忙，没有及时回复。

@huangfuProject
Copy link
Author

issue 描述中的startLocaltion 不是标准的时间触发格式，如果转成正常的时间触发格式呢？
我不太明白您说的正常的时间触发格式，数据库的数据是存在大量的2001-09-03之前的数据的，默认毫秒的话，这个时间之前的数据全部都不满足长度的！

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

针对时间精度，默认为毫秒 huangfuProject/chunjun
2 participants
@FlechazoW @huangfuProject