Error synchronizing measurements with points in the name

[franco-toniolo]

I have an error when I try to run syncflux in a database where there are measurements that have a point in the name. For example if I have the measurement: CITY.TEMPERATURE when I execute the following command, this error appears

/opt/syncflux # ./bin/syncflux -action "copy" -start -1h
INFO[2019-08-05 18:54:50] CFG :&{General:{InstanceID: LogDir:./log HomeDir: DataDir: LogLevel:debug SyncMode:onlyslave CheckInterval:10s MinSyncInterval:20s MasterDB:influxdb01 SlaveDB:influxdb02 InitialReplication:none MonitorRetryInterval:1m0s DataChunkDuration:5m0s MaxRetentionInterval:8760h0m0s RWMaxRetries:5 RWRetryDelay:10s NumWorkers:4 MaxPointsOnSingleWrite:20000} HTTP:{BindAddr:0.0.0.0:4090 AdminUser:admin AdminPassword:admin CookieID:mysupercokie} InfluxArray:[0xc0001b13e0 0xc0001b14a0]}
INFO[2019-08-05 18:54:50] Set Master DB influxdb01 from Command Line parameters
INFO[2019-08-05 18:54:50] Set Slave DB influxdb02 from Command Line parameters
WARN[2019-08-05 18:54:50] The response for Query is null, get Fields from DB WHEATHER Measurement CITY.TEMPERATURE error!

This is because “CITY” is taken as retention policy.

[toni-moreno]

Hi @franco-toniolo , I have released 0.6.5 version. Could you test if you can now run syncflux with your measurement names containing points in the name?

[maxadamo]

@franco-toniolo @toni-moreno I am afraid this is slightly different. I have fixed this one as well (but I didn't make a PR).
Everything will start working by simply commenting out this line here: https://github.com/toni-moreno/syncflux/blob/master/pkg/agent/client.go#L244
I don't think field names have "retention policy" property. Retention policy applies only to data... does it make sense to you?

p.s.: I believe the user franco-toniolo is gone (he has an empty github account, without any activity).... but you can discuss the issue with me 😄

[sbengo]

@maxadamo , as you can have different retention policies it means that there can be different measurements and fields as well.

You had recently make a PR (already merged) #27 and I'm wondering ifit isn't related.
Can you give us more detail about your db schema? (db name, rp name,...)

Thanks,
Regards!

[maxadamo]

@sbengo no, the PR is not related.
The error shown by Franco (which is also happening to me), is a WARN, and it happens when Syncflux runs:
show field keys from "measurement_name".
And it's saying "sorry, I didn't find any field", and it wouldn't be a problem, but it is happening on each and every table (as you probably know influx doesn't even allow to create empty measurement, hence it would even be strange to have an empty measurement).
Speaking about fields, they are the column names, and data inside the field can have indeed retention, but I believe the the column names are just metadata and retention apply to the data only.

I could be entirely wrong, but I don't see that you can query for retention on field names:
https://docs.influxdata.com/influxdb/v1.7/query_language/schema_exploration/#show-field-keys

[sbengo]

Hi @maxadamo , thanks for the answer!

I assume that Franco's issue is actually solved by already merged PR.

My comment was addressed entirely to you:

• Fields can differ from each shard and, each shard is stored within DB and RP combo info
• According to this issue, fields can differ for each RP: [bug] SHOW FIELD KEYS missing results influxdata/influxdb#8589

Can you give us some info about your schema and some logs?

Thanks,
Regards!

[maxadamo]

I am working on acceptance, I have 30 days of retention:

> SHOW RETENTION POLICIES ON "sensu";
name    duration shardGroupDuration replicaN default
----    -------- ------------------ -------- -------
autogen 0s       168h0m0s           1        false
30days  720h0m0s 24h0m0s            1        true

inside sensu there is a bunch of measurements... nothing weird I'd say.
But perhaps I am understanding what you have written: are you saying the the Go libraries does not detect the default RP, and I need to specify it, otherwise it throws an empty set?
If I do it from the CLI I don't need to specify it.

I am gonna compile it again, and check this once again.

[maxadamo]

@sbengo I finally found some spare time today. I noticed that it loops across the retention policies, and when it goes through the one which is not in use, it throws this message.

The message is misleading. because it's a log.Warnf, but then, it says error!
It doesn't show the retention policy name, hence it creates confusion.
It doesn't show any success information for the correct retention policy.

inside the file pkg/agent/client.go:
log.Warnf("The response for Query is null, get Fields from DB %s Measurement %s error!\n", sdb, meas)

I already have another PR, and I don't want to mess up with different things: I won't make a PR for this now.
IMO it should be a WARN and it should print the name of the retention policy.