Consul doesnt resolve CNAME records for registered services

[garry-t]

Overview of the Issue

Consul returns CNAME for registered services but doesn't return appropriate A record, in result not possible access to service using traditional methods. Direct dig query works, ping ,curl, application code no.

Reproduction Steps

Follow this instruction
https://developer.hashicorp.com/consul/tutorials/networking/dns-forwarding#dnsmasq-setup

Consul info for both Client and Server

Client info

Output from client 'consul info' command here
agent:
        check_monitors = 0
        check_ttls = 0
        checks = 0
        services = 2
build:
        prerelease = 
        revision = 3111cb8c
        version = 1.8.0
consul:
        acl = enabled
        known_servers = 2
        server = false
runtime:
        arch = amd64
        cpu_count = 3
        goroutines = 73
        max_procs = 3
        os = linux
        version = go1.14.4
serf_lan:
        coordinate_resets = 0
        encrypted = true
        event_queue = 0
        event_time = 364014
        failed = 0
        health_score = 0
        intent_queue = 0
        left = 0
        member_time = 22750
        members = 23
        query_queue = 0
        query_time = 5

In my case agent token and default token are the same
Client agent HCL config
{
    "acl": {
        "default_policy": "deny",
        "down_policy": "extend-cache",
        "enable_token_persistence": true,
        "enabled": true,
        "token_ttl": "30s",
        "tokens": {
            "agent": "TOKEN",
            "default": "TOKEN"
        }
    },
    "addresses": {
        "dns": "127.0.0.1",
        "grpc": "127.0.0.1",
        "http": "127.0.0.1",
        "https": "127.0.0.1"
    },
    "advertise_addr": "IP",
    "advertise_addr_wan": "IP",
    "bind_addr": "IP",
    "client_addr": "127.0.0.1",
    "data_dir": "/opt/consul",
    "datacenter": "fsn1",
    "disable_update_check": false,
    "domain": "consul",
    "enable_local_script_checks": true,
    "enable_script_checks": false,
    "encrypt": "TOKEN",
    "encrypt_verify_incoming": true,
    "encrypt_verify_outgoing": true,
    "log_file": "/var/log/consul/consul.log",
    "log_level": "DEBUG",
    "log_rotate_bytes": 0,
    "log_rotate_duration": "24h",
    "log_rotate_max_files": 0,
    "performance": {
        "leave_drain_time": "5s",
        "raft_multiplier": 1,
        "rpc_hold_timeout": "7s"
    },
    "ports": {
        "dns": 8600,
        "grpc": -1,
        "http": 8500,
        "https": -1,
        "serf_lan": 8301,
        "serf_wan": 8302,
        "server": 8300
    },
    "primary_datacenter": "fsn1",
    "raft_protocol": 3,
    "retry_interval": "30s",
    "retry_join": [
        "IP1",
        "IP2"
    ],
    "retry_max": 0,
    "server": false,
    "translate_wan_addrs": false,
    "ui": true,
    "node_name": "rabbitmq-01",
    "auto_encrypt": {
        "tls": true
    },
    "retry_join_wan": [
        "IP1",
        "IP2"
    ]
}

Server info

Operating system and Environment details

Ubuntu 22.04,
consul 1.8
dnsmasq installed via public ansible role

dnsmasq config

/etc/dnsmasq.d/10-consul
server=/consul/127.0.0.1#8600
server=8.8.8.8
server=8.8.4.4

No other config changes from dnsmasq side. All by default.

Log Fragments

dig rabbitmq.service.consul

; <<>> DiG 9.18.18-0ubuntu0.22.04.2-Ubuntu <<>> rabbitmq.service.consul
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 62925
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;rabbitmq.service.consul. IN A

;; ANSWER SECTION:
rabbitmq.service.consul. 0 IN CNAME  rabbitmq-01.

;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP)
;; WHEN: Thu Apr 04 07:39:07 UTC 2024
;; MSG SIZE  rcvd: 108

systemd[1]: Started dnsmasq - A lightweight DHCP and caching DNS server.
dnsmasq[18416]: query[A] rabbitmq.service.consul from 127.0.0.1
dnsmasq[18416]: forwarded rabbitmq.service.consul to 127.0.0.1#8600
dnsmasq[18416]: reply rabbitmq.service.consul is <CNAME>
dnsmasq[18416]: reply rabbitmq-01 is NODATA-IPv4

ACL policy for both tokens includes

 node_prefix "" {
      policy = "read"
    }

 service_prefix "" {
      policy = "read"
    }

Why consul returns node name without dns appropriate suffix?