elastic-agent: failed to parse field [event.dataset] only accepts values that are equal to the value defined in the mappings [system.process.summary], but got [system.process_summary]\ "}}, dropping event! #4750

davidg-datascene · 2024-05-14T13:53:32Z

Error reported in elastic-agent log file with debug logging:

Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Date(2024, time.May, 14, 11, 21, 18, 572370253, time.Local), Meta:{\"input_id\":\"system-metrics\",\"raw_index\":\"metrics-system.process_summary-default\"}
 
failed to parse field [event.dataset] of type [constant_keyword] in document with id 'LaXWdo8B3mT0fKffBUgR'. Preview of field's value: 'system.process_summary'\",\"caused_by\":{\"type\":\"illegal_argument_exception\",\"reason\":\"[constant_keyword] field [event.dataset] only accepts values that are equal to the value defined in the mappings [system.process.summary], but got [system.process_summary]\"}}, dropping event!

If we change the index template to be the generic metrics as per screen shot then data is sent to the index. However changing it to metrics-system.process.summary index template which should be the correct one then fails with the mentioned error.

Using metrics index template

Should be able to use this index template but errors as above

For confirmed bugs, please report:

Version:
Elastic Agent: 8.13.2 running on Kubernetes as daemon set
Elastic deployment: 8.11.3
Steps to Reproduce:
Install elastic agent standalone onto Kubernetes using yaml as provided.
Install k8s integration in Kibana
Check elastic-agents for 400 errors

Snippet showing DataStream

         - data_stream:
              dataset: system.process_summary
              type: metrics
            period: 10s
            metricsets:
              - process_summary

Elastic agent standalone yaml

agentNodeDatastreams:
  agentYml: |
    outputs:
      default:
        type: elasticsearch
        hosts:
          - >-
            ${ES_HOST}
        username: ${ES_USERNAME}
        password: ${ES_PASSWORD}
    agent:
      monitoring:
        enabled: false
        use_output: default
        logs: false
        metrics: false
    providers.kubernetes:
      node: ${NODE_NAME}
      scope: node
      #Uncomment to enable hints' support
      #hints.enabled: true
    inputs:
      - id: kubernetes-cluster-metrics
        condition: ${kubernetes_leaderelection.leader} == true
        type: kubernetes/metrics
        use_output: default
        meta:
          package:
            name: kubernetes
            version: 1.52.0
        data_stream:
          namespace: default
        streams:
          - data_stream:
              dataset: kubernetes.apiserver
              type: metrics
            metricsets:
              - apiserver
            bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
            hosts:
              - 'https://${env.KUBERNETES_SERVICE_HOST}:${env.KUBERNETES_SERVICE_PORT}'
            period: 30s
            ssl.certificate_authorities:
              - /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
            processors:
            - add_fields:
                target: orchestrator.cluster
                fields:
                  name: "tools-prod"
            - drop_fields:
                fields: [/kubernetes.apiserver.*/, /kubernetes.namespace_labels.*/]
            - add_kubernetes_metadata:
          - data_stream:
              dataset: kubernetes.event
              type: metrics
            metricsets:
              - event
            period: 10s
            processors:
            - add_fields:
                target: orchestrator.cluster
                fields:
                  name: "tools-prod"
            - drop_fields:
                fields: [/kubernetes.apiserver.*/, /kubernetes.namespace_labels.*/]
            - add_kubernetes_metadata:
            add_metadata: true
          - data_stream:
              dataset: kubernetes.state_container
              type: metrics
            metricsets:
              - state_container
            add_metadata: true
            hosts:
              - 'prometheus-operator-kube-state-metrics.monitoring.svc.cluster.local:8080'
            period: 10s
            processors:
            - add_fields:
                target: orchestrator.cluster
                fields:
                  name: "tools-prod"
            - drop_fields:
                fields: [/kubernetes.apiserver.*/, /kubernetes.namespace_labels.*/]
            - add_kubernetes_metadata:
          - data_stream:
              dataset: kubernetes.state_cronjob
              type: metrics
            metricsets:
              - state_cronjob
            add_metadata: true
            hosts:
              - 'prometheus-operator-kube-state-metrics.monitoring.svc.cluster.local:8080'
            period: 10s
            processors:
            - add_fields:
                target: orchestrator.cluster
                fields:
                  name: "tools-prod"
            - drop_fields:
                fields: [/kubernetes.apiserver.*/, /kubernetes.namespace_labels.*/]
            - add_kubernetes_metadata:
          - data_stream:
              dataset: kubernetes.state_daemonset
              type: metrics
            metricsets:
              - state_daemonset
            add_metadata: true
            hosts:
              - 'prometheus-operator-kube-state-metrics.monitoring.svc.cluster.local:8080'
            period: 10s
            processors:
            - add_fields:
                target: orchestrator.cluster
                fields:
                  name: "tools-prod"
            - drop_fields:
                fields: [/kubernetes.apiserver.*/, /kubernetes.namespace_labels.*/]
            - add_kubernetes_metadata:
          - data_stream:
              dataset: kubernetes.state_deployment
              type: metrics
            metricsets:
              - state_deployment
            add_metadata: true
            hosts:
              - 'prometheus-operator-kube-state-metrics.monitoring.svc.cluster.local:8080'
            period: 10s
            processors:
            - add_fields:
                target: orchestrator.cluster
                fields:
                  name: "tools-prod"
            - drop_fields:
                fields: [/kubernetes.apiserver.*/, /kubernetes.namespace_labels.*/]
            - add_kubernetes_metadata:
          - data_stream:
              dataset: kubernetes.state_job
              type: metrics
            metricsets:
              - state_job
            add_metadata: true
            hosts:
              - 'prometheus-operator-kube-state-metrics.monitoring.svc.cluster.local:8080'
            period: 10s
            processors:
            - add_fields:
                target: orchestrator.cluster
                fields:
                  name: "tools-prod"
            - drop_fields:
                fields: [/kubernetes.apiserver.*/, /kubernetes.namespace_labels.*/]
            - add_kubernetes_metadata:
          - data_stream:
              dataset: kubernetes.state_namespace
              type: metrics
            metricsets:
              - state_namespace
            add_metadata: true
            hosts:
              - 'prometheus-operator-kube-state-metrics.monitoring.svc.cluster.local:8080'
            period: 10s
            processors:
            - add_fields:
                target: orchestrator.cluster
                fields:
                  name: "tools-prod"
            - drop_fields:
                fields: [/kubernetes.apiserver.*/, /kubernetes.namespace_labels.*/]
            - add_kubernetes_metadata:
          - data_stream:
              dataset: kubernetes.state_node
              type: metrics
            metricsets:
              - state_node
            add_metadata: true
            hosts:
              - 'prometheus-operator-kube-state-metrics.monitoring.svc.cluster.local:8080'
            period: 10s
            processors:
            - add_fields:
                target: orchestrator.cluster
                fields:
                  name: "tools-prod"
            - drop_fields:
                fields: [/kubernetes.apiserver.*/, /kubernetes.namespace_labels.*/]
            - add_kubernetes_metadata:
          - data_stream:
              dataset: kubernetes.state_persistentvolume
              type: metrics
            metricsets:
              - state_persistentvolume
            add_metadata: true
            hosts:
              - 'prometheus-operator-kube-state-metrics.monitoring.svc.cluster.local:8080'
            period: 10s
            processors:
            - add_fields:
                target: orchestrator.cluster
                fields:
                  name: "tools-prod"
            - drop_fields:
                fields: [/kubernetes.apiserver.*/, /kubernetes.namespace_labels.*/]
            - add_kubernetes_metadata:
          - data_stream:
              dataset: kubernetes.state_persistentvolumeclaim
              type: metrics
            metricsets:
              - state_persistentvolumeclaim
            add_metadata: true
            hosts:
              - 'prometheus-operator-kube-state-metrics.monitoring.svc.cluster.local:8080'
            period: 10s
            processors:
            - add_fields:
                target: orchestrator.cluster
                fields:
                  name: "tools-prod"
            - drop_fields:
                fields: [/kubernetes.apiserver.*/, /kubernetes.namespace_labels.*/]
            - add_kubernetes_metadata:
          - data_stream:
              dataset: kubernetes.state_pod
              type: metrics
            metricsets:
              - state_pod
            add_metadata: true
            hosts:
              - 'prometheus-operator-kube-state-metrics.monitoring.svc.cluster.local:8080'
            period: 10s
            processors:
            - add_fields:
                target: orchestrator.cluster
                fields:
                  name: "tools-prod"
            - drop_fields:
                fields: [/kubernetes.apiserver.*/, /kubernetes.namespace_labels.*/]
            - add_kubernetes_metadata:
          - data_stream:
              dataset: kubernetes.state_replicaset
              type: metrics
            metricsets:
              - state_replicaset
            add_metadata: true
            hosts:
              - 'prometheus-operator-kube-state-metrics.monitoring.svc.cluster.local:8080'
            period: 10s
            processors:
            - add_fields:
                target: orchestrator.cluster
                fields:
                  name: "tools-prod"
            - drop_fields:
                fields: [/kubernetes.apiserver.*/, /kubernetes.namespace_labels.*/]
            - add_kubernetes_metadata:
          - data_stream:
              dataset: kubernetes.state_resourcequota
              type: metrics
            metricsets:
              - state_resourcequota
            add_metadata: true
            hosts:
              - 'prometheus-operator-kube-state-metrics.monitoring.svc.cluster.local:8080'
            period: 10s
            processors:
            - add_fields:
                target: orchestrator.cluster
                fields:
                  name: "tools-prod"
            - drop_fields:
                fields: [/kubernetes.apiserver.*/, /kubernetes.namespace_labels.*/]
            - add_kubernetes_metadata:
          - data_stream:
              dataset: kubernetes.state_service
              type: metrics
            metricsets:
              - state_service
            add_metadata: true
            hosts:
              - 'prometheus-operator-kube-state-metrics.monitoring.svc.cluster.local:8080'
            period: 10s
            processors:
            - add_fields:
                target: orchestrator.cluster
                fields:
                  name: "tools-prod"
            - drop_fields:
                fields: [/kubernetes.apiserver.*/, /kubernetes.namespace_labels.*/]
            - add_kubernetes_metadata:
          - data_stream:
              dataset: kubernetes.state_statefulset
              type: metrics
            metricsets:
              - state_statefulset
            add_metadata: true
            hosts:
              - 'prometheus-operator-kube-state-metrics.monitoring.svc.cluster.local:8080'
            period: 10s
            processors:
            - add_fields:
                target: orchestrator.cluster
                fields:
                  name: "tools-prod"
            - drop_fields:
                fields: [/kubernetes.apiserver.*/, /kubernetes.namespace_labels.*/]
            - add_kubernetes_metadata:
          - data_stream:
              dataset: kubernetes.state_storageclass
              type: metrics
            metricsets:
              - state_storageclass
            add_metadata: true
            hosts:
              - 'prometheus-operator-kube-state-metrics.monitoring.svc.cluster.local:8080'
            period: 10s
            processors:
            - add_fields:
                target: orchestrator.cluster
                fields:
                  name: "tools-prod"
            - drop_fields:
                fields: [/kubernetes.apiserver.*/, /kubernetes.namespace_labels.*/]
            - add_kubernetes_metadata:
      - id: system-logs
        type: logfile
        use_output: default
        meta:
          package:
            name: system
            version: 1.20.4
        data_stream:
          namespace: default
        streams:
          - data_stream:
              dataset: system.auth
              type: logs
            paths:
              - /var/log/auth.log*
              - /var/log/secure*
            exclude_files:
              - .gz$
            multiline:
              pattern: ^s
              match: after
            processors:
            - add_locale: null
            - add_fields:
                target: orchestrator.cluster
                fields:
                  name: "tools-prod"
            ignore_older: 72h
          - data_stream:
              dataset: system.syslog
              type: logs
            paths:
              - /var/log/messages*
              - /var/log/syslog*
            exclude_files:
              - .gz$
            multiline:
              pattern: ^s
              match: after
            processors:
            - add_locale: null
            - add_fields:
                target: orchestrator.cluster
                fields:
                  name: "tools-prod"
            ignore_older: 72h
      - id: system-metrics
        type: system/metrics
        use_output: default
        meta:
          package:
            name: system
            version: 1.20.4
        data_stream:
          namespace: default
        streams:
          - data_stream:
              dataset: system.cpu
              type: metrics
            period: 10s
            cpu.metrics:
              - percentages
              - normalized_percentages
            metricsets:
              - cpu
            processors:
            - add_fields:
                target: orchestrator.cluster
                fields:
                  name: "tools-prod"
          - data_stream:
              dataset: system.diskio
              type: metrics
            period: 10s
            diskio.include_devices: null
            metricsets:
              - diskio
            processors:
            - add_fields:
                target: orchestrator.cluster
                fields:
                  name: "tools-prod"
          - data_stream:
              dataset: system.filesystem
              type: metrics
            period: 1m
            metricsets:
              - filesystem
            processors:
            - drop_event.when.regexp:
                system.filesystem.mount_point: ^/(sys|cgroup|proc|dev|etc|host|lib|snap)($|/)
            - add_fields:
                target: orchestrator.cluster
                fields:
                  name: "tools-prod"
          - data_stream:
              dataset: system.fsstat
              type: metrics
            period: 1m
            metricsets:
              - fsstat
            processors:
            - drop_event.when.regexp:
                system.fsstat.mount_point: ^/(sys|cgroup|proc|dev|etc|host|lib|snap)($|/)
            - add_fields:
                target: orchestrator.cluster
                fields:
                  name: "tools-prod"
          - data_stream:
              dataset: system.load
              type: metrics
            condition: '${host.platform} != ''windows'''
            period: 10s
            metricsets:
              - load
            processors:
            - add_fields:
                target: orchestrator.cluster
                fields:
                  name: "tools-prod"
          - data_stream:
              dataset: system.memory
              type: metrics
            period: 10s
            metricsets:
              - memory
            processors:
            - add_fields:
                target: orchestrator.cluster
                fields:
                  name: "tools-prod"
          - data_stream:
              dataset: system.network
              type: metrics
            period: 10s
            network.interfaces: null
            metricsets:
              - network
            processors:
            - add_fields:
                target: orchestrator.cluster
                fields:
                  name: "tools-prod"
          - data_stream:
              dataset: system.process
              type: metrics
            period: 10s
            processes:
              - .*
            process.include_top_n.by_cpu: 5
            process.include_top_n.by_memory: 5
            process.cmdline.cache.enabled: true
            process.cgroups.enabled: false
            process.include_cpu_ticks: false
            metricsets:
              - process
            process.include_cpu_ticks: false
            processors:
            - add_fields:
                target: orchestrator.cluster
                fields:
                  name: "tools-prod"
          - data_stream:
              dataset: system.process_summary
              type: metrics
            period: 10s
            metricsets:
              - process_summary
            processors:
            - add_fields:
                target: orchestrator.cluster
                fields:
                  name: "tools-prod"
          - data_stream:
              dataset: system.socket_summary
              type: metrics
            period: 10s
            metricsets:
              - socket_summary
            processors:
            - add_fields:
                target: orchestrator.cluster
                fields:
                  name: "tools-prod"
          - data_stream:
              type: metrics
              dataset: system.uptime
            metricsets:
              - uptime
            period: 10s
            processors:
            - add_fields:
                target: orchestrator.cluster
                fields:
                  name: "tools-prod"
      - id: kubernetes-node-metrics
        type: kubernetes/metrics
        use_output: default
        meta:
          package:
            name: kubernetes
            version: 1.52.0
        data_stream:
          namespace: default
        streams:
          - data_stream:
              dataset: kubernetes.controllermanager
              type: metrics
            metricsets:
              - controllermanager
            bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
            hosts:
              - 'https://${kubernetes.pod.ip}:10257'
            period: 10s
            ssl.verification_mode: none
            condition: ${kubernetes.labels.component} == 'kube-controller-manager'
            processors:
            - add_fields:
                target: orchestrator.cluster
                fields:
                  name: "tools-prod"
            - drop_fields:
                fields: [/kubernetes.apiserver.*/, /kubernetes.namespace_labels.*/]
            - add_kubernetes_metadata:
          - data_stream:
              dataset: kubernetes.scheduler
              type: metrics
            metricsets:
              - scheduler
            bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
            hosts:
              - 'https://${kubernetes.pod.ip}:10259'
            period: 10s
            ssl.verification_mode: none
            condition: ${kubernetes.labels.component} == 'kube-scheduler'
            processors:
            - add_fields:
                target: orchestrator.cluster
                fields:
                  name: "tools-prod"
            - drop_fields:
                fields: [/kubernetes.apiserver.*/, /kubernetes.namespace_labels.*/]
            - add_kubernetes_metadata:
          - data_stream:
              dataset: kubernetes.proxy
              type: metrics
            metricsets:
              - proxy
            hosts:
              - 'localhost:10249'
            period: 10s
            processors:
            - add_fields:
                target: orchestrator.cluster
                fields:
                  name: "tools-prod"
            - drop_fields:
                fields: [/kubernetes.apiserver.*/, /kubernetes.namespace_labels.*/]
            - add_kubernetes_metadata:
          - data_stream:
              dataset: kubernetes.container
              type: metrics
            metricsets:
              - container
            add_metadata: true
            bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
            hosts:
              - 'https://${env.NODE_NAME}:10250'
            period: 10s
            ssl.verification_mode: none
            processors:
            - add_fields:
                target: orchestrator.cluster
                fields:
                  name: "tools-prod"
            - drop_fields:
                fields: [/kubernetes.apiserver.*/, /kubernetes.namespace_labels.*/]
            - add_kubernetes_metadata:
          - data_stream:
              dataset: kubernetes.node
              type: metrics
            metricsets:
              - node
            add_metadata: true
            bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
            hosts:
              - 'https://${env.NODE_NAME}:10250'
            period: 10s
            ssl.verification_mode: none
            processors:
            - add_fields:
                target: orchestrator.cluster
                fields:
                  name: "tools-prod"
            - drop_fields:
                fields: [/kubernetes.apiserver.*/, /kubernetes.namespace_labels.*/]
            - add_kubernetes_metadata:
          - data_stream:
              dataset: kubernetes.pod
              type: metrics
            metricsets:
              - pod
            add_metadata: true
            bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
            hosts:
              - 'https://${env.NODE_NAME}:10250'
            period: 10s
            ssl.verification_mode: none
            processors:
            - add_fields:
                target: orchestrator.cluster
                fields:
                  name: "tools-prod"
            - drop_fields:
                fields: [/kubernetes.apiserver.*/, /kubernetes.namespace_labels.*/]
            - add_kubernetes_metadata:
          - data_stream:
              dataset: kubernetes.system
              type: metrics
            metricsets:
              - system
            add_metadata: true
            bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
            hosts:
              - 'https://${env.NODE_NAME}:10250'
            period: 10s
            ssl.verification_mode: none
            processors:
            - add_fields:
                target: orchestrator.cluster
                fields:
                  name: "tools-prod"
            - drop_fields:
                fields: [/kubernetes.apiserver.*/, /kubernetes.namespace_labels.*/]
            - add_kubernetes_metadata:
          - data_stream:
              dataset: kubernetes.volume
              type: metrics
            metricsets:
              - volume
            add_metadata: true
            bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
            hosts:
              - 'https://${env.NODE_NAME}:10250'
            period: 10s
            ssl.verification_mode: none
            processors:
            - add_fields:
                target: orchestrator.cluster
                fields:
                  name: "tools-prod"
            - drop_fields:
                fields: [/kubernetes.apiserver.*/, /kubernetes.namespace_labels.*/]
            - add_kubernetes_metadata:

The text was updated successfully, but these errors were encountered:

davidg-datascene · 2024-05-14T16:14:51Z

Snippet from the metrics-system.process.summary Managed index template

failed to parse field [event.dataset] of type [constant_keyword] Preview of field's value: 'system.process_summary'","caused_by":{"type":"illegal_argument_exception","reason":"[constant_keyword] field [event.dataset] only accepts values that are equal to the value defined in the mappings [system.process.summary], but got [system.process_summary]"}}, dropping event!

"event": {
          "properties": {
            "agent_id_status": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "dataset": {
              "type": "constant_keyword",
              "value": "system.process.summary" <----- **_Shouldn't this be system.process_summary_**
            },
            "ingested": {
              "type": "date",
              "format": "strict_date_time_no_millis||strict_date_optional_time||epoch_millis"
            },
            "module": {
              "type": "constant_keyword",
              "value": "system"
            }
          }
        },

botelastic · 2024-05-14T19:27:21Z

This issue doesn't have a Team:<team> label.

cmacknz · 2024-05-14T20:37:06Z

The system integration is using system.process.summary so that is the correct one:

https://github.com/elastic/integrations/blob/80eedb5182dfa5ac60f43d9aa5a3971ed6f0d590/packages/system/data_stream/process_summary/manifest.yml#L1-L2

title: System process_summary metrics
dataset: system.process.summary

It appears that our reference YAML is out of sync:

elastic-agent/deploy/kubernetes/elastic-agent-standalone/elastic-agent-standalone-ksm-daemonset-configmap.yaml

Lines 272 to 277 in ee5c1ea

    
                     - data_stream: 
        
                         dataset: system.process_summary 
        
                         type: metrics 
        
                       period: 10s 
        
                       metricsets: 
        
                         - process_summary

davidg-datascene added the bug Something isn't working label May 14, 2024

ycombinator transferred this issue from elastic/elastic-agent May 14, 2024

botelastic bot added the needs_team label May 14, 2024

cmacknz transferred this issue from elastic/beats May 14, 2024

cmacknz added the Team:Cloudnative-Monitoring Label for the Cloud Native Monitoring team label May 14, 2024

ycombinator removed the needs_team label May 16, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

elastic-agent: failed to parse field [event.dataset] only accepts values that are equal to the value defined in the mappings [system.process.summary], but got [system.process_summary]\ "}}, dropping event! #4750

elastic-agent: failed to parse field [event.dataset] only accepts values that are equal to the value defined in the mappings [system.process.summary], but got [system.process_summary]\ "}}, dropping event! #4750

davidg-datascene commented May 14, 2024 •

edited

davidg-datascene commented May 14, 2024

botelastic bot commented May 14, 2024

cmacknz commented May 14, 2024

elastic-agent: failed to parse field [event.dataset] only accepts values that are equal to the value defined in the mappings [system.process.summary], but got [system.process_summary]\ "}}, dropping event! #4750

elastic-agent: failed to parse field [event.dataset] only accepts values that are equal to the value defined in the mappings [system.process.summary], but got [system.process_summary]\ "}}, dropping event! #4750

Comments

davidg-datascene commented May 14, 2024 • edited

davidg-datascene commented May 14, 2024

botelastic bot commented May 14, 2024

cmacknz commented May 14, 2024

davidg-datascene commented May 14, 2024 •

edited