Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature Request: Support tags filter on nomad service discovery #11775

Open
devminded opened this issue Dec 27, 2022 · 6 comments · May be fixed by #14108
Open

Feature Request: Support tags filter on nomad service discovery #11775

devminded opened this issue Dec 27, 2022 · 6 comments · May be fixed by #14108

Comments

@devminded
Copy link

devminded commented Dec 27, 2022

Proposal

Use the same way of filtering services by tags as we can do with consul_sd_configs.
This let's us have a single job "service-metrics" that auto-detects, filters and scrapes any service which are marked with tags=prometheus,metrics .
Should also be backported to LTS version.

- job_name: service-metrics
  sample_limit: 10000
  scheme: http
  nomad_sd_configs:
    - server: 'http://127.0.0.1:4646'
      tags: ['prometheus', 'metrics']  # like with consul_sd_configs
  relabel_configs:
  - source_labels: [__meta_nomad_service_id]
    target_label: 'serviceid'
  - source_labels: [__meta_nomad_dc]
    target_label: 'dc'
  - source_labels: [__meta_nomad_service]
    target_label: 'service'
  - source_labels: [__meta_nomad_tags]
    regex: '.*?,environment=(.*?),.*'
    target_label: 'environment'
  - source_labels: [__meta_nomad_tags]
    regex: '.*?,metrics-path=(.*?),.*'
    target_label: '__metrics_path__'
@devminded
Copy link
Author

Related to: #11465

@devminded devminded changed the title Support tags filter on nomad service discovery Feature Request: Support tags filter on nomad service discovery Dec 28, 2022
@devminded
Copy link
Author

2 months bump.

@devminded
Copy link
Author

6 months bump. 🤔

@SamMousa
Copy link

I'm working on a different approach, this is still a WIP. The idea is to use Nomad's templating in combination with Prometheus HTTP discovery; this is my HCL:

job "prometheus-discovery" {
  datacenters = ["dc1"]


  namespace = "cluster"
  type = "service"
  update {
    max_parallel      = 1
    health_check      = "checks"
    min_healthy_time  = "10s"
    healthy_deadline  = "1m"
    progress_deadline = "2m"
    auto_revert       = true
    auto_promote      = true
    canary            = 1
    stagger           = "10s"
  }
  group "prometheus-discovery" {
    count = 2
    ephemeral_disk {
      size = 150
    }
    network {
      port "http" {
        to = 80
      }
    }

    task "nginx" {


      service {
        name = "prometheus-discovery"
        port = "http"
        provider = "nomad"


        check {
          name = "status endpoint"
          type     = "http"
          path     = "/ping"
          interval = "5s"
          timeout  = "1s"
          method   = "GET"
          header {
            User-Agent = ["Nomad"]
          }
        }
      }

      logs {
        max_files     = 10
        max_file_size = 1
      }

      driver = "docker"

      config {
        image = "nginx:stable-alpine"
        entrypoint = ["nginx", "-c", "/local/nginx.conf"]
        ports = ["http"]
        init = true
      }

      template {
        destination = "local/nginx.conf"
        env         = false
        change_mode = "restart"
        data        = <<-EOH
    daemon off;
    user nobody;
    worker_processes 1;

    events {
        use           epoll;
        worker_connections  128;
    }
    http {
        server {
            root /local/www;

            location /ping {
                default_type text/plain;
                return 200 "OK";
            }
        }
    }
EOH
      }

template {
        destination = "local/www/services.json"
        env         = false
        change_mode = "noop"
        data        = <<-EOH
{{- sprig_toPrettyJson nomadServices -}}
EOH
      }
      resources {
        cpu    = 100
        memory = 64
      }

    }

  }
}

I could run this and point prometheus to it (optionally through my load balancer).
Using the HCL template it then becomes trivial to filter using tags.
And of course I'll need to change the output to be the format prometheus expects.

@devminded
Copy link
Author

No one wants to implement this? My Go-fu is at level 1 so I can't contribute.

@SamMousa
Copy link

SamMousa commented Mar 4, 2024

The approach above works for me in prod

jaloren added a commit to jaloren/prometheus that referenced this issue May 15, 2024
Fixes prometheus#11775

Signed-off-by: Joseph Lorenzini <jaloren@fastmail.com>
jaloren added a commit to jaloren/prometheus that referenced this issue May 15, 2024
Fixes prometheus#11775

Signed-off-by: Joseph Lorenzini <jaloren@fastmail.com>
jaloren added a commit to jaloren/prometheus that referenced this issue May 15, 2024
Fixes prometheus#11775

Signed-off-by: Joseph Lorenzini <jaloren@fastmail.com>
jaloren added a commit to jaloren/prometheus that referenced this issue May 15, 2024
Fixes prometheus#11775

Signed-off-by: Joseph Lorenzini <jaloren@fastmail.com>
jaloren added a commit to jaloren/prometheus that referenced this issue May 15, 2024
Fixes prometheus#11775

Signed-off-by: Joseph Lorenzini <jaloren@fastmail.com>
jaloren added a commit to jaloren/prometheus that referenced this issue May 16, 2024
Fixes prometheus#11775

Signed-off-by: Joseph Lorenzini <jaloren@fastmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants