RFE: Target discrete Pool_Member entities with ConfigMap

[sectoreleven]

Title

Target discrete Pool_Member entities with ConfigMap

Description

Allow CIS to target discrete Pool_Member entities in a given AS3 Pool when using ConfigMap.

Actual Problem

AS3 and BIG-IP LTM allow application flow designers to treat subsets of pool members differently by setting member-level properties such as adminState, priorityGroup, ratio, connectionLimit, et cetera. This lets the BIG-IP have a single pool with dynamic traffic characteristics based on the availability and capacity of the various backend members.

CIS, however, currently targets a single pool, and replaces the entire pool contents with all members of a given K8s Service. This prevents us from doing the above fine-grained control. When using ConfigMap, if multiple Pool_Members are defined in the AS3 declaration, they are not respected and only one is used. (It is unclear if it is order-based or randomly selected.)

Solution Proposed

Enhance the pool-targeting capabilities for ConfigMap to allow the selection of a given Pool_Member, allowing us to define multiple Services targeting the same pool but different member groups; when the resultant AS3 declaration is composed by CIS, respect the properties defined on each Pool_Member.

A field would need to be selected in (or a new field added to) the AS3 Pool_Member schema to identify a given member; for the purposes of this RFE, we utilize the existing remark field. It may be better to add a name field instead.

For example - suppose an application that is running on a cluster where the nodes are geographically separated into availability zones. Imagine also that the secondary zone does not have the same compute capability as the primary.

To ensure maximum availability, the application team runs two Deployments of their product in the cluster; pods from these Deployments are scheduled to their respective nodes (and thus availability zones) via nodeSelector. The failsafe group should not activate unless the primary is down. They choose to implement this using Priority Activation Groups on the BIG-IP.

Given this scenario, we propose the following resources:

1. A ConfigMap containing an AS3 declaration with a Pool that looks as follows:

"test-pool": {
  "class": "Pool",
  "minimumMembersActive": 1,
  "members": [
    {
      "servicePort": 443,
      "serverAddresses": [],
      "enabled": true,
      "rateLimit": 5000,
      "priorityGroup": 10,
      "remark": "primary-group"
    },
    {
      "servicePort": 443,
      "serverAddresses": [],
      "enabled": true,
      "rateLimit": 2000,
      "priorityGroup": 0,
      "remark": "failsafe-group"
    },
  ]
}

2. A Service that targets the primary priority activation group (selectors for the pods not shown):

apiVersion: v1
kind: Service
metadata:
  name: test-app
  namespace: test-app-frontend
  labels:
    app: test-app-primary
    cis.f5.com/as3-tenant: test-tenant
    cis.f5.com/as3-app: app
    cis.f5.com/as3-pool: test-pool
    cis.f5.com/as3-pool-member: primary-group

3. A Service that targets the failsafe priority activation group:

apiVersion: v1
kind: Service
metadata:
  name: test-app
  namespace: test-app-frontend
  labels:
    app: test-app-failsafe
    cis.f5.com/as3-tenant: test-tenant
    cis.f5.com/as3-app: app
    cis.f5.com/as3-pool: test-pool
    cis.f5.com/as3-pool-member: failsafe-group

This would build a single Pool on the BIG-IP with members dynamically populated from the Services, where the primary priority group on the BIG-IP routes to the Service in the primary zone, and likewise for the failsafe.

Alternatives

We have built an alternative prototype that would utilize discrete BIG-IP pools, but that pushes selection logic to request-time via iRule or LTM Policy. Using the platform-native capabilities to build pools is significantly preferred, and allows highly-customizable scenarios where apps can be up continually in the Kubernetes cluster, having the BIG-IP device provide advanced traffic-routing capabilities.

Additional context

This relates to #3277 and expands on discussion from that bug report.

While the scenario above uses Priority Group Activation, this would be beneficial for any scenario where a single BIG-IP pool is desired containing members with different characteristics, represented by different Services in the Kubernetes cluster.

[trinaths]

Created [CONTCNTR-4698] for internal tracking.