Invalid iRule generation for HTTP/2 full proxy mode

[shkarface]

Setup Details

CIS Version : 2.16.1
Build: f5networks/k8s-bigip-ctlr:2.16.1
BIGIP Version: BIG-IP 17.1.0.3 Build 0.75.4 Engineering Hotfix
AS3 Version: 3.x
Agent Mode: AS3
Orchestration: K8S
Orchestration Version: v1.28.7
Pool Mode: Nodeport
Additional Setup details: Cilium CNI

Description

When generating the irule for TLS virtual servers in the getTLSIRule function in routing.go, the generated iRule seems to be invalid and generates this log in BIG-IP.

TCL error: /k8s-dev/Shared/payroll_gateway_443_tls_irule <SERVER_CONNECTED> - can't read "sslprofile": no such variable while executing "if { not ($sslprofile equals "false") } { SSL::profile $reen }"

Here are the used configurations:

Virtual Server:

apiVersion: cis.f5.com/v1
kind: VirtualServer
metadata:
  annotations:
    externaldns: enabled
    meta.helm.sh/release-name: gateway
    meta.helm.sh/release-namespace: payroll
    objectset.rio.cattle.io/id: default-payroll-gitops-gateway
  labels:
    app.kubernetes.io/managed-by: Helm
    f5cr: 'true'
    objectset.rio.cattle.io/hash: a67b146037fca2e49ddc407b58b5821983ed7d48
  name: gateway
  namespace: payroll
spec:
  host: payroll.dev.krd
  httpMrfRoutingEnabled: true
  ipamLabel: default
  policyName: gateway
  pools:
    - path: /
      service: traefik
      serviceNamespace: traefik
      servicePort: 443
  tlsProfileName: payroll-tls
  virtualServerName: payroll-gateway

TLS Profile:

apiVersion: cis.f5.com/v1
kind: TLSProfile
metadata:
  annotations:
    meta.helm.sh/release-name: gateway
    meta.helm.sh/release-namespace: payroll
    objectset.rio.cattle.io/id: default-payroll-gitops-gateway
  labels:
    app.kubernetes.io/managed-by: Helm
    f5cr: 'true'
    objectset.rio.cattle.io/hash: a67b146037fca2e49ddc407b58b5821983ed7d48
  name: payroll-tls
  namespace: payroll
spec:
  hosts:
    - payroll.dev.krd
  tls:
    clientSSL: payroll-tls-ssl
    clientSSLParams:
      profileReference: secret
      renegotiationEnabled: false
    reference: hybrid
    serverSSL: /Common/serverssl
    serverSSLParams:
      profileReference: bigip
      renegotiationEnabled: false
    termination: reencrypt

Policy:

apiVersion: cis.f5.com/v1
kind: Policy
metadata:
  annotations:
    meta.helm.sh/release-name: gateway
    meta.helm.sh/release-namespace: payroll
    objectset.rio.cattle.io/id: default-payroll-gitops-gateway
  labels:
    app.kubernetes.io/managed-by: Helm
    f5cr: 'true'
    objectset.rio.cattle.io/hash: a67b146037fca2e49ddc407b58b5821983ed7d48
  name: gateway
  namespace: payroll
spec:
  profiles:
    http: /Common/http-k8s
    http2:
      client: /Common/http2

Generated iRule:

when CLIENT_ACCEPTED { TCP::collect }
		proc select_ab_pool {path default_pool domainpath} {
			set last_slash [string length $path]
			set ab_class "/k8s-dev/Shared/payroll_gateway_443_ab_deployment_dg"
			while {$last_slash >= 0} {
				if {[class match $path equals $ab_class]} then {
					break
				}
				set last_slash [string last "/" $path $last_slash]
				incr last_slash -1
				set path [string range $path 0 $last_slash]
			}
			if {$last_slash >= 0} {
				set ab_rule [class match -value $path equals $ab_class]
				if {$ab_rule != ""} then {
					set weight_selection [expr {rand()}]
					set service_rules [split $ab_rule ";"]
                    set active_pool ""
					foreach service_rule $service_rules {
						set fields [split $service_rule ","]
						set pool_name [lindex $fields 0]
                        if { [active_members $pool_name] >= 1 } {
						    set active_pool $pool_name
						}
						set weight [expr {double([lindex $fields 1])}]
						if {$weight_selection <= $weight} then {
                            #check if active pool members are available
						    if { [active_members $pool_name] >= 1 } {
							    return $pool_name
						    } else {
						          # select the any of pool with active members
						          if {$active_pool!= ""} then {
						              return $active_pool
						          }
						    }
						}
					}
				}
				# If we had a match, but all weights were 0 then
				# retrun a 503 (Service Unavailable)
				HTTP::respond 503
			}
			return $default_pool
		}
		when CLIENT_DATA {
			# Byte 0 is the content type.
			# Bytes 1-2 are the TLS version.
			# Bytes 3-4 are the TLS payload length.
			# Bytes 5-$tls_payload_len are the TLS payload.
			binary scan [TCP::payload] cSS tls_content_type tls_version tls_payload_len
			if { ! [ expr { [info exists tls_content_type] && [string is integer -strict $tls_content_type] } ] }  { reject ; event disable all; return; }
			if { ! [ expr { [info exists tls_version] && [string is integer -strict $tls_version] } ] }  { reject ; event disable all; return; }
			switch -exact $tls_version {
				"769" -
				"770" -
				"771" {
					# Content type of 22 indicates the TLS payload contains a handshake.
					if { $tls_content_type == 22 } {
						# Byte 5 (the first byte of the handshake) indicates the handshake
						# record type, and a value of 1 signifies that the handshake record is
						# a ClientHello.
						binary scan [TCP::payload] @5c tls_handshake_record_type
						if { ! [ expr { [info exists tls_handshake_record_type] && [string is integer -strict $tls_handshake_record_type] } ] }  { reject ; event disable all; return; }
						if { $tls_handshake_record_type == 1 } {
							# Bytes 6-8 are the handshake length (which we ignore).
							# Bytes 9-10 are the TLS version (which we ignore).
							# Bytes 11-42 are random data (which we ignore).
							# Byte 43 is the session ID length.  Following this are three
							# variable-length fields which we shall skip over.
							set record_offset 43
							# Skip the session ID.
							binary scan [TCP::payload] @${record_offset}c tls_session_id_len
							if { ! [ expr { [info exists tls_session_id_len] && [string is integer -strict $tls_session_id_len] } ] }  { reject ; event disable all; return; }
							incr record_offset [expr {1 + $tls_session_id_len}]
							# Skip the cipher_suites field.
							binary scan [TCP::payload] @${record_offset}S tls_cipher_suites_len
							if { ! [ expr { [info exists tls_cipher_suites_len] && [string is integer -strict $tls_cipher_suites_len] } ] }  { reject ; event disable all; return; }
							incr record_offset [expr {2 + $tls_cipher_suites_len}]
							# Skip the compression_methods field.
							binary scan [TCP::payload] @${record_offset}c tls_compression_methods_len
							if { ! [ expr { [info exists tls_compression_methods_len] && [string is integer -strict $tls_compression_methods_len] } ] }  { reject ; event disable all; return; }
							incr record_offset [expr {1 + $tls_compression_methods_len}]
							# Get the number of extensions, and store the extensions.
							binary scan [TCP::payload] @${record_offset}S tls_extensions_len
							if { ! [ expr { [info exists tls_extensions_len] && [string is integer -strict $tls_extensions_len] } ] }  { reject ; event disable all; return; }
							incr record_offset 2
							binary scan [TCP::payload] @${record_offset}a* tls_extensions
							if { ! [info exists tls_extensions] }  { reject ; event disable all; return; }
							for { set extension_start 0 }
									{ $tls_extensions_len - $extension_start == abs($tls_extensions_len - $extension_start) }
									{ incr extension_start 4 } {
								# Bytes 0-1 of the extension are the extension type.
								# Bytes 2-3 of the extension are the extension length.
								binary scan $tls_extensions @${extension_start}SS extension_type extension_len
								if { ! [ expr { [info exists extension_type] && [string is integer -strict $extension_type] } ] }  { reject ; event disable all; return; }
								if { ! [ expr { [info exists extension_len] && [string is integer -strict $extension_len] } ] }  { reject ; event disable all; return; }
								# Extension type 00 is the ServerName extension.
								if { $extension_type == "00" } {
									# Bytes 4-5 of the extension are the SNI length (we ignore this).
									# Byte 6 of the extension is the SNI type.
									set sni_type_offset [expr {$extension_start + 6}]
									binary scan $tls_extensions @${sni_type_offset}S sni_type
									if { ! [ expr { [info exists sni_type] && [string is integer -strict $sni_type] } ] }  { reject ; event disable all; return; }
									# Type 0 is host_name.
									if { $sni_type == "0" } {
										# Bytes 7-8 of the extension are the SNI data (host_name)
										# length.
										set sni_len_offset [expr {$extension_start + 7}]
										binary scan $tls_extensions @${sni_len_offset}S sni_len
										if { ! [ expr { [info exists sni_len] && [string is integer -strict $sni_len] } ] }  { reject ; event disable all; return; }
										# Bytes 9-$sni_len are the SNI data (host_name).
										set sni_start [expr {$extension_start + 9}]
										binary scan $tls_extensions @${sni_start}A${sni_len} tls_servername
									}
								}
								incr extension_start $extension_len
							}
							if { [info exists tls_servername] } {
								set servername_lower [string tolower $tls_servername]
                            	set domain_length [llength [split $servername_lower "."]]
                				set domain_wc [domain $servername_lower [expr {$domain_length - 1}] ]
                				# Set wc_host with the wildcard domain
								set wc_host ".$domain_wc"
								set passthru_class "/k8s-dev/Shared/payroll_gateway_443_ssl_passthrough_servername_dg"
								if { [class exists $passthru_class] } {
                                    # check if the passthrough data group has a record with the servername
                                    set passthru_dg_key [class match $servername_lower equals $passthru_class]
								    set passthru_dg_wc_key [class match $wc_host equals $passthru_class]
								    if { $passthru_dg_key != 0 || $passthru_dg_wc_key != 0 } {
										SSL::disable serverside
										set dflt_pool_passthrough ""
										# Disable Serverside SSL for Passthrough Class
										set dflt_pool_passthrough [class match -value $servername_lower equals $passthru_class]
										# If no match, try wildcard domain
										if { $dflt_pool_passthrough == "" } {
											if { [class match $wc_host equals $passthru_class] } {
													set dflt_pool_passthrough [class match -value $wc_host equals $passthru_class]
											}
										}
										if { not ($dflt_pool_passthrough equals "") } {
											SSL::disable
											HTTP::disable
										}
										set ab_class "/k8s-dev/Shared/payroll_gateway_443_ab_deployment_dg"
										if { not [class exists $ab_class] } {
											if { $dflt_pool_passthrough == "" } then {
												log local0.debug "Failed to find pool for $servername_lower $"
											} else {
												pool $dflt_pool_passthrough
											}
										} else {
											set selected_pool [call select_ab_pool $servername_lower $dflt_pool_passthrough ""]
											if { $selected_pool == "" } then {
												log local0.debug "Failed to find pool for $servername_lower"
											} else {
												pool $selected_pool
											}
										}
                                    }
								}
							}
						}
					}
				}
			}
			TCP::release
		}
		when CLIENTSSL_HANDSHAKE {
 			SSL::collect
		}
         when CLIENTSSL_DATA {
            if { [llength [split [SSL::payload]]] < 1 }{
                reject ; event disable all; return;
                }
            set sslpath [lindex [split [SSL::payload]] 1]
			set domainpath $sslpath
            set routepath ""
            set wc_routepath ""
            if { [info exists tls_servername] } {
				set servername_lower [string tolower $tls_servername]
            	set domain_length [llength [split $servername_lower "."]]
				set domain_wc [domain $servername_lower [expr {$domain_length - 1}] ]
				set wc_host ".$domain_wc"
				# Set routepath as combination of servername and url path
				append routepath $servername_lower $sslpath
     			append wc_routepath $wc_host $sslpath
				set routepath [string tolower $routepath]
				set wc_routepath [string tolower $wc_routepath]
				set sslpath $routepath
				# Find the number of "/" in the routepath
				set rc 0
				foreach x [split $routepath {}] {
				   if {$x eq "/"} {
					   incr rc
				   }
				}
				# Disable serverside ssl and enable only for reencrypt routes
                SSL::disable serverside
				set reencrypt_class "/k8s-dev/Shared/payroll_gateway_443_ssl_reencrypt_servername_dg"
				set edge_class "/k8s-dev/Shared/payroll_gateway_443_ssl_edge_servername_dg"
                if { [class exists $reencrypt_class] || [class exists $edge_class] } {
					# Compares the routepath with the entries in ssl_reencrypt_servername_dg and
					# ssl_edge_servername_dg.
					for {set i $rc} {$i >= 0} {incr i -1} {
						if { [class exists $reencrypt_class] } {
							set reen_pool [class match -value $routepath equals $reencrypt_class]
                            # Check for wildcard domain
                            if { $reen_pool equals "" } {
							    if { [class match $wc_routepath equals $reencrypt_class] } {
							        set reen_pool [class match -value $wc_routepath equals $reencrypt_class]
                                }
                            }
							if { not ($reen_pool equals "") } {
								set dflt_pool $reen_pool
								SSL::enable serverside
							}
						}
						if { [class exists $edge_class] } {
							set edge_pool [class match -value $routepath equals $edge_class]
                            # Check for wildcard domain
                            if { $edge_pool equals "" } {
							    if { [class match $wc_routepath equals $edge_class] } {
							        set edge_pool [class match -value $wc_routepath equals $edge_class]
							    }
                            }
							if { not ($edge_pool equals "") } {
							    set dflt_pool $edge_pool
							}
						}
                        if { not [info exists dflt_pool] } {
                            set routepath [
                                string range $routepath 0 [
                                    expr {[string last "/" $routepath]-1}
                                ]
                            ]
							set wc_routepath [
                                string range $wc_routepath 0 [
                                    expr {[string last "/" $wc_routepath]-1}
                                ]
                            ]
                        }
                        else {
                            break
						}
					}
                }
				# handle the default pool for virtual server
				set default_class "/k8s-dev/Shared/payroll_gateway_443_default_pool_servername_dg"
                 if { [class exists $default_class] } {
                    set dflt_pool [class match -value "defaultPool" equals $default_class]
                 }
                # Handle requests sent to unknown hosts.
                # For valid hosts, Send the request to respective pool.
                if { not [info exists dflt_pool] } then {
                	 # Allowing HTTP2 traffic to be handled by policies and closing the connection for HTTP/1.1 unknown hosts.
                	 if { not ([SSL::payload] starts_with "PRI * HTTP/2.0") } {
                	    reject ; event disable all;
                        log local0.debug "Failed to find pool for $servername_lower"
                        return;
                    }
                } else {
                	pool $dflt_pool
                }
				set ab_class "/k8s-dev/Shared/payroll_gateway_443_ab_deployment_dg"
                if { [class exists $ab_class] } {
                    set selected_pool [call select_ab_pool $servername_lower $dflt_pool $domainpath]
                    if { $selected_pool == "" } then {
                        log local0.debug "Unable to find pool for $servername_lower"
                    } else {
                        pool $selected_pool
                    }
                }
            }
            SSL::release
        }
		when SERVER_CONNECTED {
		    log local0. "CUSTOM: SERVER_CONNECTED"
			set reencryptssl_class "/k8s-dev/Shared/payroll_gateway_443_ssl_reencrypt_serverssl_dg"
			set edgessl_class "/k8s-dev/Shared/payroll_gateway_443_ssl_edge_serverssl_dg"
			if { [info exists sslpath] and [class exists $reencryptssl_class] } {
				# Find the nearest child path which matches the reencrypt_class
				for {set i $rc} {$i >= 0} {incr i -1} {
					if { [class exists $reencryptssl_class] } {
						set reen [class match -value $sslpath equals $reencryptssl_class]
                        # check for wildcard domain match
                        if { $reen equals "" } {
						    if { [class match $wc_routepath equals $reencryptssl_class] } {
						        set reen [class match -value $wc_routepath equals $reencryptssl_class]
						    }
                        }
						if { not ($reen equals "") } {
							    set sslprofile $reen
						}
					}
					if { [class exists $edgessl_class] } {
						set edge [class match -value $sslpath equals $edgessl_class]
                        # check for wildcard domain match
                        if { $edge equals "" } {
						    if { [class match $wc_routepath equals $edgessl_class] } {
						        set edge [class match -value $wc_routepath equals $edgessl_class]
						    }
                        }
						if { not ($edge equals "") } {
							    set sslprofile $edge
						}
					}
					if { not [info exists sslprofile] } {
						set sslpath [
							string range $sslpath 0 [
								expr {[string last "/" $sslpath]-1}
							]
						]
                        set wc_routepaath [
							string range $wc_routepath 0 [
								expr {[string last "/" $wc_routepath]-1}
							]
						]
					}
					else {
						break
					}
				}
				# Assign respective SSL profile based on ssl_reencrypt_serverssl_dg
				if { not ($sslprofile equals "false") } {
				 		SSL::profile $reen
				}
			}
        }

In this configuration, when connecting to the VS we get an HTTP/2 stream error because the iRule is invalid.

By removing the last few lines of code from the iRule:

# Assign respective SSL profile based on ssl_reencrypt_serverssl_dg
# if { not ($sslprofile equals "false") } {
#   SSL::profile $reen
# }

Now this works fine and HTTP/2 full proxy is working as expected

[trinaths]

@shkarface Please share CIS configuration and error log, steps to reproduce this issue to automation_toolchain_pm automation_toolchain_pm@f5.com

[trinaths]

Created [CONTCNTR-4712] for internal tracking.

[shkarface]

Thank you, I will do so.

[vidyasagar-m]

@shkarface We are not able to reproduce this issue in local. Could you try with this build and confirm whether the error is still present

Build: cisbot/k8s-bigip-ctlr:browserUpdateIssue