Flux – Usage

Flagger: How it works

Mon, 01 Jan 0001 00:00:00 +0000

Flagger can be configured to automate the release process for Kubernetes workloads with a custom resource named canary.

Canary resource

The canary custom resource defines the release process of an application running on Kubernetes and is portable across clusters, service meshes and ingress providers.

For a deployment named podinfo, a canary release with progressive traffic shifting can be defined as:

apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
 name: podinfo
spec:
 targetRef:
 apiVersion: apps/v1
 kind: Deployment
 name: podinfo
 service:
 port: 9898
 analysis:
 interval: 1m
 threshold: 10
 maxWeight: 50
 stepWeight: 5
 metrics:
 - name: request-success-rate
 thresholdRange:
 min: 99
 interval: 1m
 - name: request-duration
 thresholdRange:
 max: 500
 interval: 1m
 webhooks:
 - name: load-test
 url: http://flagger-loadtester.test/
 metadata:
 cmd: "hey -z 1m -q 10 -c 2 http://podinfo-canary.test:9898/"

When you deploy a new version of an app, Flagger gradually shifts traffic to the canary, and at the same time, measures the requests success rate as well as the average response duration. You can extend the canary analysis with custom metrics, acceptance and load testing to harden the validation process of your app release process.

If you are running multiple service meshes or ingress controllers in the same cluster, you can override the global provider for a specific canary with spec.provider.

Canary target

A canary resource can target a Kubernetes Deployment or DaemonSet.

Kubernetes Deployment example:

spec:
 progressDeadlineSeconds: 60
 targetRef:
 apiVersion: apps/v1
 kind: Deployment
 name: podinfo
 autoscalerRef:
 apiVersion: autoscaling/v2beta2
 kind: HorizontalPodAutoscaler
 name: podinfo
 primaryScalerReplicas:
 minReplicas: 2
 maxReplicas: 5

Based on the above configuration, Flagger generates the following Kubernetes objects:

deployment/<targetRef.name>-primary
hpa/<autoscalerRef.name>-primary

The primary deployment is considered the stable release of your app, by default all traffic is routed to this version and the target deployment is scaled to zero. Flagger will detect changes to the target deployment (including secrets and configmaps) and will perform a canary analysis before promoting the new version as primary.

Use .spec.autoscalerRef.primaryScalerReplicas to override the replica scaling configuration for the generated primary HorizontalPodAutoscaler. This is useful for situations when you want to have a different scaling configuration for the primary workload as opposed to using the same values from the original workload HorizontalPodAutoscaler.

Note that the target deployment must have a single label selector in the format app: <DEPLOYMENT-NAME>:

apiVersion: apps/v1
kind: Deployment
metadata:
 name: podinfo
spec:
 selector:
 matchLabels:
 app: podinfo
 template:
 metadata:
 labels:
 app: podinfo

In addition to app, Flagger supports name and app.kubernetes.io/name selectors. If you use a different convention you can specify your label with the -selector-labels=my-app-label command flag in the Flagger deployment manifest under containers args or by setting --set selectorLabels=my-app-label when installing Flagger with Helm.

If the target deployment uses secrets and/or configmaps, Flagger will create a copy of each object using the -primary suffix and will reference these objects in the primary deployment. If you annotate your ConfigMap or Secret with flagger.app/config-tracking: disabled, Flagger will use the same object for the primary deployment instead of making a primary copy. You can disable the secrets/configmaps tracking globally with the -enable-config-tracking=false command flag in the Flagger deployment manifest under containers args or by setting --set configTracking.enabled=false when installing Flagger with Helm, but disabling config-tracking using the per Secret/ConfigMap annotation may fit your use-case better.

The autoscaler reference is optional, when specified, Flagger will pause the traffic increase while the target and primary deployments are scaled up or down. HPA can help reduce the resource usage during the canary analysis. When the autoscaler reference is specified, any changes made to the autoscaler are only made active in the primary autoscaler when a rollout for the deployment starts and completes successfully. Optionally, you can create two HPAs, one for canary and one for the primary to update the HPA without doing a new rollout. As the canary deployment will be scaled to 0, the HPA on the canary will be inactive.

Note Flagger requires autoscaling/v2 or autoscaling/v2beta2 API version for HPAs.

The progress deadline represents the maximum time in seconds for the canary deployment to make progress before it is rolled back, defaults to ten minutes.

Canary service

A canary resource dictates how the target workload is exposed inside the cluster. The canary target should expose a TCP port that will be used by Flagger to create the ClusterIP Services.

spec:
 service:
 name: podinfo
 port: 9898
 portName: http
 appProtocol: http
 targetPort: 9898
 portDiscovery: true

The container port from the target workload should match the service.port or service.targetPort. The service.name is optional, defaults to spec.targetRef.name. The service.targetPort can be a container port number or name. The service.portName is optional (defaults to http), if your workload uses gRPC then set the port name to grpc. The service.appProtocol is optional, more details can be found here.

If port discovery is enabled, Flagger scans the target workload and extracts the containers ports excluding the port specified in the canary service and service mesh sidecar ports. These ports will be used when generating the ClusterIP services.

Based on the canary spec service, Flagger creates the following Kubernetes ClusterIP service:

<service.name>.<namespace>.svc.cluster.local

selector app=<name>-primary
<service.name>-primary.<namespace>.svc.cluster.local

selector app=<name>-primary
<service.name>-canary.<namespace>.svc.cluster.local

selector app=<name>

This ensures that traffic to podinfo.test:9898 will be routed to the latest stable release of your app. The podinfo-canary.test:9898 address is available only during the canary analysis and can be used for conformance testing or load testing.

You can configure Flagger to set annotations and labels for the generated services with:

spec:
 service:
 port: 9898
 apex:
 annotations:
 test: "test"
 labels:
 test: "test"
 canary:
 annotations:
 test: "test"
 labels:
 test: "test"
 primary:
 annotations:
 test: "test"
 labels:
 test: "test"

Note that the apex annotations are added to both the generated Kubernetes Service and the generated service mesh/ingress object. This allows using external-dns with Istio VirtualServices and TraefikServices. Beware of configuration conflicts here.

Besides port mapping and metadata, the service specification can contain URI match and rewrite rules, timeout and retry polices:

spec:
 service:
 port: 9898
 match:
 - uri:
 prefix: /
 rewrite:
 uri: /
 retries:
 attempts: 3
 perTryTimeout: 1s
 timeout: 5s

When using Istio as the mesh provider, you can also specify HTTP header operations, CORS and traffic policies, Istio gateways and hosts. The Istio routing configuration can be found here.

Canary status

You can use kubectl to get the current status of canary deployments cluster wide:

kubectl get canaries --all-namespaces

NAMESPACE NAME STATUS WEIGHT LASTTRANSITIONTIME
test podinfo Progressing 15 2019-06-30T14:05:07Z
prod frontend Succeeded 0 2019-06-30T16:15:07Z
prod backend Failed 0 2019-06-30T17:05:07Z

The status condition reflects the last known state of the canary analysis:

kubectl -n test get canary/podinfo -oyaml | awk '/status/,0'

A successful rollout status:

status:
 canaryWeight: 0
 failedChecks: 0
 iterations: 0
 lastAppliedSpec: "14788816656920327485"
 lastPromotedSpec: "14788816656920327485"
 conditions:
 - lastTransitionTime: "2019-07-10T08:23:18Z"
 lastUpdateTime: "2019-07-10T08:23:18Z"
 message: Canary analysis completed successfully, promotion finished.
 reason: Succeeded
 status: "True"
 type: Promoted

The Promoted status condition can have one of the following reasons: Initialized, Waiting, Progressing, WaitingPromotion, Promoting, Finalising, Succeeded or Failed. A failed canary will have the promoted status set to false, the reason to failed and the last applied spec will be different to the last promoted one.

Wait for a successful rollout:

kubectl wait canary/podinfo --for=condition=promoted

CI example:

# update the container image
kubectl set image deployment/podinfo podinfod=stefanprodan/podinfo:3.0.1

# wait for Flagger to detect the change
ok=false
until ${ok}; do
 kubectl get canary/podinfo | grep 'Progressing' && ok=true || ok=false
 sleep 5
done

# wait for the canary analysis to finish
kubectl wait canary/podinfo --for=condition=promoted --timeout=5m

# check if the deployment was successful 
kubectl get canary/podinfo | grep Succeeded

Canary finalizers

The default behavior of Flagger on canary deletion is to leave resources that aren’t owned by the controller in their current state. This simplifies the deletion action and avoids possible deadlocks during resource finalization. In the event the canary was introduced with existing resource(s) (i.e. service, virtual service, etc.), they would be mutated during the initialization phase and no longer reflect their initial state. If the desired functionality upon deletion is to revert the resources to their initial state, the revertOnDeletion attribute can be enabled.

spec:
 revertOnDeletion: true

When a deletion action is submitted to the cluster, Flagger will attempt to revert the following resources:

Canary target replicas will be updated to the primary replica count
Canary service selector will be reverted
Mesh/Ingress traffic routed to the target

The recommended approach to disable canary analysis would be utilization of the skipAnalysis attribute, which limits the need for resource reconciliation. Utilizing the revertOnDeletion attribute should be enabled when you no longer plan to rely on Flagger for deployment management.

Note When this feature is enabled expect a delay in the delete action due to the reconciliation.

Canary analysis

The canary analysis defines:

the type of deployment strategy
the metrics used to validate the canary version
the webhooks used for conformance testing, load testing and manual gating
the alerting settings

Spec:

 analysis:
 # schedule interval (default 60s)
 interval:
 # max number of failed metric checks before rollback
 threshold:
 # max traffic percentage routed to canary
 # percentage (0-100)
 maxWeight:
 # canary increment step
 # percentage (0-100)
 stepWeight:
 # promotion increment step
 # percentage (0-100)
 stepWeightPromotion:
 # total number of iterations
 # used for A/B Testing and Blue/Green
 iterations:
 # threshold of primary pods that need to be available to consider it ready
 # before starting rollout. this is optional and the default is 100
 # percentage (0-100)
 primaryReadyThreshold: 100
 # threshold of canary pods that need to be available to consider it ready
 # before starting rollout. this is optional and the default is 100
 # percentage (0-100)
 canaryReadyThreshold: 100
 # canary match conditions
 # used for A/B Testing
 match:
 - # HTTP header
 # key performance indicators
 metrics:
 - # metric check
 # alerting
 alerts:
 - # alert provider
 # external checks
 webhooks:
 - # hook

The canary analysis runs periodically until it reaches the maximum traffic weight or the number of iterations. On each run, Flagger calls the webhooks, checks the metrics and if the failed checks threshold is reached, stops the analysis and rolls back the canary. If alerting is configured, Flagger will post the analysis result using the alert providers.

Canary suspend

The suspend field can be set to true to suspend the Canary. If a Canary is suspended, its reconciliation is completely paused. This means that changes to target workloads, tracked ConfigMaps and Secrets don’t trigger a Canary run and changes to resources generated by Flagger are not corrected. If the Canary was suspended during an active Canary run, then the run is paused without disturbing the workloads or the traffic weights.

Flagger: Deployment Strategies

Mon, 01 Jan 0001 00:00:00 +0000

Flagger can run automated application analysis, promotion and rollback for the following deployment strategies:

Canary Release (progressive traffic shifting)
- Istio, Linkerd, App Mesh, NGINX, Skipper, Contour, Gloo Edge, Traefik, Open Service Mesh, Kuma, Gateway API, Apache APISIX
A/B Testing (HTTP headers and cookies traffic routing)
- Istio, App Mesh, NGINX, Contour, Gloo Edge, Gateway API
Blue/Green (traffic switching)
- Kubernetes CNI, Istio, Linkerd, App Mesh, NGINX, Contour, Gloo Edge, Open Service Mesh, Gateway API, Apache APISIX
Blue/Green Mirroring (traffic shadowing)
- Istio
Canary Release with Session Affinity (progressive traffic shifting combined with cookie based routing)
- Istio

For Canary releases and A/B testing you’ll need a Layer 7 traffic management solution like a service mesh or an ingress controller. For Blue/Green deployments no service mesh or ingress controller is required.

A canary analysis is triggered by changes in any of the following objects:

Deployment PodSpec (container image, command, ports, env, resources, etc)
ConfigMaps mounted as volumes or mapped to environment variables
Secrets mounted as volumes or mapped to environment variables

Canary Release

Flagger implements a control loop that gradually shifts traffic to the canary while measuring key performance indicators like HTTP requests success rate, requests average duration and pod health. Based on analysis of the KPIs a canary is promoted or aborted.

The canary analysis runs periodically until it reaches the maximum traffic weight or the failed checks threshold.

Spec:

 analysis:
 # schedule interval (default 60s)
 interval: 1m
 # max number of failed metric checks before rollback
 threshold: 10
 # max traffic percentage routed to canary
 # percentage (0-100)
 maxWeight: 50
 # canary increment step
 # percentage (0-100)
 stepWeight: 2
 # promotion increment step (default 100)
 # percentage (0-100)
 stepWeightPromotion: 100
 # deploy straight to production without
 # the metrics and webhook checks
 skipAnalysis: false

The above analysis, if it succeeds, will run for 25 minutes while validating the HTTP metrics and webhooks every minute. You can determine the minimum time it takes to validate and promote a canary deployment using this formula:

interval * (maxWeight / stepWeight)

And the time it takes for a canary to be rollback when the metrics or webhook checks are failing:

interval * threshold

When stepWeightPromotion is specified, the promotion phase happens in stages, the traffic is routed back to the primary pods in a progressive manner, the primary weight is increased until it reaches 100%.

In emergency cases, you may want to skip the analysis phase and ship changes directly to production. At any time you can set the spec.skipAnalysis: true. When skip analysis is enabled, Flagger checks if the canary deployment is healthy and promotes it without analysing it. If an analysis is underway, Flagger cancels it and runs the promotion.

Gated canary promotion stages:

scan for canary deployments
check primary and canary deployment status
- halt advancement if a rolling update is underway
- halt advancement if pods are unhealthy
call confirm-rollout webhooks and check results
- halt advancement if any hook returns a non HTTP 2xx result
call pre-rollout webhooks and check results
- halt advancement if any hook returns a non HTTP 2xx result
- increment the failed checks counter
increase canary traffic weight percentage from 0% to 2% (step weight)
call rollout webhooks and check results
check canary HTTP request success rate and latency
- halt advancement if any metric is under the specified threshold
- increment the failed checks counter
check if the number of failed checks reached the threshold
- route all traffic to primary
- scale to zero the canary deployment and mark it as failed
- call post-rollout webhooks
- post the analysis result to Slack
- wait for the canary deployment to be updated and start over
increase canary traffic weight by 2% (step weight) till it reaches 50% (max weight)
- halt advancement if any webhook call fails
- halt advancement while canary request success rate is under the threshold
- halt advancement while canary request duration P99 is over the threshold
- halt advancement while any custom metric check fails
- halt advancement if the primary or canary deployment becomes unhealthy
- halt advancement while canary deployment is being scaled up/down by HPA
call confirm-promotion webhooks and check results
- halt advancement if any hook returns a non HTTP 2xx result
promote canary to primary
- copy ConfigMaps and Secrets from canary to primary
- copy canary deployment spec template over primary
wait for primary rolling update to finish
- halt advancement if pods are unhealthy
route all traffic to primary
scale to zero the canary deployment
mark rollout as finished
call post-rollout webhooks
send notification with the canary analysis result
wait for the canary deployment to be updated and start over

Rollout Weights

By default Flagger uses linear weight values for the promotion, with the start value, the step and the maximum weight value in 0 to 100 range.

Example:

canary:
 analysis:
 promotion:
 maxWeight: 50
 stepWeight: 20

This configuration performs analysis starting from 20, increasing by 20 until weight goes above 50.
We would have steps (canary weight : primary weight):

20 (20 : 80)
40 (40 : 60)
60 (60 : 40)
promotion

In order to enable non-linear promotion a new parameter was introduced:

stepWeights - determines the ordered array of weights, which shall be used during canary promotion.

Example:

canary:
 analysis:
 promotion:
 stepWeights: [1, 2, 10, 80]

This configuration performs analysis starting from 1, going through stepWeights values till 80.
We would have steps (canary weight : primary weight):

1 (1 : 99)
2 (2 : 98)
10 (10 : 90)
80 (80 : 20)
promotion

A/B Testing

For frontend applications that require session affinity you should use HTTP headers or cookies match conditions to ensure a set of users will stay on the same version for the whole duration of the canary analysis.

You can enable A/B testing by specifying the HTTP match conditions and the number of iterations. If Flagger finds a HTTP match condition, it will ignore the maxWeight and stepWeight settings.

Istio example:

 analysis:
 # schedule interval (default 60s)
 interval: 1m
 # total number of iterations
 iterations: 10
 # max number of failed iterations before rollback
 threshold: 2
 # canary match condition
 match:
 - headers:
 x-canary:
 regex: ".*insider.*"
 - headers:
 cookie:
 regex: "^(.*?;)?(canary=always)(;.*)?$"

The above configuration will run an analysis for ten minutes targeting the Safari users and those that have a test cookie. You can determine the minimum time that it takes to validate and promote a canary deployment using this formula:

interval * iterations

And the time it takes for a canary to be rollback when the metrics or webhook checks are failing:

interval * threshold

Istio example:

 analysis:
 interval: 1m
 threshold: 10
 iterations: 2
 match:
 - headers:
 x-canary:
 exact: "insider"
 - headers:
 cookie:
 regex: "^(.*?;)?(canary=always)(;.*)?$"
 - sourceLabels:
 app.kubernetes.io/name: "scheduler"

The header keys must be lowercase and use hyphen as the separator. Header values are case-sensitive and formatted as follows:

exact: "value" for exact string match
prefix: "value" for prefix-based match
suffix: "value" for suffix-based match
regex: "value" for RE2 style regex-based match

Note that the sourceLabels match conditions are applicable only when the mesh gateway is included in the canary.service.gateways list.

App Mesh example:

 analysis:
 interval: 1m
 threshold: 10
 iterations: 2
 match:
 - headers:
 user-agent:
 regex: ".*Chrome.*"

Note that App Mesh supports a single condition.

Contour example:

 analysis:
 interval: 1m
 threshold: 10
 iterations: 2
 match:
 - headers:
 user-agent:
 prefix: "Chrome"

Note that Contour does not support regex, you can use prefix, suffix or exact.

NGINX example:

 analysis:
 interval: 1m
 threshold: 10
 iterations: 2
 match:
 - headers:
 x-canary:
 exact: "insider"
 - headers:
 cookie:
 exact: "canary"

Note that the NGINX ingress controller supports only exact matching for cookies names where the value must be set to always. Starting with NGINX ingress v0.31, regex matching is supported for header values.

The above configurations will route users with the x-canary header or canary cookie to the canary instance during analysis:

curl -H 'X-Canary: insider' http://app.example.com
curl -b 'canary=always' http://app.example.com

Blue/Green Deployments

For applications that are not deployed on a service mesh, Flagger can orchestrate blue/green style deployments with Kubernetes L4 networking. When using Istio you have the option to mirror traffic between blue and green.

You can use the blue/green deployment strategy by replacing stepWeight/maxWeight with iterations in the analysis spec:

 analysis:
 # schedule interval (default 60s)
 interval: 1m
 # total number of iterations
 iterations: 10
 # max number of failed iterations before rollback
 threshold: 2

With the above configuration Flagger will run conformance and load tests on the canary pods for ten minutes. If the metrics analysis succeeds, live traffic will be switched from the old version to the new one when the canary is promoted.

The blue/green deployment strategy is supported for all service mesh providers.

Blue/Green rollout steps for service mesh:

detect new revision (deployment spec, secrets or configmaps changes)
scale up the canary (green)
run conformance tests for the canary pods
run load tests and metric checks for the canary pods every minute
abort the canary release if the failure threshold is reached
route traffic to canary (This doesn’t happen when using the kubernetes provider)
promote canary spec over primary (blue)
wait for primary rollout
route traffic to primary
scale down canary

After the analysis finishes, the traffic is routed to the canary (green) before triggering the primary (blue) rolling update, this ensures a smooth transition to the new version avoiding dropping in-flight requests during the Kubernetes deployment rollout.

Blue/Green with Traffic Mirroring

Traffic Mirroring is a pre-stage in a Canary (progressive traffic shifting) or Blue/Green deployment strategy. Traffic mirroring will copy each incoming request, sending one request to the primary and one to the canary service. The response from the primary is sent back to the user. The response from the canary is discarded. Metrics are collected on both requests so that the deployment will only proceed if the canary metrics are healthy.

Mirroring should be used for requests that are idempotent or capable of being processed twice (once by the primary and once by the canary). Reads are idempotent. Before using mirroring on requests that may be writes, you should consider what will happen if a write is duplicated and handled by the primary and canary.

To use mirroring, set spec.analysis.mirror to true.

Istio example:

 analysis:
 # schedule interval (default 60s)
 interval: 1m
 # total number of iterations
 iterations: 10
 # max number of failed iterations before rollback
 threshold: 2
 # Traffic shadowing (compatible with Istio only)
 mirror: true
 # Weight of the traffic mirrored to your canary (defaults to 100%)
 mirrorWeight: 100

Mirroring rollout steps for service mesh:

detect new revision (deployment spec, secrets or configmaps changes)
scale from zero the canary deployment
wait for the HPA to set the canary minimum replicas
check canary pods health
run the acceptance tests
abort the canary release if tests fail
start the load tests
mirror 100% of the traffic from primary to canary
check request success rate and request duration every minute
abort the canary release if the failure threshold is reached
stop traffic mirroring after the number of iterations is reached
route live traffic to the canary pods
promote the canary (update the primary secrets, configmaps and deployment spec)
wait for the primary deployment rollout to finish
wait for the HPA to set the primary minimum replicas
check primary pods health
switch live traffic back to primary
scale to zero the canary
send notification with the canary analysis result

Canary Release with Session Affinity

This deployment strategy mixes a Canary Release with A/B testing. A Canary Release is helpful when we’re trying to expose new features to users progressively, but because of the very nature of its routing (weight based), users can land on the application’s old version even after they have been routed to the new version previously. This can be annoying, or worse break how other services interact with our application. To address this issue, we borrow some things from A/B testing.

Since A/B testing is particularly helpful for applications that require session affinity, we integrate cookie based routing with regular weight based routing. This means once a user is exposed to the new version of our application (based on the traffic weights), they’re always routed to that version, i.e. they’re never routed back to the old version of our application.

You can enable this, by specifying .spec.analsyis.sessionAffinity in the Canary (only Istio is supported):

 analysis:
 # schedule interval (default 60s)
 interval: 1m
 # max number of failed metric checks before rollback
 threshold: 10
 # max traffic percentage routed to canary
 # percentage (0-100)
 maxWeight: 50
 # canary increment step
 # percentage (0-100)
 stepWeight: 2
 # session affinity config
 sessionAffinity:
 # name of the cookie used
 cookieName: flagger-cookie
 # max age of the cookie (in seconds)
 # optional; defaults to 86400
 maxAge: 21600

.spec.analysis.sessionAffinity.cookieName is the name of the Cookie that is stored. The value of the cookie is a randomly generated string of characters that act as a unique identifier. For the above config, the response header of a request routed to the canary deployment during a Canary run will look like:

Set-Cookie: flagger-cookie=LpsIaLdoNZ; Max-Age=21600

After a Canary run is over and all traffic is shifted back to the primary deployment, all responses will have the following header:

Set-Cookie: flagger-cookie=LpsIaLdoNZ; Max-Age=-1

This tells the client to delete the cookie, making sure there are no junk cookies lying around in the user’s system.

If a new Canary run is triggered, the response header will set a new cookie for all requests routed to the Canary deployment:

Set-Cookie: flagger-cookie=McxKdLQoIN; Max-Age=21600

Flagger: Metrics Analysis

Mon, 01 Jan 0001 00:00:00 +0000

As part of the analysis process, Flagger can validate service level objectives (SLOs) like availability, error rate percentage, average response time and any other objective based on app specific metrics. If a drop in performance is noticed during the SLOs analysis, the release will be automatically rolled back with minimum impact to end-users.

Builtin metrics

Flagger comes with two builtin metric checks: HTTP request success rate and duration.

 analysis:
 metrics:
 - name: request-success-rate
 interval: 1m
 # minimum req success rate (non 5xx responses)
 # percentage (0-100)
 thresholdRange:
 min: 99
 - name: request-duration
 interval: 1m
 # maximum req duration P99
 # milliseconds
 thresholdRange:
 max: 500

For each metric you can specify a range of accepted values with thresholdRange and the window size or the time series with interval. The builtin checks are available for every service mesh / ingress controller and are implemented with Prometheus queries.

Custom metrics

The canary analysis can be extended with custom metric checks. Using a MetricTemplate custom resource, you configure Flagger to connect to a metric provider and run a query that returns a float64 value. The query result is used to validate the canary based on the specified threshold range.

apiVersion: flagger.app/v1beta1
kind: MetricTemplate
metadata:
 name: my-metric
spec:
 provider:
 type: # can be prometheus, datadog, etc
 address: # API URL
 insecureSkipVerify: # if set to true, disables the TLS cert validation
 secretRef:
 name: # name of the secret containing the API credentials
 query: # metric query

The following variables are available in query templates:

name (canary.metadata.name)
namespace (canary.metadata.namespace)
target (canary.spec.targetRef.name)
service (canary.spec.service.name)
ingress (canary.spec.ingresRef.name)
interval (canary.spec.analysis.metrics[].interval)
variables (canary.spec.analysis.metrics[].templateVariables)

A canary analysis metric can reference a template with templateRef:

 analysis:
 metrics:
 - name: "my metric"
 templateRef:
 name: my-metric
 # namespace is optional
 # when not specified, the canary namespace will be used
 namespace: flagger
 # accepted values
 thresholdRange:
 min: 10
 max: 1000
 # metric query time window
 interval: 1m

A canary analysis metric can reference a set of custom variables with templateVariables. These variables will be then injected into the query defined in the referred MetricTemplate object during canary analysis:

 analysis:
 metrics:
 - name: "my metric"
 templateRef:
 name: my-metric
 namespace: flagger
 # accepted values
 thresholdRange:
 min: 10
 max: 1000
 # metric query time window
 interval: 1m
 # custom variables used within the referenced metric template
 templateVariables:
 direction: inbound

apiVersion: flagger.app/v1beta1
kind: MetricTemplate
metadata:
 name: my-metric
spec:
 provider:
 type: prometheus
 address: http://prometheus.linkerd-viz:9090
 query: |
 histogram_quantile(
 0.99,
 sum(
 rate(
 response_latency_ms_bucket{
 namespace="{{ namespace }}",
 deployment=~"{{ target }}",
 direction="{{ variables.direction }}"
 }[{{ interval }}]
 )
 ) by (le)
 )

Prometheus

You can create custom metric checks targeting a Prometheus server by setting the provider type to prometheus and writing the query in PromQL.

Prometheus template example:

apiVersion: flagger.app/v1beta1
kind: MetricTemplate
metadata:
 name: not-found-percentage
 namespace: istio-system
spec:
 provider:
 type: prometheus
 address: http://prometheus.istio-system:9090
 query: |
 100 - sum(
 rate(
 istio_requests_total{
 reporter="destination",
 destination_workload_namespace="{{ namespace }}",
 destination_workload="{{ target }}",
 response_code!="404"
 }[{{ interval }}]
 )
 )
 /
 sum(
 rate(
 istio_requests_total{
 reporter="destination",
 destination_workload_namespace="{{ namespace }}",
 destination_workload="{{ target }}"
 }[{{ interval }}]
 )
 ) * 100

Reference the template in the canary analysis:

 analysis:
 metrics:
 - name: "404s percentage"
 templateRef:
 name: not-found-percentage
 namespace: istio-system
 thresholdRange:
 max: 5
 interval: 1m

The above configuration validates the canary by checking if the HTTP 404 req/sec percentage is below 5 percent of the total traffic. If the 404s rate reaches the 5% threshold, then the canary fails.

Prometheus gRPC error rate example:

apiVersion: flagger.app/v1beta1
kind: MetricTemplate
metadata:
 name: grpc-error-rate-percentage
 namespace: flagger
spec:
 provider:
 type: prometheus
 address: http://flagger-prometheus.flagger-system:9090
 query: |
 100 - sum(
 rate(
 grpc_server_handled_total{
 grpc_code!="OK",
 kubernetes_namespace="{{ namespace }}",
 kubernetes_pod_name=~"{{ target }}-[0-9a-zA-Z]+(-[0-9a-zA-Z]+)"
 }[{{ interval }}]
 )
 )
 /
 sum(
 rate(
 grpc_server_started_total{
 kubernetes_namespace="{{ namespace }}",
 kubernetes_pod_name=~"{{ target }}-[0-9a-zA-Z]+(-[0-9a-zA-Z]+)"
 }[{{ interval }}]
 )
 ) * 100

The above template is for gRPC services instrumented with go-grpc-prometheus.

Prometheus authentication

If your Prometheus API requires basic authentication, you can create a secret in the same namespace as the MetricTemplate with the basic-auth credentials:

apiVersion: v1
kind: Secret
metadata:
 name: prom-auth
 namespace: flagger
data:
 username: your-user
 password: your-password

or if you require bearer token authentication (via a SA token):

apiVersion: v1
kind: Secret
metadata:
 name: prom-auth
 namespace: flagger
data:
 token: ey1234...

Then reference the secret in the MetricTemplate:

apiVersion: flagger.app/v1beta1
kind: MetricTemplate
metadata:
 name: my-metric
 namespace: flagger
spec:
 provider:
 type: prometheus
 address: http://prometheus.monitoring:9090
 secretRef:
 name: prom-auth

Datadog

You can create custom metric checks using the Datadog provider.

Create a secret with your Datadog API credentials:

apiVersion: v1
kind: Secret
metadata:
 name: datadog
 namespace: istio-system
data:
 datadog_api_key: your-datadog-api-key
 datadog_application_key: your-datadog-application-key

Datadog template example:

apiVersion: flagger.app/v1beta1
kind: MetricTemplate
metadata:
 name: not-found-percentage
 namespace: istio-system
spec:
 provider:
 type: datadog
 address: https://api.datadoghq.com
 secretRef:
 name: datadog
 query: |
 100 - (
 sum:istio.mesh.request.count{
 reporter:destination,
 destination_workload_namespace:{{ namespace }},
 destination_workload:{{ target }},
 !response_code:404
 }.as_count()
 /
 sum:istio.mesh.request.count{
 reporter:destination,
 destination_workload_namespace:{{ namespace }},
 destination_workload:{{ target }}
 }.as_count()
 ) * 100

Reference the template in the canary analysis:

 analysis:
 metrics:
 - name: "404s percentage"
 templateRef:
 name: not-found-percentage
 namespace: istio-system
 thresholdRange:
 max: 5
 interval: 1m

Amazon CloudWatch

You can create custom metric checks using the CloudWatch metrics provider.

CloudWatch template example:

apiVersion: flagger.app/v1alpha1
kind: MetricTemplate
metadata:
 name: cloudwatch-error-rate
spec:
 provider:
 type: cloudwatch
 region: ap-northeast-1 # specify the region of your metrics
 query: |
 [
 {
 "Id": "e1",
 "Expression": "m1 / m2",
 "Label": "ErrorRate"
 },
 {
 "Id": "m1",
 "MetricStat": {
 "Metric": {
 "Namespace": "MyKubernetesCluster",
 "MetricName": "ErrorCount",
 "Dimensions": [
 {
 "Name": "appName",
 "Value": "{{ name }}.{{ namespace }}"
 }
 ]
 },
 "Period": 60,
 "Stat": "Sum",
 "Unit": "Count"
 },
 "ReturnData": false
 },
 {
 "Id": "m2",
 "MetricStat": {
 "Metric": {
 "Namespace": "MyKubernetesCluster",
 "MetricName": "RequestCount",
 "Dimensions": [
 {
 "Name": "appName",
 "Value": "{{ name }}.{{ namespace }}"
 }
 ]
 },
 "Period": 60,
 "Stat": "Sum",
 "Unit": "Count"
 },
 "ReturnData": false
 }
 ]

The query format documentation can be found here.

Reference the template in the canary analysis:

 analysis:
 metrics:
 - name: "app error rate"
 templateRef:
 name: cloudwatch-error-rate
 thresholdRange:
 max: 0.1
 interval: 1m

Note that Flagger need AWS IAM permission to perform cloudwatch:GetMetricData to use this provider.

New Relic

You can create custom metric checks using the New Relic provider.

Create a secret with your New Relic Insights credentials:

apiVersion: v1
kind: Secret
metadata:
 name: newrelic
 namespace: istio-system
data:
 newrelic_account_id: your-account-id
 newrelic_query_key: your-insights-query-key

New Relic template example:

apiVersion: flagger.app/v1beta1
kind: MetricTemplate
metadata:
 name: newrelic-error-rate
 namespace: ingress-nginx
spec:
 provider:
 type: newrelic
 secretRef:
 name: newrelic
 query: |
 SELECT
 filter(sum(nginx_ingress_controller_requests), WHERE status >= '500') /
 sum(nginx_ingress_controller_requests) * 100
 FROM Metric
 WHERE metricName = 'nginx_ingress_controller_requests'
 AND ingress = '{{ ingress }}' AND namespace = '{{ namespace }}'

Reference the template in the canary analysis:

 analysis:
 metrics:
 - name: "error rate"
 templateRef:
 name: newrelic-error-rate
 namespace: ingress-nginx
 thresholdRange:
 max: 5
 interval: 1m

Graphite

You can create custom metric checks using the Graphite provider.

Graphite template example:

apiVersion: flagger.app/v1beta1
kind: MetricTemplate
metadata:
 name: graphite-request-success-rate
spec:
 provider:
 type: graphite
 address: http://graphite.monitoring
 query: |
 target=summarize(
 asPercent(
 sumSeries(
 stats.timers.httpServerRequests.app.{{target}}.exception.*.method.*.outcome.{CLIENT_ERROR,INFORMATIONAL,REDIRECTION,SUCCESS}.status.*.uri.*.count
 ),
 sumSeries(
 stats.timers.httpServerRequests.app.{{target}}.exception.*.method.*.outcome.*.status.*.uri.*.count
 )
 ),
 {{interval}},
 'avg'
 )

Reference the template in the canary analysis:

 analysis:
 metrics:
 - name: "success rate"
 templateRef:
 name: graphite-request-success-rate
 thresholdRange:
 min: 90
 interval: 1min

Graphite authentication

If your Graphite API requires basic authentication, you can create a secret in the same namespace as the MetricTemplate with the basic-auth credentials:

apiVersion: v1
kind: Secret
metadata:
 name: graphite-basic-auth
 namespace: flagger
data:
 username: your-user
 password: your-password

Then, reference the secret in the MetricTemplate:

apiVersion: flagger.app/v1beta1
kind: MetricTemplate
metadata:
 name: my-metric
 namespace: flagger
spec:
 provider:
 type: graphite
 address: http://graphite.monitoring
 secretRef:
 name: graphite-basic-auth

Google Cloud Monitoring (Stackdriver)

Enable Workload Identity on your cluster, create a service account key that has read access to the Cloud Monitoring API and then create an IAM policy binding between the GCP service account and the Flagger service account on Kubernetes. You can take a look at this guide

Annotate the flagger service account

kubectl annotate serviceaccount flagger \
 --namespace <namespace> \
 iam.gke.io/gcp-service-account=<gcp-serviceaccount-name>@<project-id>.iam.gserviceaccount.com

Alternatively, you can download the json keys and add it to your secret with the key serviceAccountKey (This method is not recommended).

Create a secret that contains your project-id (and, if workload identity is not enabled on your cluster, your service account json).

 kubectl create secret generic gcloud-sa --from-literal=project=<project-id>

Then reference the secret in the metric template. Note: The particular MQL query used here works if Istio is installed on GKE.

apiVersion: flagger.app/v1beta1
kind: MetricTemplate
metadata:
 name: bytes-sent
 namespace: test
spec:
 provider:
 type: stackdriver
 secretRef:
 name: gcloud-sa
 query: |
 fetch k8s_container
 | metric 'istio.io/service/server/response_latencies'
 | filter
 (metric.destination_service_name == '{{ service }}-canary'
 && metric.destination_service_namespace == '{{ namespace }}')
 | align delta(1m)
 | every 1m
 | group_by [],
 [value_response_latencies_percentile:
 percentile(value.response_latencies, 99)]

The reference for the query language can be found here

InfluxDB

The InfluxDB provider uses the flux query language.

Create a secret that contains your authentication token that can be found in the InfluxDB UI.

 kubectl create secret generic influx-token --from-literal=token=<token>

Then reference the secret in the metric template.

Note: The particular MQL query used here works if Istio is installed on GKE.

apiVersion: flagger.app/v1beta1
kind: MetricTemplate
metadata:
 name: not-found
 namespace: test
spec:
 provider:
 type: influxdb
 secretRef:
 name: influx-token
 query: |
 from(bucket: "default")
 |> range(start: -2h)
 |> filter(fn: (r) => r["_measurement"] == "istio_requests_total")
 |> filter(fn: (r) => r[" destination_workload_namespace"] == "{{ namespace }}")
 |> filter(fn: (r) => r["destination_workload"] == "{{ target }}")
 |> filter(fn: (r) => r["response_code"] == "500")
 |> count()
 |> yield(name: "count")

Dynatrace

You can create custom metric checks using the Dynatrace provider.

Create a secret with your Dynatrace token:

apiVersion: v1
kind: Secret
metadata:
 name: dynatrace
 namespace: istio-system
data:
 dynatrace_token: ZHQwYz...

Dynatrace metric template example:

apiVersion: flagger.app/v1beta1
kind: MetricTemplate
metadata:
 name: response-time-95pct
 namespace: istio-system
spec:
 provider:
 type: dynatrace
 address: https://xxxxxxxx.live.dynatrace.com
 secretRef:
 name: dynatrace
 query: |
 builtin:service.response.time:filter(eq(dt.entity.service,SERVICE-ABCDEFG0123456789)):percentile(95)

Reference the template in the canary analysis:

 analysis:
 metrics:
 - name: "response-time-95pct"
 templateRef:
 name: response-time-95pct
 namespace: istio-system
 thresholdRange:
 max: 1000
 interval: 1m

Flagger: Webhooks

Mon, 01 Jan 0001 00:00:00 +0000

The canary analysis can be extended with webhooks. Flagger will call each webhook URL and determine from the response status code (HTTP 2xx) if the canary is failing or not.

There are several types of hooks:

confirm-rollout hooks are executed before scaling up the canary deployment and can be used for manual approval. The rollout is paused until the hook returns a successful HTTP status code.
pre-rollout hooks are executed before routing traffic to canary. The canary advancement is paused if a pre-rollout hook fails and if the number of failures reach the threshold the canary will be rollback.
rollout hooks are executed during the analysis on each iteration before the metric checks. If a rollout hook call fails the canary advancement is paused and eventfully rolled back.
confirm-traffic-increase hooks are executed right before the weight on the canary is increased. The canary advancement is paused until this hook returns HTTP 200.
confirm-promotion hooks are executed before the promotion step. The canary promotion is paused until the hooks return HTTP 200. While the promotion is paused, Flagger will continue to run the metrics checks and rollout hooks.
post-rollout hooks are executed after the canary has been promoted or rolled back. If a post rollout hook fails the error is logged.
rollback hooks are executed while a canary deployment is in either Progressing or Waiting status. This provides the ability to rollback during analysis or while waiting for a confirmation. If a rollback hook returns a successful HTTP status code, Flagger will stop the analysis and mark the canary release as failed.
event hooks are executed every time Flagger emits a Kubernetes event. When configured, every action that Flagger takes during a canary deployment will be sent as JSON via an HTTP POST request.

Spec:

 analysis:
 webhooks:
 - name: "start gate"
 type: confirm-rollout
 url: http://flagger-loadtester.test/gate/approve
 - name: "helm test"
 type: pre-rollout
 url: http://flagger-helmtester.flagger/
 timeout: 3m
 metadata:
 type: "helmv3"
 cmd: "test podinfo -n test"
 - name: "load test"
 type: rollout
 url: http://flagger-loadtester.test/
 timeout: 15s
 metadata:
 cmd: "hey -z 1m -q 5 -c 2 http://podinfo-canary.test:9898/"
 - name: "traffic increase gate"
 type: confirm-traffic-increase
 url: http://flagger-loadtester.test/gate/approve
 - name: "promotion gate"
 type: confirm-promotion
 url: http://flagger-loadtester.test/gate/approve
 - name: "notify"
 type: post-rollout
 url: http://telegram.bot:8080/
 timeout: 5s
 metadata:
 some: "message"
 - name: "rollback gate"
 type: rollback
 url: http://flagger-loadtester.test/rollback/check
 - name: "send to Slack"
 type: event
 url: http://event-recevier.notifications/slack
 metadata:
 environment: "test"
 cluster: "flagger-test"

Note that the sum of all rollout webhooks timeouts should be lower than the analysis interval.

Webhook payload (HTTP POST):

{
 "name": "podinfo",
 "namespace": "test",
 "phase": "Progressing",
 "metadata": {
 "test": "all",
 "token": "16688eb5e9f289f1991c"
 }
}

Response status codes:

200-202 - advance canary by increasing the traffic weight
timeout or non-2xx - halt advancement and increment failed checks

On a non-2xx response Flagger will include the response body (if any) in the failed checks log and Kubernetes events.

Event payload (HTTP POST):

{
 "name": "string (canary name)",
 "namespace": "string (canary namespace)",
 "phase": "string (canary phase)",
 "metadata": {
 "eventMessage": "string (canary event message)",
 "eventType": "string (canary event type)",
 "timestamp": "string (unix timestamp ms)"
 }
}

The event receiver can create alerts based on the received phase (possible values: Initialized, Waiting, Progressing, Promoting, Finalising, Succeeded or Failed).

Load Testing

For workloads that are not receiving constant traffic Flagger can be configured with a webhook, that when called, will start a load test for the target workload. If the target workload doesn’t receive any traffic during the canary analysis, Flagger metric checks will fail with “no values found for metric request-success-rate”.

Flagger comes with a load testing service based on rakyll/hey that generates traffic during analysis when configured as a webhook.

First you need to deploy the load test runner in a namespace with sidecar injection enabled:

kubectl apply -k https://github.com/fluxcd/flagger//kustomize/tester?ref=main

Or by using Helm:

helm repo add flagger https://flagger.app

helm upgrade -i flagger-loadtester flagger/loadtester \
--namespace=test \
--set cmd.timeout=1h \
--set cmd.namespaceRegexp=''

When deployed the load tester API will be available at http://flagger-loadtester.test/.

Now you can add webhooks to the canary analysis spec:

webhooks:
 - name: load-test-get
 url: http://flagger-loadtester.test/
 timeout: 5s
 metadata:
 type: cmd
 cmd: "hey -z 1m -q 10 -c 2 http://podinfo-canary.test:9898/"
 - name: load-test-post
 url: http://flagger-loadtester.test/
 timeout: 5s
 metadata:
 type: cmd
 cmd: "hey -z 1m -q 10 -c 2 -m POST -d '{test: 2}' http://podinfo-canary.test:9898/echo"

When the canary analysis starts, Flagger will call the webhooks and the load tester will run the hey commands in the background, if they are not already running. This will ensure that during the analysis, the podinfo-canary.test service will receive a steady stream of GET and POST requests.

If your workload is exposed outside the mesh you can point hey to the public URL and use HTTP2.

webhooks:
 - name: load-test-get
 url: http://flagger-loadtester.test/
 timeout: 5s
 metadata:
 type: cmd
 cmd: "hey -z 1m -q 10 -c 2 -h2 https://podinfo.example.com/"

For gRPC services you can use bojand/ghz which is a similar tool to Hey but for gRPC:

webhooks:
 - name: grpc-load-test
 url: http://flagger-loadtester.test/
 timeout: 5s
 metadata:
 type: cmd
 cmd: "ghz -z 1m -q 10 -c 2 --insecure podinfo.test:9898"

ghz uses reflection to identify which gRPC method to call. If you do not wish to enable reflection for your gRPC service you can implement a standardized health check from the grpc-proto library. To use this health check schema without reflection you can pass a parameter to ghz like this

webhooks:
 - name: grpc-load-test-no-reflection
 url: http://flagger-loadtester.test/
 timeout: 5s
 metadata:
 type: cmd
 cmd: "ghz --insecure --proto=/tmp/ghz/health.proto --call=grpc.health.v1.Health/Check podinfo.test:9898"

The load tester can run arbitrary commands as long as the binary is present in the container image. For example if you want to replace hey with another CLI, you can create your own Docker image:

FROM weaveworks/flagger-loadtester:<VER>

RUN curl -Lo /usr/local/bin/my-cli https://github.com/user/repo/releases/download/ver/my-cli \
 && chmod +x /usr/local/bin/my-cli

Load Testing Delegation

The load tester can also forward testing tasks to external tools, by now nGrinder is supported.

To use this feature, add a load test task of type ’ngrinder’ to the canary analysis spec:

webhooks:
 - name: load-test-post
 url: http://flagger-loadtester.test/
 timeout: 5s
 metadata:
 # type of this load test task, cmd or ngrinder
 type: ngrinder
 # base url of your nGrinder controller server
 server: http://ngrinder-server:port
 # id of the test to clone from, the test must have been defined.
 clone: 100
 # user name and base64 encoded password to authenticate against the nGrinder server
 username: admin
 passwd: YWRtaW4=
 # the interval between between nGrinder test status polling, default to 1s
 pollInterval: 5s

When the canary analysis starts, the load tester will initiate a clone_and_start request to the nGrinder server and start a new performance test. the load tester will periodically poll the nGrinder server for the status of the test, and prevent duplicate requests from being sent in subsequent analysis loops.

K6 Load Tester

You can also delegate load testing to a third-party webhook. An example of this is the k6 webhook. This webhook uses k6, a very featureful load tester, to run load or smoke tests on canaries. For all features available, see the source repository.

Here’s an example integrating this webhook as a pre-rollout step, to load test a service before any traffic is sent to it:

webhooks:
- name: k6-load-test
 timeout: 5m
 type: pre-rollout
 url: http://k6-loadtester.flagger/launch-test
 metadata:
 script: |
 import http from 'k6/http';
 import { sleep } from 'k6';
 export const options = {
 vus: 2,
 duration: '30s',
 thresholds: {
 http_req_duration: ['p(95)<50']
 },
 ext: {
 loadimpact: {
 name: '<cluster>/<your_service>',
 projectID: <project id>,
 },
 },
 };

 export default function () {
 http.get('http://<your_service>-canary.<namespace>:80/');
 sleep(0.10);
 }

Integration Testing

Flagger comes with a testing service that can run Helm tests, Bats tests or Concord tests when configured as a webhook.

Deploy the Helm test runner in the kube-system namespace using the tiller service account:

helm repo add flagger https://flagger.app

helm upgrade -i flagger-helmtester flagger/loadtester \
--namespace=kube-system \
--set serviceAccountName=tiller

When deployed the Helm tester API will be available at http://flagger-helmtester.kube-system/.

Now you can add pre-rollout webhooks to the canary analysis spec:

 analysis:
 webhooks:
 - name: "smoke test"
 type: pre-rollout
 url: http://flagger-helmtester.kube-system/
 timeout: 3m
 metadata:
 type: "helm"
 cmd: "test {{ .Release.Name }} --cleanup"

When the canary analysis starts, Flagger will call the pre-rollout webhooks before routing traffic to the canary. If the helm test fails, Flagger will retry until the analysis threshold is reached and the canary is rolled back.

If you are using Helm v3, you’ll have to create a dedicated service account and add the release namespace to the test command:

 analysis:
 webhooks:
 - name: "smoke test"
 type: pre-rollout
 url: http://flagger-helmtester.kube-system/
 timeout: 3m
 metadata:
 type: "helmv3"
 cmd: "test {{ .Release.Name }} --timeout 3m -n {{ .Release.Namespace }}"

If the test hangs or logs error messages hinting to insufficient permissions it can be related to RBAC, check the Troubleshooting section for an example configuration.

As an alternative to Helm you can use the Bash Automated Testing System to run your tests.

 analysis:
 webhooks:
 - name: "acceptance tests"
 type: pre-rollout
 url: http://flagger-batstester.default/
 timeout: 5m
 metadata:
 type: "bash"
 cmd: "bats /tests/acceptance.bats"

Note that you should create a ConfigMap with your Bats tests and mount it inside the tester container.

You can also configure the test runner to start a Concord process.

 analysis:
 webhooks:
 - name: "concord integration test"
 type: pre-rollout
 url: http://flagger-concordtester.default/
 timeout: 60s
 metadata:
 type: "concord"
 org: "your-concord-org"
 project: "your-concord-project"
 repo: "your-concord-repo"
 entrypoint: "your-concord-entrypoint"
 apiKeyPath: "/tmp/concord-api-key"
 endpoint: "https://canary-endpoint/"
 pollInterval: "5"
 pollTimeout: "60"

org, project, repo and entrypoint represents where your test process runs in Concord. In order to authenticate to Concord, you need to set apiKeyPath to a path of a file containing a valid Concord API key on the flagger-helmtester container. This can be done via mounting a Kubernetes secret in the tester’s Deployment. pollInterval represents the interval in seconds the web-hook will call Concord to see if the process has finished (Default is 5s). pollTimeout represents the time in seconds the web-hook will try to call Concord before timing out (Default is 30s).

Manual Gating

For manual approval of a canary deployment you can use the confirm-rollout and confirm-promotion webhooks. The confirmation rollout hooks are executed before the pre-rollout hooks. For manually approving traffic weight increase, you can use the confirm-traffic-increase webhook. Flagger will halt the canary traffic shifting and analysis until the confirm webhook returns HTTP status 200.

For manual rollback of a canary deployment you can use the rollback webhook. The rollback hook will be called during the analysis and confirmation states. If a rollback webhook returns a successful HTTP status code, Flagger will shift all traffic back to the primary instance and fail the canary.

Manual gating with Flagger’s tester:

 analysis:
 webhooks:
 - name: "gate"
 type: confirm-rollout
 url: http://flagger-loadtester.test/gate/halt

The /gate/halt returns HTTP 403 thus blocking the rollout.

If you have notifications enabled, Flagger will post a message to Slack or MS Teams if a canary rollout is waiting for approval.

The notifications can be disabled with:

 analysis:
 webhooks:
 - name: "gate"
 type: confirm-rollout
 url: http://flagger-loadtester.test/gate/halt
 muteAlert: true

Change the URL to /gate/approve to start the canary analysis:

 analysis:
 webhooks:
 - name: "gate"
 type: confirm-rollout
 url: http://flagger-loadtester.test/gate/approve

Manual gating can be driven with Flagger’s tester API. Set the confirmation URL to /gate/check:

 analysis:
 webhooks:
 - name: "ask for confirmation"
 type: confirm-rollout
 url: http://flagger-loadtester.test/gate/check

By default the gate is closed, you can start or resume the canary rollout with:

kubectl -n test exec -it flagger-loadtester-xxxx-xxxx sh

curl -d '{"name": "podinfo","namespace":"test"}' http://localhost:8080/gate/open

You can pause the rollout at any time with:

curl -d '{"name": "podinfo","namespace":"test"}' http://localhost:8080/gate/close

If a canary analysis is paused the status will change to waiting:

kubectl get canary/podinfo

NAME STATUS WEIGHT
podinfo Waiting 0

The confirm-promotion hook type can be used to manually approve the canary promotion. While the promotion is paused, Flagger will continue to run the metrics checks and load tests.

 analysis:
 webhooks:
 - name: "promotion gate"
 type: confirm-promotion
 url: http://flagger-loadtester.test/gate/halt

The rollback hook type can be used to manually rollback the canary promotion. As with gating, rollbacks can be driven with Flagger’s tester API by setting the rollback URL to /rollback/check

 analysis:
 webhooks:
 - name: "rollback"
 type: rollback
 url: http://flagger-loadtester.test/rollback/check

By default, rollback is closed, you can rollback a canary rollout with:

kubectl -n test exec -it flagger-loadtester-xxxx-xxxx sh

curl -d '{"name": "podinfo","namespace":"test"}' http://localhost:8080/rollback/open

You can close the rollback with:

curl -d '{"name": "podinfo","namespace":"test"}' http://localhost:8080/rollback/close

If you have notifications enabled, Flagger will post a message to Slack or MS Teams if a canary has been rolled back.

Troubleshooting

Manually check if helm test is running

To debug in depth any issues with helm tests, you can execute commands on the flagger-loadtester pod.

kubectl exec -it deploy/flagger-loadtester -- bash
helmv3 test <release> -n <namespace> --debug

Helm tests hang during canary deployment

If test execution hangs or displays insufficient permissions, check your RBAC settings.

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
 name: helm-smoke-tester
rules:
 - apiGroups: [""]
 resources: ["secrets"]
 verbs: ["get", "watch", "list", "update"]
 # choose the permission based on helm test type (Pod or Job) 
 - apiGroups: [""]
 resources: ["pods", "pods/log"]
 verbs: ["create", "list", "delete", "watch"]
 - apiGroups: ["batch"]
 resources: ["jobs", "jobs/log"]
 verbs: ["create", "list", "delete", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
 name: helm-smoke-tester
 # Don't forget to update accordingly
 namespace: namespace-of-the-tested-release
subjects:
 - kind: User
 name: system:serviceaccount:linkerd:default
 apiGroup: rbac.authorization.k8s.io
roleRef:
 kind: ClusterRole
 name: helm-smoke-tester
 apiGroup: rbac.authorization.k8s.io

Flagger: Alerting

Mon, 01 Jan 0001 00:00:00 +0000

Flagger can be configured to send alerts to various chat platforms. You can define a global alert provider at install time or configure alerts on a per canary basis.

Global configuration

Slack

Slack Configuration

Flagger requires a custom webhook integration from slack, instead of the new slack app system.

The webhook can be generated by following the legacy slack documentation

Flagger configuration

Once the webhook has been generated. Flagger can be configured to send Slack notifications:

helm upgrade -i flagger flagger/flagger \
--set slack.url=https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK \
--set slack.proxy=my-http-proxy.com \ # optional http/s proxy
--set slack.channel=some-channel-name \
--set slack.user=flagger \
--set clusterName=my-cluster

Once configured with a Slack incoming webhook, Flagger will post messages when a canary deployment has been initialised, when a new revision has been detected and if the canary analysis failed or succeeded.

A canary deployment will be rolled back if the progress deadline exceeded or if the analysis reached the maximum number of failed checks:

Microsoft Teams

Flagger can be configured to send notifications to Microsoft Teams:

helm upgrade -i flagger flagger/flagger \
--set msteams.url=https://outlook.office.com/webhook/YOUR/TEAMS/WEBHOOK \
--set msteams.proxy-url=my-http-proxy.com # optional http/s proxy

Similar to Slack, Flagger alerts on canary analysis events:

Canary configuration

Configuring alerting globally has several limitations as it’s not possible to specify different channels or configure the verbosity on a per canary basis. To make the alerting move flexible, the canary analysis can be extended with a list of alerts that reference an alert provider. For each alert, users can configure the severity level. The alerts section overrides the global setting.

Slack example:

apiVersion: flagger.app/v1beta1
kind: AlertProvider
metadata:
 name: on-call
 namespace: flagger
spec:
 type: slack
 channel: on-call-alerts
 username: flagger
 # webhook address (ignored if secretRef is specified)
 address: https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK
 # optional http/s proxy
 proxy: http://my-http-proxy.com
 # secret containing the webhook address (optional)
 secretRef:
 name: on-call-url
---
apiVersion: v1
kind: Secret
metadata:
 name: on-call-url
 namespace: flagger
data:
 address: <encoded-url>
 token: <encoded-slack-bot-token>

The alert provider type can be: slack, msteams, rocket or discord. When set to discord, Flagger will use Slack formatting and will append /slack to the Discord address.

For using a Slack bot token, you have to add the token to the Kubernetes secret referred in secretRef.

When not specified, channel defaults to general and username defaults to flagger.

When secretRef is specified, the Kubernetes secret must contain a data field named address, the address in the secret will take precedence over the address field in the provider spec.

The canary analysis can have a list of alerts, each alert referencing an alert provider:

 analysis:
 alerts:
 - name: "on-call Slack"
 severity: error
 providerRef:
 name: on-call
 namespace: flagger
 - name: "qa Discord"
 severity: warn
 providerRef:
 name: qa-discord
 - name: "dev MS Teams"
 severity: info
 providerRef:
 name: dev-msteams

Alert fields:

name (required)
severity levels: info, warn, error (default info)
providerRef.name alert provider name (required)
providerRef.namespace alert provider namespace (defaults to the canary namespace)

When the severity is set to warn, Flagger will alert when waiting on manual confirmation or if the analysis fails. When the severity is set to error, Flagger will alert only if the canary analysis fails.

To differentiate alerts based on the cluster name, you can configure Flagger with the -cluster-name=my-cluster command flag, or with Helm --set clusterName=my-cluster.

Prometheus Alert Manager

You can use Alertmanager to trigger alerts when a canary deployment failed:

 - alert: canary_rollback
 expr: flagger_canary_status > 1
 for: 1m
 labels:
 severity: warning
 annotations:
 summary: "Canary failed"
 description: "Workload {{ $labels.name }} namespace {{ $labels.namespace }}"

Flagger: Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Grafana

Flagger comes with a Grafana dashboard made for canary analysis. Install Grafana with Helm:

helm upgrade -i flagger-grafana flagger/grafana \
--set url=http://prometheus:9090

The dashboard shows the RED and USE metrics for the primary and canary workloads:

Logging

The canary errors and latency spikes have been recorded as Kubernetes events and logged by Flagger in json format:

kubectl -n istio-system logs deployment/flagger --tail=100 | jq .msg

Starting canary deployment for podinfo.test
Advance podinfo.test canary weight 5
Advance podinfo.test canary weight 10
Advance podinfo.test canary weight 15
Advance podinfo.test canary weight 20
Advance podinfo.test canary weight 25
Advance podinfo.test canary weight 30
Advance podinfo.test canary weight 35
Halt podinfo.test advancement success rate 98.69% < 99%
Advance podinfo.test canary weight 40
Halt podinfo.test advancement request duration 1.515s > 500ms
Advance podinfo.test canary weight 45
Advance podinfo.test canary weight 50
Copying podinfo.test template spec to podinfo-primary.test
Halt podinfo-primary.test advancement waiting for rollout to finish: 1 old replicas are pending termination
Scaling down podinfo.test
Promotion completed! podinfo.test

Event Webhook

Flagger can be configured to send event payloads to a specified webhook:

helm upgrade -i flagger flagger/flagger \
--set eventWebhook=https://example.com/flagger-canary-event-webhook

The environment variable EVENT_WEBHOOK_URL can be used for activating the event-webhook, too. This is handy for using a secret to store a sensible value that could contain api keys for example.

When configured, every action that Flagger takes during a canary deployment will be sent as JSON via an HTTP POST request. The JSON payload has the following schema:

{
 "name": "string (canary name)",
 "namespace": "string (canary namespace)",
 "phase": "string (canary phase)",
 "metadata": {
 "eventMessage": "string (canary event message)",
 "eventType": "string (canary event type)",
 "timestamp": "string (unix timestamp ms)"
 }
}

Example:

{
 "name": "podinfo",
 "namespace": "default",
 "phase": "Progressing",
 "metadata": {
 "eventMessage": "New revision detected! Scaling up podinfo.default",
 "eventType": "Normal",
 "timestamp": "1578607635167"
 }
}

The event webhook can be overwritten at canary level with:

 analysis:
 webhooks:
 - name: "send to Slack"
 type: event
 url: http://event-recevier.notifications/slack

Metrics

Flagger exposes Prometheus metrics that can be used to determine the canary analysis status and the destination weight values:

# Flagger version and mesh provider gauge
flagger_info{version="0.10.0", mesh_provider="istio"} 1

# Canaries total gauge
flagger_canary_total{namespace="test"} 1

# Canary promotion last known status gauge
# 0 - running, 1 - successful, 2 - failed
flagger_canary_status{name="podinfo" namespace="test"} 1

# Canary traffic weight gauge
flagger_canary_weight{workload="podinfo-primary" namespace="test"} 95
flagger_canary_weight{workload="podinfo" namespace="test"} 5

# Seconds spent performing canary analysis histogram
flagger_canary_duration_seconds_bucket{name="podinfo",namespace="test",le="10"} 6
flagger_canary_duration_seconds_bucket{name="podinfo",namespace="test",le="+Inf"} 6
flagger_canary_duration_seconds_sum{name="podinfo",namespace="test"} 17.3561329
flagger_canary_duration_seconds_count{name="podinfo",namespace="test"} 6

# Last canary metric analysis result per different metrics
flagger_canary_metric_analysis{metric="podinfo-http-successful-rate",name="podinfo",namespace="test"} 1
flagger_canary_metric_analysis{metric="podinfo-custom-metric",name="podinfo",namespace="test"} 0.918223108974359