Skip to main content
Version: 0.35.0

Progressive Delivery Using Flagger Enterprise

Built upon the core tenets of continuous integration and continuous delivery (CI/CD), progressive delivery involves gradually rolling out features to small groups of select users to balance performance with speed. Developers and DevOps teams use fine-grained controls to minimize the risks of pushing new features to the production environment. If the newly released feature proves to be stable and performant, it can then be released to all users.

Flagger is a progressive delivery operator for Kubernetes and part of the Flux family of open source projects. It reduces the risk of introducing new software versions and automates production releases to improve your time to delivery. Flagger implements deployment strategies—canary releases, A/B testing, Blue/Green mirroring—using a service mesh (App Mesh, Istio, Linkerd, Kuma, Open Service Mesh) or an ingress controller (Contour, Gloo, NGINX, Skipper, Traefik, APISIX) for traffic routing. For release analysis, Flagger can query Prometheus, InfluxDB, Datadog, New Relic, CloudWatch, Stackdriver, or Graphite. For alerting it uses Slack, MS Teams, Discord, and Rocket. Using Flux allows us to manage our cluster applications in a declarative way through changes in a Git repository.

Weave GitOps Enterprise integrates with Flagger in order to provide a view on progressive delivery deployments. This includes the ability to view all the resources that Flagger manages during its operation. The default ClusterRole gitops-canaries-reader includes the minimum permissions necessary for a user to be able to view canary object details, metric template object details and canary related events.

The WGE UI's Applications > Delivery view provides an "at a glance" view so that you can understand the status of your progressive delivery rollouts across a fleet of connected clusters. This removes the cognitive overhead of having to know which objects to query and where they are located. You can also drill down into each rollout to understand its status and configuration, and view near-to-realtime data on any summary or details page.

Applications Delivery view

How to use WGE's progressive delivery offering:

  • if you don’t have Flagger installed on any clusters, you'll receive an onboarding message about installing it
  • click on the delivery tab on the menu bar to retrieve a table view of canaries with key summary information regarding their location and state
  • click on a canary to see more detailed information about status, gates, and other elements
  • click on the events tab on the detail page to see the most recent Kubernetes events for that canary and learn more about deployment history
  • click on the yaml tab on the detail page to see the raw yaml of the canary
  • view objects from any cluster/namespace that you have the appropriate permissions for, and nothing else

Supported deployment strategies include:

canary release icon Canary Release: the system gradually shifts traffic to a new version of an application and assesses performance—either promoting the release or abandoning it, based on performance.

a b testing icon A/B Testing: uses HTTP headers or cookies to ensure users remain on the same version of an application during a canary analysis.

blue green testing icon Blue/Green: Traffic is switched from the current application to a new version based on the success of testing.

blue green mirroring icon Blue/Green with Traffic Mirroring: sends copies of incoming requests to the new version of an application. The user receives the response from the current application and the other is discarded. The new version is promoted only if metrics are healthy.

This guide uses Flux manifests to install Flagger and Linkerd, a CNCF project and service mesh for Kubernetes and beyond. We will walk you through a full end-to-end scenario where you will:

Prerequisites

  • This guide assumes you already have a Kubernetes cluster running and have bootstrapped Flux. To apply the manifests listed here, you will need to commit them to a repository being reconciled with Flux.
  • Flagger requires the autoscaling/v2 or autoscaling/v2beta2 API to be installed on your cluster. You can use kubectl api-resources to check which API versions are supported.
  • The step CLI installed to generate certificates that support mTLS connections.

Installing Linkerd Using Flux

To install Linkerd we'll use a Kustomization file. It will allow us to specify the order and default namespace for the installed resources, and to generate Secrets from certificate files via the use of a secretGenerator.

To support mTLS connections between meshed pods, Linkerd requires a trust anchor certificate and an issuer certificate with its corresponding key. These certificates are automatically created via the linkerd install command. However, when using a Helm chart to install Linkerd, you must provide these certificates deliberately. The step CLI, listed above, allows us to generate these certificates.

To generate the trust anchor certificate, run:

step certificate create root.linkerd.cluster.local ca.crt ca.key \
--profile root-ca --no-password --insecure

To generate the issuer certificate, run:

step certificate create identity.linkerd.cluster.local issuer.crt issuer.key \
--profile intermediate-ca --not-after 8760h --no-password --insecure \
--ca ca.crt --ca-key ca.key

Add the ca.crt, issuer.crt, and issuer.key files to the cluster repository under a linkerd directory.

Let's add the three manifests for Linkerd components under the ./linkerd directory:

  • A Namespace resource to control where the components are installed
  • A HelmRepository resource to make the Linkerd Helm repo available on the cluster
  • A HelmRelease resource to install the latest version of Linkerd from the HelmRepository
Expand to see and copy-paste the three Linkerd manifests to add
linkerd/namespace.yaml
---
apiVersion: v1
kind: Namespace
metadata:
name: linkerd
labels:
config.linkerd.io/admission-webhooks: disabled
linkerd/source.yaml
---
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: HelmRepository
metadata:
name: linkerd
spec:
interval: 1h
url: https://helm.linkerd.io/stable
tip

The value for the spec.values.identity.issuer.crtExpiry field below depends on the parameter value used during the creation of the issuer certificate. In this example, it should be set to one year from the certificate creation.

linkerd/releases.yaml
---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
name: linkerd
spec:
interval: 10m
chart:
spec:
chart: linkerd2
reconcileStrategy: ChartVersion
sourceRef:
kind: HelmRepository
name: linkerd
install:
crds: Create
upgrade:
crds: CreateReplace
valuesFrom:
- kind: Secret
name: linkerd-certs
valuesKey: ca.crt
targetPath: identityTrustAnchorsPEM
- kind: Secret
name: linkerd-certs
valuesKey: issuer.crt
targetPath: identity.issuer.tls.crtPEM
- kind: Secret
name: linkerd-certs
valuesKey: issuer.key
targetPath: identity.issuer.tls.keyPEM
values:
installNamespace: false
identity:
issuer:
crtExpiry: "2023-07-18T20:00:00Z" # Change this to match generated certificate expiry date
---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
name: linkerd-viz
spec:
interval: 10m
dependsOn:
- name: linkerd
chart:
spec:
chart: linkerd-viz
reconcileStrategy: ChartVersion
sourceRef:
kind: HelmRepository
name: linkerd

Next, add the following manifests. The first file instructs Kustomize to patch any Secrets that are referenced in HelmRelease manifests. The second file is a Kustomization that references all the other linkerd resource files.

Expand to see the Linkerd Kustomization manifests
linkerd/kustomizeconfig.yaml
nameReference:
- kind: Secret
version: v1
fieldSpecs:
- path: spec/valuesFrom/name
kind: HelmRelease
linkerd/kustomization.yaml
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: linkerd
configurations:
- kustomizeconfig.yaml
resources:
- namespace.yaml
- source.yaml
- releases.yaml
secretGenerator:
- name: linkerd-certs
files:
- ca.crt
- issuer.crt
- issuer.key
Note

The secretGenerator generates Secrets from the files you've just created.

At this point the linkerd directory in your cluster repository should look like this:

> tree linkerd
linkerd
├── ca.crt
├── issuer.crt
├── issuer.key
├── kustomization.yaml
├── kustomizeconfig.yaml
├── namespace.yaml
├── releases.yaml
└── source.yaml

Once Flux reconciles this directory to the cluster, Linkerd should be installed.

Before proceeding to the next step, check that all the Linkerd pods have started successfully:

> kubectl get pods -n linkerd 
NAME READY STATUS RESTARTS AGE
linkerd-destination-66d5668b-4mw49 4/4 Running 0 10m
linkerd-identity-6b4658c74b-6nc97 2/2 Running 0 10m
linkerd-proxy-injector-6b76789cb4-8vqj4 2/2 Running 0 10m

> kubectl get pods -n linkerd-viz
NAME READY STATUS RESTARTS AGE
grafana-db56d7cb4-xlnn4 2/2 Running 0 10m
metrics-api-595c7b564-724ps 2/2 Running 0 10m
prometheus-5d4dffff55-8fscd 2/2 Running 0 10m
tap-6dcb89d487-5ns8n 2/2 Running 0 10m
tap-injector-54895654bb-9xn7k 2/2 Running 0 10m
web-6b6f65dbc7-wltdg 2/2 Running 0 10m
Note

Any new directories that you add to the cluster repository while following this guide must be included in a path that Flux reconciles.

Installing Flagger Using Flux

To install Flagger, you'll use a Kustomization file that will define the installation order and provide a default namespace for the installed resources.

Create a new flagger directory. Make sure to locate it under a repository path that Flux reconciles.

Now add under this directory the three resource manifests for Flagger:

  • A Namespace resource to control where the components are installed
  • A HelmRepository resource to make the Flagger Helm repo available on the cluster
  • A HelmRelease resource to install the latest version of Flagger and the load tester app (which generates synthetic traffic during the analysis phase), from that HelmRepository
Expand to see the three Flagger resource manifests
flagger/namespace.yaml
---
apiVersion: v1
kind: Namespace
metadata:
name: flagger
flagger/source.yaml
---
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: HelmRepository
metadata:
name: flagger
spec:
interval: 1h
url: https://flagger.app
flagger/releases.yaml
---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
name: flagger
spec:
releaseName: flagger
install:
crds: Create
upgrade:
crds: CreateReplace
interval: 10m
chart:
spec:
chart: flagger
reconcileStrategy: ChartVersion
sourceRef:
kind: HelmRepository
name: flagger
values:
metricsServer: http://prometheus.linkerd-viz:9090
meshProvider: linkerd
---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
name: loadtester
spec:
interval: 10m
chart:
spec:
chart: loadtester
reconcileStrategy: ChartVersion
sourceRef:
kind: HelmRepository
name: flagger

Now add the following Kustomization file. It references all of the previous files that you've added:

Expand to see the Flagger Kustomization manifest
flagger/kustomization.yaml
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: flagger
resources:
- namespace.yaml
- source.yaml
- releases.yaml

The flagger directory in the cluster repository should look like this:

> tree flagger
flagger
├── kustomization.yaml
├── namespace.yaml
├── releases.yaml
└── source.yaml

Once Flux reconciles this directory to the cluster, Flagger and the load tester app should be installed.

Before proceeding to the next step, check that all of your Flagger pods have started successfully:

> kubectl get pods -n flagger
NAME READY STATUS RESTARTS AGE
flagger-7d456d4fc7-knf2g 1/1 Running 0 4m
loadtester-855b4d77f6-scl6r 1/1 Running 0 4m

Custom Resources Generated by Flagger

When Flagger is configured to integrate with a service mesh such as Linkerd or Istio for the rollout, this ClusterRole needs to be extended so that it can read the additional service mesh resources that Flagger generates. To display service mesh- or ingress-related resources, we require spec.provider to be set in each canary resource.

The following table provides a list of all the custom resources that Flagger generates grouped by provider:

ProviderAPI GroupResource
AppMeshappmesh.k8s.awsvirtualnode
appmesh.k8s.awsvirtualrouter
appmesh.k8s.awsvirtualservice
Linkerdsplit.smi-spec.iotrafficsplit
Istionetworking.istio.iodestinationrule
networking.istio.iovirtualservice
Contourprojectcontour.iohttpproxy
Gloogateway.solo.ioroutetable
gloo.solo.ioupstream
Nginxnetworking.k8s.ioingress
Skippernetworking.k8s.ioingress
Traefiktraefik.containo.ustraefikservice
Open Service Meshsplit.smi-spec.iotrafficsplit
Kumakuma.iotrafficroute
GatewayAPIgateway.networking.k8s.iohttproute

For example, the following manifest shows how gitops-canaries-reader has been extended to allow the user for viewing TrafficSplit resources when Linkerd is used:

Expand to see example canary reader RBAC
gitops-canaries-reader.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: gitops-canaries-reader
rules:
- apiGroups:
- flagger.app
resources:
- canaries
- metrictemplates
verbs:
- get
- list
- apiGroups:
- ""
resources:
- events
verbs:
- get
- watch
- list
# Additional permissions for Linkerd resources are added below
- apiGroups:
- split.smi-spec.io
resources:
- trafficsplits
verbs:
- get
- list

Setting up Remote Cluster Permissions

In order to view canaries in a remote cluster from the management cluster, you need to consider the following:

  • The service account used to access the remote cluster needs to be able to list namespaces and custom resource definitions in the given cluster. It additionally needs to be able to impersonate users and groups.
  • The user or group that logs in to the management cluster, needs appropriate permissions to certain resources of the remote cluster.

For example, applying the following manifest on remote clusters, ensures that the wego-admin user will be able to view canary information from within the Weave GitOps Enterprise UI on the management cluster:

Expand to see example of remote cluster canary reader
remote-cluster-service-user-rbac.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: user-groups-impersonator
rules:
- apiGroups: [""]
resources: ["users", "groups"]
verbs: ["impersonate"]
- apiGroups: [""]
resources: ["namespaces"]
verbs: ["get", "list"]
- apiGroups: ["apiextensions.k8s.io"]
resources: ["customresourcedefinitions"]
verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: impersonate-user-groups
subjects:
- kind: ServiceAccount
name: remote-cluster-01 # Service account created in remote cluster
namespace: default
roleRef:
kind: ClusterRole
name: user-groups-impersonator
apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: canary-reader
rules:
- apiGroups: [""]
resources: [ "events", "services" ]
verbs: [ "get", "list", "watch" ]
- apiGroups: [ "apps" ]
resources: [ "*" ]
verbs: [ "get", "list" ]
- apiGroups: [ "autoscaling" ]
resources: [ "*" ]
verbs: [ "get", "list" ]
- apiGroups: [ "flagger.app" ]
resources: [ "canaries", "metrictemplates"]
verbs: [ "get", "list", "watch" ]
- apiGroups: [ "helm.toolkit.fluxcd.io" ]
resources: [ "helmreleases" ]
verbs: [ "get", "list" ]
- apiGroups: [ "kustomize.toolkit.fluxcd.io" ]
resources: [ "kustomizations" ]
verbs: [ "get", "list" ]
- apiGroups: [ "source.toolkit.fluxcd.io" ]
resources: [ "buckets", "helmcharts", "gitrepositories", "helmrepositories", "ocirepositories" ]
verbs: [ "get", "list" ]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: read-canaries
subjects:
- kind: User
name: wego-admin # User logged in management cluster, impersonated via service account
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: ClusterRole
name: canary-reader
apiGroup: rbac.authorization.k8s.io

You may need to add more users/groups to the read-canaries ClusterRoleBinding to ensure additional users can view canary information from within the Weave GitOps Enterprise UI.

Deploy a Canary Release

To demonstrate the progressive rollout of an application, we'll use a tiny sample web app called podinfo and configure a canary release strategy.

In our example, Flagger will scale up a new version of podinfo (the canary) alongside the existing version (the primary). It will gradually increase traffic to the new version in increments of 5%, up to a maximum of 50%. Flagger will continuously monitor the new version for an acceptable request response rate and average request duration. Based on this analysis, Flagger will either update the primary to the new version or abandon the promotion, then scale the canary back down to zero.

Create a new test directory and add these three canary resource manifests under it:

  • A Namespace resource to control where the components are installed
  • A Deployment and HorizontalPodAutoscaler for the podinfo application
  • A Canary resource which references the Deployment and HorizontalPodAutoscaler resources

We don't need to define a service resource. This is specified within the canary definition and created by Flagger.

Expand to see the three canary resource manifests
test/namespace.yaml
---
apiVersion: v1
kind: Namespace
metadata:
name: test
annotations:
linkerd.io/inject: enabled
test/deployment.yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: podinfo
labels:
app: podinfo
spec:
minReadySeconds: 5
revisionHistoryLimit: 5
progressDeadlineSeconds: 60
strategy:
rollingUpdate:
maxUnavailable: 1
type: RollingUpdate
selector:
matchLabels:
app: podinfo
template:
metadata:
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "9797"
labels:
app: podinfo
spec:
containers:
- name: podinfod
image: ghcr.io/stefanprodan/podinfo:6.1.8
imagePullPolicy: IfNotPresent
ports:
- name: http
containerPort: 9898
protocol: TCP
- name: http-metrics
containerPort: 9797
protocol: TCP
- name: grpc
containerPort: 9999
protocol: TCP
command:
- ./podinfo
- --port=9898
- --port-metrics=9797
- --grpc-port=9999
- --grpc-service-name=podinfo
- --level=info
- --random-delay=false
- --random-error=false
env:
- name: PODINFO_UI_COLOR
value: "#34577c"
livenessProbe:
exec:
command:
- podcli
- check
- http
- localhost:9898/healthz
initialDelaySeconds: 5
timeoutSeconds: 5
readinessProbe:
exec:
command:
- podcli
- check
- http
- localhost:9898/readyz
initialDelaySeconds: 5
timeoutSeconds: 5
resources:
limits:
cpu: 2000m
memory: 512Mi
requests:
cpu: 100m
memory: 64Mi

---
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
name: podinfo
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: podinfo
minReplicas: 2
maxReplicas: 4
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
# scale up if usage is above
# 99% of the requested CPU (100m)
averageUtilization: 99
test/canary.yaml
---
apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
name: podinfo
spec:
# deployment reference
targetRef:
apiVersion: apps/v1
kind: Deployment
name: podinfo
# HPA reference (optional)
autoscalerRef:
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
name: podinfo
# the maximum time in seconds for the canary deployment
# to make progress before it is rollback (default 600s)
progressDeadlineSeconds: 60
service:
# ClusterIP port number
port: 9898
# container port number or name (optional)
targetPort: 9898
analysis:
# schedule interval (default 60s)
interval: 30s
# max number of failed metric checks before rollback
threshold: 5
# max traffic percentage routed to canary
# percentage (0-100)
maxWeight: 50
# canary increment step
# percentage (0-100)
stepWeight: 5
# Linkerd Prometheus checks
metrics:
- name: request-success-rate
# minimum req success rate (non 5xx responses)
# percentage (0-100)
thresholdRange:
min: 99
interval: 1m
- name: request-duration
# maximum req duration P99
# milliseconds
thresholdRange:
max: 500
interval: 30s
# testing (optional)
webhooks:
- name: acceptance-test
type: pre-rollout
url: http://loadtester.flagger/
timeout: 30s
metadata:
type: bash
cmd: "curl -sd 'test' http://podinfo-canary.test:9898/token | grep token"
- name: load-test
type: rollout
url: http://loadtester.flagger/
metadata:
cmd: "hey -z 2m -q 10 -c 2 http://podinfo-canary.test:9898/"

Add a Kustomization file to apply all resources to the test namespace:

Expand to see the Canary Kustomization manifest
test/kustomization.yaml
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: test
resources:
- namespace.yaml
- deployment.yaml
- canary.yaml

At this point, the test directory in the cluster repository should look like this:

> tree test
test
├── canary.yaml
├── deployment.yaml
├── kustomization.yaml
└── namespace.yaml

After a short time, the status of the canary object should be set to Initialized:

Canary rollout initialized

> kubectl get canary podinfo -n test
NAME STATUS WEIGHT LASTTRANSITIONTIME
podinfo Initialized 0 2022-07-22T12:37:58Z

Trigger a new rollout by bumping the version of podinfo:

> kubectl set image deployment/podinfo podinfod=ghcr.io/stefanprodan/podinfo:6.0.1 -n test

During the progressive rollout, the canary object reports on its current status:

Canary rollout progressing

> kubectl get canary podinfo -n test
NAME STATUS WEIGHT LASTTRANSITIONTIME
podinfo Progressing 5 2022-07-22T12:41:57Z

After a short time the rollout is completed and the status of the canary object is set to Succeeded:

Canary rollout succeeded

> kubectl get canary podinfo -n test
NAME STATUS WEIGHT LASTTRANSITIONTIME
podinfo Succeeded 0 2022-07-22T12:47:58Z

Summary

Congratulations, you have now completed a progressive delivery rollout with Flagger and Linkerd! 🎉

Next steps: