Skip to main content
Version: 0.36.0

Multi Tenancy Enterprise

Multi Tenancy

Multi tenancy provides users with the ability to define boundaries to multiple engineering teams working on a single cluster. Through a simple interface it adds permissions to the necessary Kubernetes resources to make it easy for customers to manage their multiple tenants.

WGE multi tenancy expands on the multi tenancy feature provided by flux. In addition to creating the necessary Kubernetes tenancy resources that flux adds, multi tenancy in WGE also adds the following:

  • Defining tenancy using a single yaml file that serves as a source of truth for the organization
  • Makes use of WGE policy features to enforce non Kubernetes native permissions

Prerequisites

How it works

gitops command line tool is responsible for creating the multi tenancy resources. The tool is distributed as part of WGE offering. It reads the definitions of a yaml file and can either apply the necessary changes directly to the cluster or output it to stdout so it can be saved into a file and pushed to a repo to be reconciled by flux.

To make use of the policy features, policy agent needs to be installed in the necessary cluster(s).

Tenancy file

Below is an example of a tenancy file:

Expand to view
tenancy.yaml
---
tenants:
- name: first-tenant
namespaces:
- first-ns
- name: second-tenant
namespaces:
- second-test-ns
- second-dev-ns
allowedRepositories:
- kind: GitRepository
url: https://github.com/testorg/testrepo
- kind: GitRepository
url: https://github.com/testorg/testinfo
- kind: Bucket
url: minio.example.com
- kind: HelmRepository
url: https://testorg.github.io/testrepo
allowedClusters:
- kubeConfig: cluster-1-kubeconfig
- kubeConfig: cluster-2-kubeconfig
teamRBAC:
groupNames:
- foo-group
- bar-group
rules:
- apiGroups:
- ''
resources:
- 'namespaces'
- 'pods'
verbs:
- 'list'
- 'get'
deploymentRBAC:
bindRoles:
- name: foo-role
kind: Role
rules:
- apiGroups:
- ''
resources:
- 'namespaces'
- 'pods'
verbs:
- 'list'
- 'get'
serviceAccount:
name: "reconcilerServiceAccount"

The file above defines two tenants: first-tenant and second-tenant as follows:

  • namespaces: describes which namespaces should be part of the tenant. Meaning that users who are part of the tenant would have access on those namespaces.
  • allowedRepositories: limits the flux repositories sources that can be used in the tenant's namespaces. This is done through policies and thus requires policy-agent to be deployed on the cluster which will stop these sources from being deployed if they aren't allowed as part of the tenant. IT consists of:
    • kind: the flux source kind. Can be: GitRepository, Bucket and HelmRepository.
    • url: the URL for that source.
  • allowedClusters: limits which secrets containing cluster configuraton can be used. It stops WGE GitopsCluster and flux Kustomization from being deployed if they point to a secret not in the list, essentially giving control on which cluster can be added to a multi-cluster setup. Requires policy-agent.
    • kubeConfig: name of the secret that can be used for this tenant.
  • teamRBAC: Generate Roles and Rolebindings for a list of groupNames. This allows you to easily give an OIDC group access to a tenant's resources. When the Weave Gitops Enterprise UI is configured with your OIDC provider, tenants can log in and view the status of the resources they have been granted access to.
  • deploymentRBAC: generate Roles and Rolebindings for a service account. Can additionally bind to an existing Roles/ClusterRoles. Would use the global service account if specified in the tenants file, otherwise it will use the created service account which takes the tenant name. If not specified a Rolebinding would be created that binds to cluster-admin ClusterRole.

Global options:

  • serviceAccount: Override the name of the generated ServiceAccount for all tenants. This allows you to easily use the flux controllers' --default-service-account feature. Tenants do not need to make sure they correctly specify the serviceAccount when using Kustomization or HelmRelease resources. The kustomization-controller and helm-controller will instead look for the default-service-account in the namespace being reconciled to and use that. Just configure serviceAccount.name and --default-service-account to the same value.

Gitops create tenants command

The command creates the necessary resources to apply multi tenancy on the user's cluster. To use the command to apply the resources directly the user needs to have the necessary configuration to connect to the desired cluster. The command considers the tenancy file as a source of truth and will change the cluster state to match what is currently described in the file.

For more control on a specific tenant a tenancy file should be used, the command allows the creation of the base resources that defines a tenancy through the arguments:

gitops create tenants --name test-tenant --namespace test-ns1 --namespace test-ns2
Expand to view command output
namespace/test-ns1 created
test-ns1/serviceaccount/test-tenant created
test-ns1/rolebinding.rbac.authorization.k8s.io/test-tenant-service-account-cluster-admin created
namespace/test-ns2 created
test-ns2/serviceaccount/test-tenant created
test-ns2/rolebinding.rbac.authorization.k8s.io/test-tenant-service-account-cluster-admin created
policy.pac.weave.works/weave.policies.tenancy.test-tenant-allowed-application-deploy created

The above will create the namespaces and permissions through a ServiceAccount with the same name as the tenant, test-tenant in the case of the above example, in each required namespace. The same can be done through a file as follows:

tenants:
- name: test-tenant
namespaces:
- test-ns1
- test-ns2
gitops create tenants --from-file tenants.yaml
Expand to view command output
namespace/test-ns1 created
test-ns1/serviceaccount/test-tenant created
test-ns1/rolebinding.rbac.authorization.k8s.io/test-tenant-service-account-cluster-admin created
namespace/test-ns2 created
test-ns2/serviceaccount/test-tenant created
test-ns2/rolebinding.rbac.authorization.k8s.io/test-tenant-service-account-cluster-admin created
policy.pac.weave.works/weave.policies.tenancy.test-tenant-allowed-application-deploy created

To check the resources that would be deployed first use the export flag:

gitops create tenants --from-file tenants.yaml --export
Expand to view command output
apiVersion: v1
kind: Namespace
metadata:
creationTimestamp: null
labels:
toolkit.fluxcd.io/tenant: test-tenant
name: test-ns1
spec: {}
status: {}
---
apiVersion: v1
kind: ServiceAccount
metadata:
creationTimestamp: null
labels:
toolkit.fluxcd.io/tenant: test-tenant
name: test-tenant
namespace: test-ns1
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
creationTimestamp: null
labels:
toolkit.fluxcd.io/tenant: test-tenant
name: test-tenant-service-account-cluster-admin
namespace: test-ns1
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: test-tenant
namespace: test-ns1
---
apiVersion: v1
kind: Namespace
metadata:
creationTimestamp: null
labels:
toolkit.fluxcd.io/tenant: test-tenant
name: test-ns2
spec: {}
status: {}
---
apiVersion: v1
kind: ServiceAccount
metadata:
creationTimestamp: null
labels:
toolkit.fluxcd.io/tenant: test-tenant
name: test-tenant
namespace: test-ns2
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
creationTimestamp: null
labels:
toolkit.fluxcd.io/tenant: test-tenant
name: test-tenant-service-account-cluster-admin
namespace: test-ns2
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: test-tenant
namespace: test-ns2
---
apiVersion: pac.weave.works/v2beta2
kind: Policy
metadata:
creationTimestamp: null
labels:
toolkit.fluxcd.io/tenant: test-tenant
name: weave.policies.tenancy.test-tenant-allowed-application-deploy
spec:
category: weave.categories.tenancy
code: |
package weave.tenancy.allowed_application_deploy

controller_input := input.review.object
violation[result] {
namespaces := input.parameters.namespaces
targetNamespace := controller_input.spec.targetNamespace
not contains_array(targetNamespace, namespaces)
result = {
"issue detected": true,
"msg": sprintf("using target namespace %v is not allowed", [targetNamespace]),
}
}
violation[result] {
serviceAccountName := controller_input.spec.serviceAccountName
serviceAccountName != input.parameters.service_account_name
result = {
"issue detected": true,
"msg": sprintf("using service account name %v is not allowed", [serviceAccountName]),
}
}
contains_array(item, items) {
items[_] = item
}
description: Determines which helm release and kustomization can be used in a tenant
how_to_solve: ""
id: weave.policies.tenancy.test-tenant-allowed-application-deploy
name: test-tenant allowed application deploy
parameters:
- name: namespaces
required: false
type: array
value:
- test-ns1
- test-ns2
- name: service_account_name
required: false
type: string
value: test-tenant
provider: kubernetes
severity: high
standards: []
tags:
- tenancy
targets:
kinds:
- HelmRelease
- Kustomization
labels: []
namespaces:
- test-ns1
- test-ns2
status: {}
---

Applying the resources through the command line is not usually recommended. For WGE the recommended way is to commit the result of the create tenants command to source control and let flux handle deployment. To achieve that you can save the result of the export to a file:

gitops create tenants --from-file tenants.yaml --export > clusters/management/tenants.yaml