When upgrading service versions, you need to use release methods such as rolling upgrades, batch suspension release, blue-green release, and gray release. This section introduces how to implement gray release for application services in a CCSE cluster using the Nginx Ingress Controller.
Background
Gray and blue-green releases involve creating a production environment for the new version that is identical to the old version. Without affecting the old version, a portion of traffic is switched to the new version based on specific rules. After a test run of the new version for a certain period without problems, all user traffic is migrated from the old version to the new version.
A/B testing is a type of gray release. Some users continue to use the old version of the service and switch a portion of user traffic to the new version. If the new version runs stably, all users are gradually migrated to the new version.
Here is how to use the gray release feature in the CCSE console.
canary-* annotation method: Use canary-* Annotations to configure blue-green release and gray release. This is the official method for implementing gray release of the community.
Application Scenarios
Traffic Split Scenario Based on Client Requests
Consider a situation where Service A is providing Layer 7 services. You have developed new features and want to deploy them in a new version, Service A'. Instead of directly replacing Service A, your strategy is to forward client requests with "foo=bar" in the request header or cookie to Service A'. Once Service A' operates stably for a period, all traffic switches from Service A to Service A', allowing the smooth decommissioning of Service A.
Scenario 2: Traffic Split Scenario Based on Service Weight
Consider a situation where Service B is providing Layer 7 services. You have resolved some issues and want to deploy them in a new version, Service B'. Rather than switching all client traffic to the new version Service B', your strategy is to switch 20% of the traffic to Service B'. Once Service B' operates stably for a period, all traffic switches from Service B to Service B', allowing the smooth decommissioning of Service B.
To meet different application release needs as above, the eSurfing Cloud CCSE Ingress Controller supports various traffic splitting methods:
Traffic split based on Request Header: This method is suitable for gray release and A/B testing scenarios.
Traffic split based on Cookies: This method is suitable for gray release and A/B testing scenarios.
Traffic split based on Query Param: This method is suitable for gray release and A/B testing scenarios.
Traffic split based on service weight: This method is suitable for blue-green deployment scenarios.
canary-* Annotation
Annotation Instructions
The Nginx Ingress Controller supports the gray release mechanism of application services through the following canary-* Annotations.
Annotation | Description | Applicable CCSE Nginx Ingress Controller Version |
nginx.ingress.kubernetes.io/canary | · This Annotation must be set to · Value: o o | ≥v1.3.1 |
nginx.ingress.kubernetes.io/canary-by-header | · Represent gray release based on the name of the request header. · Special values for the name of the request header: o o · If the value of the request header name is not specified, traffic will be forwarded as long as the header exists. | ≥v1.3.1 |
nginx.ingress.kubernetes.io/canary-by-header-value | · Represent gray release based on the value of the request header. · Need to be used in conjunction with the | ≥v1.3.1 |
nginx.ingress.kubernetes.io/canary-by-cookie | · Represent gray release based on a Cookie. · Special values for the Cookie name: o o · As long as the Cookie name exists, traffic will be forwarded. | ≥v1.3.1 |
nginx.ingress.kubernetes.io/canary-weight | · Represent gray release based on weight. · Value range: 0 - total weight. · If the total weight is not set, the default total weight is 100. | ≥v1.3.1 |
Procedures
Step 1: Deploy Service
Deploy the Nginx service and provide Layer 7 domain name access for external systems through the Nginx Ingress Controller.
Create Deployment and Service.
Create nginx.yaml.
apiVersion: apps/v1 kind: Deployment metadata: name: old-nginx spec: replicas: 2 selector: matchLabels: run: old-nginx template: metadata: labels: run: old-nginx spec: containers: - image: registry- nm6b -crs.ctyun.com/ccse-sample/old-nginx imagePullPolicy: Always name: old-nginx ports: - containerPort: 80 protocol: TCP restartPolicy: Always --- apiVersion: v1 kind: Service metadata: name: old-nginx spec: ports: - port: 80 protocol: TCP targetPort: 80 selector: run: old-nginx sessionAffinity: None type: NodePort |
Execute the following command to create Deployment and Service.
kubectl apply -f nginx.yaml |
Deploy Ingress
Create ingress.yaml.
apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: gray-release spec: rules: - host: www.example.com http: paths: # Service of the old version. - path: / backend: service: name: old-nginx port: number: 80 pathType: ImplementationSpecific |
Execute the following command to deploy Ingress.
kubectl apply -f ingress.yaml |
Test the access situation.
Execute the following command to obtain the external IP.
kubectl get ingress |
curl -H "Host: www.example.com" http://<EXTERNAL_IP> |
Expected Output
old |
Step 2: Implement gray release for the service of the new version.
Deploy the Nginx service of a new version and set routing rules.
Deploy Deployment and Service of the new version.
Create nginx1.yaml.
apiVersion: apps/v1 kind: Deployment metadata: name: new-nginx spec: replicas: 1 selector: matchLabels: run: new-nginx template: metadata: labels: run: new-nginx spec: containers: - image: registry-nm6b-crs.ctyun.com/ccse-sample/new-nginx imagePullPolicy: Always name: new-nginx ports: - containerPort: 80 protocol: TCP restartPolicy: Always --- apiVersion: v1 kind: Service metadata: name: new-nginx spec: ports: - port: 80 protocol: TCP targetPort: 80 selector: run: new-nginx sessionAffinity: None type: NodePort |
Execute the following command to create Deployment and Service.
kubectl apply -f nginx.yaml |
Set routing rules for accessing the service of the new version.
CCSE supports the following three types of routing rules. Choose the appropriate routing rule based on your specific needs.
Set up a configuration where only clients that meet specific rules can access the service of the new version. In the example below, only client requests with "foo=bar" in the request header can route to the service of the new version.
Based on the conditions specified above, create a new Ingress resource named "gray-release-canary".
apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: gray-release-canary annotations: # Enable Canary. nginx.ingress.kubernetes.io/canary: "true" # The request header is "foo". nginx.ingress.kubernetes.io/canary-by-header: "foo" # Only when the value of the request header foo is "bar", the request will be routed to the service new-nginx of the new version. nginx.ingress.kubernetes.io/canary-by-header-value: "bar" spec: rules: - host: www.example.com http: paths: # Service of the new version. - path: / backend: service: name: new-nginx port: number: 80 pathType: ImplementationSpecific |
Check the routing access status.
Execute the following command to access the service.
curl -H "Host: www.example.com" http://<EXTERNAL_IP> |
Expected output:
old |
Execute the following command. This allows client requests with "foo=bar" in the request header to access the service.
curl -H "Host: www.example.com" -H "foo: bar" http://<EXTERNAL_IP> |
Expected output:
new |
On the basis of satisfying specific rules, configure a certain proportion of requests to route to the service of the new version. In the following example, client requests with "foo=bar" in the request header can access the service, and only 50% of the traffic can be routed to the service of the new version.
Modify the Ingress created in step 2 as follows.
apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: gray-release-canary annotations: # Enable Canary. nginx.ingress.kubernetes.io/canary: "true" # The request header is "foo". nginx.ingress.kubernetes.io/canary-by-header: "foo" # Only when the value of the request header foo is "bar", the request will be routed to the service new-nginx of the new version. nginx.ingress.kubernetes.io/canary-by-header-value: "bar" # On the basis of satisfying the above matching rules, only 50% of the traffic will be routed to the service new-nginx of the new version. nginx.ingress.kubernetes.io/canary-weight: "50" spec: rules: - host: www.example.com http: paths: # Service of the new version. - path: / backend: service: name: new-nginx port: number: 80 pathType: ImplementationSpecific |
Execute the following command to access the service.
curl -H "Host: www.example.com" http://<EXTERNAL_IP> |
Expected output:
old |
curl -H "Host: www.example.com" -H "foo: bar" http://<EXTERNAL_IP> |
Expected output:
new |
Repeat the above command. As you can see, only client requests with "foo=bar" in the request header can route 50% of the traffic to the service of the new version.
Repeat the above command. As you can see, only client requests with "foo=bar" in the request header can route 50% of the traffic to the service of the new version.
Modify the Ingress created in step 2 as follows.
apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: gray-release-canary annotations: # Enable Canary. nginx.ingress.kubernetes.io/canary: "true" # Only 50% of the traffic will be routed to the service new-nginx of the new version. # The default total value is 100. nginx.ingress.kubernetes.io/canary-weight: "50" spec: rules: - host: www.example.com http: paths: # Service of the new version. - path: / backend: service: name: new-nginx port: number: 80 pathType: ImplementationSpecific |
curl -H "Host: www.example.com" http://<EXTERNAL_IP> |
Repeat the above command, and you will see that only 50% of the traffic is routed to the service of the new version.
Step 3: Remove the services of the old version based on Helm release management
The system operates for a period and the service of the new version run stably and satisfactorily. Then it is necessary to decommission the service of the old version and leave only the service of the new version running online. To achieve this goal, the Service of the old version must be redirected to the Deployment of the new version. The Deployment of the old version and the Service of the new version should be removed.
Modify the Service of the old version and redirect it to the Service of the new version.
apiVersion: v1 kind: Service metadata: name: old-nginx spec: ports: - port: 80 protocol: TCP targetPort: 80 selector: # Redirect to the service of the new version. run: new-nginx sessionAffinity: None type: NodePort |
Execute the following command. This allows client requests with "foo=bar" in the request header to access the service.
curl -H "Host: www.example.com" http://<EXTERNAL_IP> |
Expected output:
new |
Repeat the above command, and you will see that all requests are routed to the service of the new version.
Execute the following command to delete the Canary Ingress resource gray-release-canary.
kubectl delete ingress gray-release-canary |
Delete the Deployment of the old version and the Service of the new version.
Execute the following command to delete the Deployment of the old version.
kubectl delete deploy old-nginx |
kubectl delete svc new-nginx |