Experimenting containerized VNFs with Kubernetes VIM¶
In the past, Tacker only supports creating virtual machine based VNF using Heat. This section covers how to deploy containerized VNF using Kubernetes VIM in Tacker.
Prepare Kubernetes VIM¶
To use Kubernetes type of VNF, firstly user must register Kubernetes VIM.
Tacker supports Kubernetes authentication with two types: basic authentication
(username and password) or Bearer token. User can secure the connection to
Kubernetes cluster by providing SSL certificate. The following
vim-config.yaml file provides necessary information to register a
Kubernetes VIM.
auth_url: "https://192.168.11.110:6443"
username: "admin"
password: "admin"
project_name: "default"
ssl_ca_cert: None
type: "kubernetes"
More details about registering Kubernetes VIM, please refer 1
Sample container TOSCA templates¶
1. One container per VDU example¶
Currently, because Kubernetes does not support multiple networks such as choosing networks, connection points for applications, therefore users only deploys their applications with default networks (Pod and Service networks). In this case, user need to provide only information about VDU to create a VNF in Tacker.
The following example shows TOSCA template of containerized VNF for pure Kubernetes environment with one container per VDU.
tosca-vnfd-containerized-two-containers.yaml
tosca_definitions_version: tosca_simple_profile_for_nfv_1_0_0
description: A sample containerized VNF with two containers per VDU
metadata:
template_name: sample-tosca-vnfd
topology_template:
node_templates:
VDU1:
type: tosca.nodes.nfv.VDU.Tacker
properties:
namespace: default
mapping_ports:
- "80:80"
- "88:88"
service_type: NodePort
vnfcs:
front_end:
num_cpus: 0.5
mem_size: 512 MB
image: nginx
ports:
- "80"
rss_reader:
num_cpus: 0.5
mem_size: 512 MB
image: nickchase/rss-php-nginx:v1
ports:
- "88"
policies:
- SP1:
type: tosca.policies.tacker.Scaling
targets: [VDU1]
properties:
min_instances: 1
max_instances: 3
target_cpu_utilization_percentage: 40
default_instances: 1 # required parameter but ignored for cnf
increment: 1 # required parameter but ignored for cnf
In “vnfcs”, there are 2 components: front_end and rss_reader. We model them as Containers 2 inside a Pod 3. To provide recover ability of these containers, we put all of containers inside a Deployment 4 object, that can warrant the number of replica with auto-healing automatically.
The following table shows details about parameter of a Container. Config is translated to ConfigMap 5, when user want to update the VNF (config), equivalent ConfigMap will be updated.
+-----------------------------------------------------------------------------------------------+
| vnfcs | Example | Description |
+-----------------------------------------------------------------------------------------------+
| name | front_end | Name of container |
+-----------------------------------------------------------------------------------------------+
| num_cpus | 0.5 | Number of CPUs |
+-----------------------------------------------------------------------------------------------+
| mem_size | 512 MB | Memory size |
+-----------------------------------------------------------------------------------------------+
| image | nginx | Image to launch container |
+-----------------------------------------------------------------------------------------------+
| ports | - "80" | Exposed ports in container |
+-----------------------------------------------------------------------------------------------+
| command | ['/bin/sh','echo'] | Command when container was started |
+-----------------------------------------------------------------------------------------------+
| args | ['hello'] | Args of command |
+-----------------------------------------------------------------------------------------------+
| config | param0: key1 | Set variables |
| | param1: key2 | |
+-----------------------------------------------------------------------------------------------+
In Tacker, VDU is modeled as a Service 6 in Kubernetes. Because Pods can be easily replaced by others, when the number of replica increased, workload should be shared between Pods. To do this task, we model VDU as Service, it acts as a Load balancer for Pods. Currently, we support some parameters as the following table.
+--------------------------------------------------------------------------------------------------------------------------------+
| VDU properties | Example | Description |
+--------------------------------------------------------------------------------------------------------------------------------+
| namespace | default | Namespace in Kubernetes where all objects are deployed |
+--------------------------------------------------------------------------------------------------------------------------------+
| mapping_ports | - "443:443" | Published ports and target ports (container ports) of Service Kubernetes |
| | - "80:8080" | |
+--------------------------------------------------------------------------------------------------------------------------------+
| labels | "app: webserver" | Labels which is set for Kubernetes objects, it is used as Selector to |
| | | Service can send requests to Pods |
+--------------------------------------------------------------------------------------------------------------------------------+
| service_type | ClusterIP | Set service type for Service object. |
| | | |
+--------------------------------------------------------------------------------------------------------------------------------+
| vnfcs | | Vnfcs are modeled by Containers and Deployment object. User can limit |
| | | resource, set image, publish container ports, set commands and variables |
+--------------------------------------------------------------------------------------------------------------------------------+
User can also set scaling policy for VDU by adding the following policy. These information is translated to Horizontal Pod Autoscaler in Kubernetes. In the current scope, we just support auto-scaling with CPU utilization, more metrics will be added in the future.
policies:
- SP1:
type: tosca.policies.tacker.Scaling
targets: [VDU1]
properties:
min_instances: 1
max_instances: 3
target_cpu_utilization_percentage: 40
default_instances: 1 # required parameter but ignored for cnf
increment: 1 # required parameter but ignored for cnf
2. Two containers per VDU example¶
Similar to the above example, in this scenario, we define 2 containers in VDU1.
tosca-vnfd-containerized.yaml
tosca_definitions_version: tosca_simple_profile_for_nfv_1_0_0
description: A sample containerized VNF with two containers per VDU
metadata:
template_name: sample-tosca-vnfd
topology_template:
node_templates:
VDU1:
type: tosca.nodes.nfv.VDU.Tacker
properties:
namespace: default
mapping_ports:
- "80:8080"
labels:
- "app: webserver"
service_type: ClusterIP
vnfcs:
web_server:
num_cpus: 0.5
mem_size: 512 MB
image: celebdor/kuryr-demo
ports:
- "8080"
config: |
param0: key1
param1: key2
policies:
- SP1:
type: tosca.policies.tacker.Scaling
targets: [VDU1]
properties:
min_instances: 1
max_instances: 3
target_cpu_utilization_percentage: 40
default_instances: 1 # required parameter but ignored for cnf
increment: 1 # required parameter but ignored for cnf
Viewing a containerized VNF¶
Create sample containerized VNF
$ openstack vnf descriptor create --vnfd-file tosca-vnfd-containerized.yaml VNFD1
Created a new vnfd:
+-----------------+-------------------------------------------------------------------------------------------------------+
| Field | Value |
+-----------------+-------------------------------------------------------------------------------------------------------+
| created_at | 2018-01-21 14:36:51.757044 |
| description | A sample containerized VNF with one container per VDU |
| id | fb4a0aa8-e410-4e73-abdc-d2808de155ef |
| name | VNFD1 |
| service_types | vnfd |
| template_source | onboarded |
| tenant_id | 2d22508be9694091bb2f03ce27911416 |
| updated_at | |
+-----------------+-------------------------------------------------------------------------------------------------------+
$ openstack vnf create --vnfd-name VNFD1 --vim-name vim-kubernetes VNF1
Created a new vnf:
+----------------+-------------------------------------------------------------------------------------------------------+
| Field | Value |
+----------------+-------------------------------------------------------------------------------------------------------+
| created_at | 2018-01-21 14:37:23.318018 |
| description | A sample containerized VNF with one container per VDU |
| error_reason | |
| id | 1faf776b-8d2b-4ee6-889d-e3b7c7310411 |
| instance_id | default,svc-vdu1-05db44 |
| mgmt_ip_address| |
| name | VNF1 |
| placement_attr | {"vim_name": "vim-kubernetes"} |
| status | PENDING_CREATE |
| tenant_id | 2d22508be9694091bb2f03ce27911416 |
| updated_at | |
| vim_id | 791830a6-45fd-468a-bd85-e07fe24e5ce3 |
| vnfd_id | fb4a0aa8-e410-4e73-abdc-d2808de155ef |
+----------------+-------------------------------------------------------------------------------------------------------+
$ openstack vnf list
+--------------------------------------+------+----------------------------+--------+--------------------------------------+--------------------------------------+
| id | name | mgmt_ip_address | status | vim_id | vnfd_id |
+--------------------------------------+------+----------------------------+--------+--------------------------------------+--------------------------------------+
| 1faf776b-8d2b-4ee6-889d-e3b7c7310411 | VNF1 | | ACTIVE | 791830a6-45fd-468a-bd85-e07fe24e5ce3 | fb4a0aa8-e410-4e73-abdc-d2808de155ef |
+--------------------------------------+------+----------------------------+--------+--------------------------------------+--------------------------------------+
To test VNF is running in Kubernetes environment, we can check by running following commands
$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 192.168.28.129 <none> 443/TCP 5h
svc-vdu1-05db44 ClusterIP 192.168.28.187 <none> 80/TCP 12m
$ kubectl get deployment
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
svc-vdu1-05db44 1 1 1 1 16m
$ kubectl get pod
NAME READY STATUS RESTARTS AGE
svc-vdu1-05db44-7dcb6b955d-wkh7d 1/1 Running 0 18m
$ kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
svc-vdu1-05db44 Deployment/svc-vdu1-05db44 <unknown> / 40% 1 3 1 17m
$ kubectl get configmap
NAME DATA AGE
svc-vdu1-05db44 2 17m
User also can scale VNF manually, by running the following commands:
$ openstack vnf scale --scaling-policy-name SP1 --scaling-type out VNF1
$ kubectl get deployment
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
svc-vdu1-651815 2 2 2 1 3h
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
svc-vdu1-651815-5b894b8bfb-b6mzq 2/2 Running 0 3h
svc-vdu1-651815-5b894b8bfb-b7f2c 2/2 Running 0 40s
In the same way, user also scale in VNF with scaling-type is ‘in’. The range of scaling manually is limited by ‘min_instances’ and ‘max_instances’ user provide in VNF template.
Multi-Interface for C-VNF¶
To use multi-interface for C-VNF, User should follow below procedure.
1. Checking kuryr.conf¶
After installation, user should check kuryr.conf configuration.
$ sudo cat /etc/kuryr/kuryr.conf | grep multi_vif_drivers
multi_vif_drivers = npwg_multiple_interfaces
2. Adding K8s CustomResourceDefinition¶
To use CustomResourceDefinition, user needs to add CRD. User can make a additional network using yaml file like below. Create yaml file like below and register it.
$ cat ./crdnetwork.yaml
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: network-attachment-definitions.k8s.cni.cncf.io
spec:
group: k8s.cni.cncf.io
version: v1
scope: Namespaced
names:
plural: network-attachment-definitions
singular: network-attachment-definition
kind: NetworkAttachmentDefinition
shortNames:
- net-attach-def
validation:
openAPIV3Schema:
properties:
spec:
properties:
config:
type: string
Register crdnetwork.yaml
$ kubectl create -f ~/crdnetwork.yaml
Get crd list
$ kubectl get crd
NAME CREATED AT
kuryrnetpolicies.openstack.org 2019-07-31T02:23:54Z
kuryrnets.openstack.org 2019-07-31T02:23:54Z
network-attachment-definitions.k8s.cni.cncf.io 2019-07-31T02:23:55Z
3. Adding neutron subnet id information to k8s CRD¶
To use neutron subnet in kubernetes, user should register neutron subnet to CRD. At first, user should create subnet.yaml file.
$ cat ./kuryr-subnetname1.yaml
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
name: subnetname1
annotations:
openstack.org/kuryr-config: '{"subnetId": "$subnet_id"}'
After making a yaml file, user should create subnet with yaml file.
$ kubectl create -f ~/kuryr-subnetname1.yaml
After created, user can check subnet info like below.
$ kubectl get net-attach-def
NAME AGE
k8s-multi-10 7d
k8s-multi-11 7d
Known Issues and Limitations¶
Does not support Volumes in Kubernetes
Horizontal Pod AutoScaler only support CPU utilization
Add support Kuryr-Kubernetes for making hybrid network in the future
References¶
- 1
- 2
https://kubernetes.io/docs/concepts/workloads/pods/init-containers
- 3
https://kubernetes.io/docs/concepts/workloads/pods/pod-overview
- 4
https://kubernetes.io/docs/concepts/workloads/controllers/deployment
- 5
https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap
- 6
https://kubernetes.io/docs/concepts/services-networking/service
