Yardstick¶

Documents¶

• User Guide: <yardstick:userguide>
• Developer Guide: <yardstick:devguide>

Software Deliverables¶

• The Yardstick Docker image: https://hub.docker.com/r/opnfv/yardstick (tag: opnfv-7.0.0)

New Test cases¶

• Generic NFVI test cases

• (e.g.) OPNFV_YARDSTICK_TCO84 - SPEC CPU 2006 for VM

• HA Test cases

• (e.g.) OPNFV_YARDSTICK_TC093 - SDN Vswitch resilience in non-HA or HA configuration

Module Version Changes¶

This is the seventh tracked release of Yardstick. It is based on following upstream versions:

• OpenStack Queens
• OpenDayLight Oxygen

Document Version Changes¶

This is the seventh tracked version of the Yardstick framework in OPNFV. It includes the following documentation updates:

• Yardstick User Guide:
  • Remove vTC chapter;
• Yardstick Developer Guide
• Yardstick Release Notes for Yardstick: this document

Feature additions¶

• Simplify Yardstick installation to use a single ansible playbook (nsb_setup.yaml)….
• Spirent support.
• vEPC testcases.
• Agnostic VNF tests cases for reuse of standard RFC-2544 test case.
• PROX enhancements and the addition of Standalone test case using SRIOV and OVS-DPDK.
• Ixia enhancements for vBNG and PPPoE traffic.
• Improvements of unit tests and gating.
• Add DPDK pktgen traffic generator.
• Kubernetes context enhancements.
• Kubernetes sample test cases of fio and lmbench added.

Known Issues/Faults¶

Corrected Faults¶

Gambia 7.2.0:

Architecture overview¶

Yardstick is mainly written in Python, and test configurations are made in YAML. Documentation is written in reStructuredText format, i.e. .rst files. Yardstick is inspired by Rally. Yardstick is intended to run on a computer with access and credentials to a cloud. The test case is described in a configuration file given as an argument.

How it works: the benchmark task configuration file is parsed and converted into an internal model. The context part of the model is converted into a Heat template and deployed into a stack. Each scenario is run using a runner, either serially or in parallel. Each runner runs in its own subprocess executing commands in a VM using SSH. The output of each scenario is written as json records to a file or influxdb or http server, we use influxdb as the backend, the test result will be shown with grafana.

Concept¶

Benchmark - assess the relative performance of something

Benchmark configuration file - describes a single test case in yaml format

Context - The set of Cloud resources used by a scenario, such as user names, image names, affinity rules and network configurations. A context is converted into a simplified Heat template, which is used to deploy onto the Openstack environment.

Data - Output produced by running a benchmark, written to a file in json format

Runner - Logic that determines how a test scenario is run and reported, for example the number of test iterations, input value stepping and test duration. Predefined runner types exist for re-usage, see Runner types.

Scenario - Type/class of measurement for example Ping, Pktgen, (Iperf, LmBench, …)

SLA - Relates to what result boundary a test case must meet to pass. For example a latency limit, amount or ratio of lost packets and so on. Action based on SLA can be configured, either just to log (monitor) or to stop further testing (assert). The SLA criteria is set in the benchmark configuration file and evaluated by the runner.

Runner types¶

There exists several predefined runner types to choose between when designing a test scenario:

Arithmetic: Every test run arithmetically steps the specified input value(s) in the test scenario, adding a value to the previous input value. It is also possible to combine several input values for the same test case in different combinations.

Snippet of an Arithmetic runner configuration:

runner:
    type: Arithmetic
    iterators:
    -
      name: stride
      start: 64
      stop: 128
      step: 64

Duration: The test runs for a specific period of time before completed.

Snippet of a Duration runner configuration:

runner:
  type: Duration
  duration: 30

Sequence: The test changes a specified input value to the scenario. The input values to the sequence are specified in a list in the benchmark configuration file.

Snippet of a Sequence runner configuration:

runner:
  type: Sequence
  scenario_option_name: packetsize
  sequence:
  - 100
  - 200
  - 250

Iteration: Tests are run a specified number of times before completed.

Snippet of an Iteration runner configuration:

runner:
  type: Iteration
  iterations: 2

Prepare the Yardstick container¶

Install docker on your guest system with the following command, if not done yet:

wget -qO- https://get.docker.com/ | sh

Pull the Yardstick Docker image (opnfv/yardstick) from the public dockerhub registry under the OPNFV account in dockerhub, with the following docker command:

sudo -EH docker pull opnfv/yardstick:stable

After pulling the Docker image, check that it is available with the following docker command:

[yardsticker@jumphost ~]$ docker images
REPOSITORY         TAG       IMAGE ID        CREATED      SIZE
opnfv/yardstick    stable    a4501714757a    1 day ago    915.4 MB

Run the Docker image to get a Yardstick container:

docker run -itd --privileged -v /var/run/docker.sock:/var/run/docker.sock \
   -p 8888:5000 --name yardstick opnfv/yardstick:stable

Description of the parameters used with docker run command

Parameters	Detail
-itd	-i: interactive, Keep STDIN open even if not attached
	-t: allocate a pseudo-TTY detached mode, in the background
–privileged	If you want to build yardstick-image in Yardstick container, this parameter is needed
-p 8888:5000	Redirect the a host port (8888) to a container port (5000)
-v /var/run/docker.sock :/var/run/docker.sock	If you want to use yardstick env grafana/influxdb to create a grafana/influxdb container out of Yardstick container
–name yardstick	The name for this container

If the host is restarted¶

The yardstick container must be started if the host is rebooted:

docker start yardstick

Configure the Yardstick container environment¶

There are three ways to configure environments for running Yardstick, explained in the following sections. Before that, access the Yardstick container:

docker exec -it yardstick /bin/bash

and then configure Yardstick environments in the Yardstick container.

yardstick env prepare

'-v /path/to/credential_file:/etc/yardstick/openstack.creds'

tail -f /var/log/yardstick/uwsgi.log

source openrc

OS_AUTH_URL
OS_USERNAME
OS_PASSWORD
OS_PROJECT_NAME
EXTERNAL_NETWORK

export OS_PASSWORD=console
export OS_PROJECT_NAME=admin
export OS_AUTH_URL=http://172.16.1.222:35357/v2.0
export OS_USERNAME=admin
export OS_VOLUME_API_VERSION=2
export EXTERNAL_NETWORK=net04_ext

openstack flavor create --disk 3 --vcpus 1 --ram 512 --swap 100 \
   yardstick-flavor

sudo -EH apt-get update && sudo -EH apt-get install -y qemu-utils kpartx

export YARD_IMG_ARCH='amd64'
echo "Defaults env_keep += \'YARD_IMG_ARCH\'" | sudo tee --append \
   /etc/sudoers > /dev/null
sudo -EH tools/yardstick-img-modify tools/ubuntu-server-cloudimg-modify.sh

openstack image create --disk-format qcow2 --container-format bare \
   --public --file /tmp/workspace/yardstick/yardstick-image.img \
    yardstick-image

openstack image create --disk-format qcow2 --container-format bare \
   --public --file $cirros_image_file cirros-0.3.5
openstack image create --disk-format qcow2 --container-format bare \
   --file $ubuntu_image_file Ubuntu-16.04

source $YARDSTICK_REPO_DIR/tests/ci/load_images.sh

The Yardstick container GUI¶

In Euphrates release, Yardstick implemented a GUI for Yardstick Docker container. After booting up Yardstick container, you can visit the GUI at <container_host_ip>:8888/gui/index.html.

For usage of Yardstick GUI, please watch our demo video at Yardstick GUI demo.

Delete the Yardstick container¶

If you want to uninstall Yardstick, just delete the Yardstick container:

sudo docker stop yardstick && docker rm yardstick

Install Yardstick¶

Prerequisite preparation:

sudo -EH apt-get update && sudo -EH apt-get install -y \
   git python-setuptools python-pip
sudo -EH easy_install -U setuptools==30.0.0
sudo -EH pip install appdirs==1.4.0
sudo -EH pip install virtualenv

Download the source code and install Yardstick from it:

git clone https://gerrit.opnfv.org/gerrit/yardstick
export YARDSTICK_REPO_DIR=~/yardstick
cd ~/yardstick
sudo -EH ./install.sh

If the host is ever restarted, nginx and uwsgi need to be restarted:

service nginx restart
uwsgi -i /etc/yardstick/yardstick.ini

Configure the Yardstick environment (Todo)¶

For installing Yardstick directly in Ubuntu, the yardstick env command is not available. You need to prepare OpenStack environment variables and create Yardstick flavor and guest images manually.

Uninstall Yardstick¶

For uninstalling Yardstick, just delete the virtual environment:

rm -rf ~/yardstick_venv

Install Yardstick¶

Prerequisite preparation:

sudo -EH zypper -n install -y gcc \
   wget \
   git \
   sshpass \
   qemu-tools \
   kpartx \
   libffi-devel \
   libopenssl-devel \
   python \
   python-devel \
   python-virtualenv \
   libxml2-devel \
   libxslt-devel \
   python-setuptools-git

Create a virtual environment:

virtualenv ~/yardstick_venv
export YARDSTICK_VENV=~/yardstick_venv
source ~/yardstick_venv/bin/activate
sudo -EH easy_install -U setuptools

Download the source code and install Yardstick from it:

git clone https://gerrit.opnfv.org/gerrit/yardstick
export YARDSTICK_REPO_DIR=~/yardstick
cd yardstick
sudo -EH python setup.py install
sudo -EH pip install -r requirements.txt

Install missing python modules:

sudo -EH pip install pyyaml \
   oslo_utils \
   oslo_serialization \
   oslo_config \
   paramiko \
   python.heatclient \
   python.novaclient \
   python.glanceclient \
   python.neutronclient \
   scp \
   jinja2

Configure the Yardstick environment¶

Source the OpenStack environment variables:

source DEVSTACK_DIRECTORY/openrc

Export the Openstack external network. The default installation of Devstack names the external network public:

export EXTERNAL_NETWORK=public
export OS_USERNAME=demo

Change the API version used by Yardstick to v2.0 (the devstack openrc sets it to v3):

export OS_AUTH_URL=http://PUBLIC_IP_ADDRESS:5000/v2.0

Uninstall Yardstick¶

For unistalling Yardstick, just delete the virtual environment:

rm -rf ~/yardstick_venv

Automatic deployment of InfluxDB and Grafana containers (recommended)¶

1. Enter the Yardstick container:

   sudo -EH docker exec -it yardstick /bin/bash
   

2. Create InfluxDB container and configure with the following command:

   yardstick env influxdb
   

3. Create and configure Grafana container:

   yardstick env grafana
   

Then you can run a test case and visit http://host_ip:1948 (admin/admin) to see the results.

docker version

Manual deployment of InfluxDB and Grafana containers¶

You can also deploy influxDB and Grafana containers manually on the Jumphost. The following sections show how to do.

Pull docker images:

sudo -EH docker pull tutum/influxdb
sudo -EH docker pull grafana/grafana

Run influxDB:

sudo -EH docker run -d --name influxdb \
   -p 8083:8083 -p 8086:8086 --expose 8090 --expose 8099 \
   tutum/influxdb
docker exec -it influxdb influx

Configure influxDB:

> CREATE USER root WITH PASSWORD 'root' WITH ALL PRIVILEGES
> CREATE DATABASE yardstick;
> use yardstick;
> show MEASUREMENTS;
> quit

Run Grafana:

sudo -EH docker run -d --name grafana -p 1948:3000 grafana/grafana

Log on to http://{YOUR_IP_HERE}:1948 using admin/admin and configure database resource to be {YOUR_IP_HERE}:8086.

Configure yardstick.conf:

sudo -EH docker exec -it yardstick /bin/bash
sudo cp etc/yardstick/yardstick.conf.sample /etc/yardstick/yardstick.conf
sudo vi /etc/yardstick/yardstick.conf

Modify yardstick.conf to add the influxdb dispatcher:

[DEFAULT]
debug = True
dispatcher = influxdb

[dispatcher_influxdb]
timeout = 5
target = http://{YOUR_IP_HERE}:8086
db_name = yardstick
username = root
password = root

Now Yardstick will store results in InfluxDB when you run a testcase.

List test cases¶

yardstick testcase list: This command line would list all test cases in Yardstick. It would show like below:

+---------------------------------------------------------------------------------------
| Testcase Name         | Description
+---------------------------------------------------------------------------------------
| opnfv_yardstick_tc001 | Measure network throughput using pktgen
| opnfv_yardstick_tc002 | measure network latency using ping
| opnfv_yardstick_tc005 | Measure Storage IOPS, throughput and latency using fio.
...
+---------------------------------------------------------------------------------------

Show a test case config file¶

Take opnfv_yardstick_tc002 for an example. This test case measure network latency. You just need to type in yardstick testcase show opnfv_yardstick_tc002, and the console would show the config yaml of this test case:

---

schema: "yardstick:task:0.1"
description: >
    Yardstick TC002 config file;
    measure network latency using ping;

{% set image = image or "cirros-0.3.5" %}

{% set provider = provider or none %}
{% set physical_network = physical_network or 'physnet1' %}
{% set segmentation_id = segmentation_id or none %}
{% set packetsize = packetsize or 100 %}

scenarios:
{% for i in range(2) %}
-
  type: Ping
  options:
    packetsize: {{packetsize}}
  host: athena.demo
  target: ares.demo

  runner:
    type: Duration
    duration: 60
    interval: 10

  sla:
    max_rtt: 10
    action: monitor
{% endfor %}

context:
  name: demo
  image: {{image}}
  flavor: yardstick-flavor
  user: cirros

  placement_groups:
    pgrp1:
      policy: "availability"

  servers:
    athena:
      floating_ip: true
      placement: "pgrp1"
    ares:
      placement: "pgrp1"

  networks:
    test:
      cidr: '10.0.1.0/24'
      {% if provider == "vlan" or provider == "sriov" %}
      provider: {{provider}}
      physical_network: {{physical_network}}
        {% if segmentation_id %}
      segmentation_id: {{segmentation_id}}
        {% endif %}
      {% endif %}

Run a Yardstick test case¶

If you want run a test case, then you need to use yardstick task start <test_case_path> this command support some parameters as below:

Parameters	Detail
-d	show debug log of yardstick running
–task-args	If you want to customize test case parameters, use “–task-args” to pass the value. The format is a json string with parameter key-value pair.
–task-args-file	If you want to use yardstick env prepare command(or related API) to load the
–parse-only
–output-file OUTPUT_FILE_PATH	Specify where to output the log. if not pass, the default value is “/tmp/yardstick/yardstick.log”
–suite TEST_SUITE_PATH	run a test suite, TEST_SUITE_PATH specify where the test suite locates

Defining the testcase scenarios¶

TODO

Defining the testcase context(s)¶

Each testcase consists of one or more contexts, which describe the environment in which the testcase will be run. Current available contexts are:

• Dummy: this is a no-op context, and is used when there is no environment to set up e.g. when testing whether OpenStack services are available
• Node: this context is used to perform operations on baremetal servers
• Heat: uses OpenStack to provision the required hosts, networks, etc.
• Kubernetes: uses Kubernetes to provision the resources required for the test.

Regardless of the context type, the context section of the testcase will consist of the following:

context:
  name: demo
  type: Dummy|Node|Heat|Kubernetes

The content of the context section will vary based on the context type.

context:
  name: my_context
  type: Dummy

# Sample Heat context config with Dummy context

schema: "yardstick:task:0.1"

scenarios:
-
  type: Dummy

  runner:
    type: Duration
    duration: 5
    interval: 1

context:
  name: {{ context_name }}
  image: yardstick-image
  flavor: yardstick-flavor
  user: ubuntu

  servers:
    athena:
      name: athena
    ares:
      name: ares

  networks:
    test:
      name: test

contexts:
-
  name: context1
  type: Heat
  ...
-
  name: context2
  type: Node
  ...

context:
  name: mycontext
  type: Heat
  flags:
    no_setup: True
    no_teardown: True
  ...

Step 0: Environment preparation¶

Running Storperf on Jump Host Requirements:

• Docker must be installed
• Jump Host must have access to the OpenStack Controller API
• Jump Host must have internet connectivity for downloading docker image
• Enough floating IPs must be available to match your agent count

Before installing Storperf into yardstick you need to check your openstack environment and other dependencies:

1. Make sure docker is installed.
2. Make sure Keystone, Nova, Neutron, Glance, Heat are installed correctly.
3. Make sure Jump Host have access to the OpenStack Controller API.
4. Make sure Jump Host must have internet connectivity for downloading docker image.
5. You need to know where to get basic openstack Keystone authorization info, such as OS_PASSWORD, OS_PROJECT_NAME, OS_AUTH_URL, OS_USERNAME.
6. To run a Storperf container, you need to have OpenStack Controller environment variables defined and passed to Storperf container. The best way to do this is to put environment variables in a “storperf_admin-rc” file. The storperf_admin-rc should include credential environment variables at least:
   • OS_AUTH_URL
   • OS_USERNAME
   • OS_PASSWORD
   • OS_PROJECT_NAME
   • OS_PROJECT_ID
   • OS_USER_DOMAIN_ID

Yardstick has a prepare_storperf_admin-rc.sh script which can be used to generate the storperf_admin-rc file, this script is located at test/ci/prepare_storperf_admin-rc.sh

#!/bin/bash
# Prepare storperf_admin-rc for StorPerf.
AUTH_URL=${OS_AUTH_URL}
USERNAME=${OS_USERNAME:-admin}
PASSWORD=${OS_PASSWORD:-console}

# OS_TENANT_NAME is still present to keep backward compatibility with legacy
# deployments, but should be replaced by OS_PROJECT_NAME.
TENANT_NAME=${OS_TENANT_NAME:-admin}
PROJECT_NAME=${OS_PROJECT_NAME:-$TENANT_NAME}
PROJECT_ID=`openstack project show admin|grep '\bid\b' |awk -F '|' '{print $3}'|sed -e 's/^[[:space:]]*//'`
USER_DOMAIN_ID=${OS_USER_DOMAIN_ID:-default}

rm -f ~/storperf_admin-rc
touch ~/storperf_admin-rc

echo "OS_AUTH_URL="$AUTH_URL >> ~/storperf_admin-rc
echo "OS_USERNAME="$USERNAME >> ~/storperf_admin-rc
echo "OS_PASSWORD="$PASSWORD >> ~/storperf_admin-rc
echo "OS_PROJECT_NAME="$PROJECT_NAME >> ~/storperf_admin-rc
echo "OS_PROJECT_ID="$PROJECT_ID >> ~/storperf_admin-rc
echo "OS_USER_DOMAIN_ID="$USER_DOMAIN_ID >> ~/storperf_admin-rc

The generated storperf_admin-rc file will be stored in the root directory. If you installed Yardstick using Docker, this file will be located in the container. You may need to copy it to the root directory of the Storperf deployed host.

Step 1: Plug-in configuration file preparation¶

To install a plug-in, first you need to prepare a plug-in configuration file in YAML format and store it in the “plugin” directory. The plugin configration file work as the input of yardstick “plugin” command. Below is the Storperf plug-in configuration file sample:

---
# StorPerf plugin configuration file
# Used for integration StorPerf into Yardstick as a plugin
schema: "yardstick:plugin:0.1"
plugins:
  name: storperf
deployment:
  ip: 192.168.23.2
  user: root
  password: root

In the plug-in configuration file, you need to specify the plug-in name and the plug-in deployment info, including node ip, node login username and password. Here the Storperf will be installed on IP 192.168.23.2 which is the Jump Host in my local environment.

Step 2: Plug-in install/remove scripts preparation¶

In yardstick/resource/scripts directory, there are two folders: an install folder and a remove folder. You need to store the plug-in install/remove scripts in these two folders respectively.

The detailed installation or remove operation should de defined in these two scripts. The name of both install and remove scripts should match the plugin-in name that you specified in the plug-in configuration file.

For example, the install and remove scripts for Storperf are both named storperf.bash.

Step 3: Install and remove Storperf¶

To install Storperf, simply execute the following command:

# Install Storperf
yardstick plugin install plugin/storperf.yaml

# Remove Storperf
yardstick plugin remove plugin/storperf.yaml

/yardstick/env/action¶

Description: This API is used to prepare Yardstick test environment. For Euphrates, it supports:

1. Prepare yardstick test environment, including setting the EXTERNAL_NETWORK environment variable, load Yardstick VM images and create flavors;
2. Start an InfluxDB Docker container and config Yardstick output to InfluxDB;
3. Start a Grafana Docker container and config it with the InfluxDB.

Which API to call will depend on the parameters.

Method: POST

Prepare Yardstick test environment Example:

{
    'action': 'prepare_env'
}

This is an asynchronous API. You need to call /yardstick/asynctask API to get the task result.

Start and config an InfluxDB docker container Example:

{
    'action': 'create_influxdb'
}

This is an asynchronous API. You need to call /yardstick/asynctask API to get the task result.

Start and config a Grafana docker container Example:

{
    'action': 'create_grafana'
}

This is an asynchronous API. You need to call /yardstick/asynctask API to get the task result.

/yardstick/asynctask¶

Description: This API is used to get the status of asynchronous tasks

Method: GET

Get the status of asynchronous tasks Example:

http://<SERVER IP>:<PORT>/yardstick/asynctask?task_id=3f3f5e03-972a-4847-a5f8-154f1b31db8c

The returned status will be 0(running), 1(finished) and 2(failed).

NOTE:

<SERVER IP>: The ip of the host where you start your yardstick container
<PORT>: The outside port of port mapping which set when you start start yardstick container

/yardstick/testcases¶

Description: This API is used to list all released Yardstick test cases.

Method: GET

Get a list of released test cases Example:

http://<SERVER IP>:<PORT>/yardstick/testcases

/yardstick/testcases/release/action¶

Description: This API is used to run a Yardstick released test case.

Method: POST

Run a released test case Example:

{
    'action': 'run_test_case',
    'args': {
        'opts': {},
        'testcase': 'opnfv_yardstick_tc002'
    }
}

This is an asynchronous API. You need to call /yardstick/results to get the result.

/yardstick/testcases/samples/action¶

Description: This API is used to run a Yardstick sample test case.

Method: POST

Run a sample test case Example:

{
    'action': 'run_test_case',
    'args': {
        'opts': {},
        'testcase': 'ping'
    }
}

This is an asynchronous API. You need to call /yardstick/results to get the result.

/yardstick/testcases/<testcase_name>/docs¶

Description: This API is used to the documentation of a certain released test case.

Method: GET

Get the documentation of a certain test case Example:

http://<SERVER IP>:<PORT>/yardstick/taskcases/opnfv_yardstick_tc002/docs

/yardstick/testsuites/action¶

Description: This API is used to run a Yardstick test suite.

Method: POST

Run a test suite Example:

{
    'action': 'run_test_suite',
    'args': {
        'opts': {},
        'testsuite': 'opnfv_smoke'
    }
}

This is an asynchronous API. You need to call /yardstick/results to get the result.

/yardstick/tasks/<task_id>/log¶

Description: This API is used to get the real time log of test case execution.

Method: GET

Get real time of test case execution Example:

http://<SERVER IP>:<PORT>/yardstick/tasks/14795be8-f144-4f54-81ce-43f4e3eab33f/log?index=0

/yardstick/results¶

Description: This API is used to get the test results of tasks. If you call /yardstick/testcases/samples/action API, it will return a task id. You can use the returned task id to get the results by using this API.

Method: GET

Get test results of one task Example:

http://<SERVER IP>:<PORT>/yardstick/results?task_id=3f3f5e03-972a-4847-a5f8-154f1b31db8c

This API will return a list of test case result

/api/v2/yardstick/openrcs¶

Description: This API provides functionality of handling OpenStack credential file (openrc). For Euphrates, it supports:

1. Upload an openrc file for an OpenStack environment;
2. Update an openrc;
3. Get openrc file information;
4. Delete an openrc file.

Which API to call will depend on the parameters.

METHOD: POST

Upload an openrc file for an OpenStack environment Example:

{
    'action': 'upload_openrc',
    'args': {
        'file': file,
        'environment_id': environment_id
    }
}

METHOD: POST

Update an openrc file Example:

{
    'action': 'update_openrc',
    'args': {
        'openrc': {
            "EXTERNAL_NETWORK": "ext-net",
            "OS_AUTH_URL": "http://192.168.23.51:5000/v3",
            "OS_IDENTITY_API_VERSION": "3",
            "OS_IMAGE_API_VERSION": "2",
            "OS_PASSWORD": "console",
            "OS_PROJECT_DOMAIN_NAME": "default",
            "OS_PROJECT_NAME": "admin",
            "OS_USERNAME": "admin",
            "OS_USER_DOMAIN_NAME": "default"
        },
        'environment_id': environment_id
    }
}

/api/v2/yardstick/openrcs/<openrc_id>¶

Description: This API provides functionality of handling OpenStack credential file (openrc). For Euphrates, it supports:

1. Get openrc file information;
2. Delete an openrc file.

METHOD: GET

Get openrc file information Example:

http://<SERVER IP>:<PORT>/api/v2/yardstick/openrcs/5g6g3e02-155a-4847-a5f8-154f1b31db8c

METHOD: DELETE

Delete openrc file Example:

http://<SERVER IP>:<PORT>/api/v2/yardstick/openrcs/5g6g3e02-155a-4847-a5f8-154f1b31db8c

/api/v2/yardstick/pods¶

Description: This API provides functionality of handling Yardstick pod file (pod.yaml). For Euphrates, it supports:

1. Upload a pod file;

Which API to call will depend on the parameters.

METHOD: POST

Upload a pod.yaml file Example:

{
    'action': 'upload_pod_file',
    'args': {
        'file': file,
        'environment_id': environment_id
    }
}

/api/v2/yardstick/pods/<pod_id>¶

Description: This API provides functionality of handling Yardstick pod file (pod.yaml). For Euphrates, it supports:

1. Get pod file information;
2. Delete an openrc file.

METHOD: GET

Get pod file information Example:

http://<SERVER IP>:<PORT>/api/v2/yardstick/pods/5g6g3e02-155a-4847-a5f8-154f1b31db8c

METHOD: DELETE

Delete openrc file Example:

http://<SERVER IP>:<PORT>/api/v2/yardstick/pods/5g6g3e02-155a-4847-a5f8-154f1b31db8c

/api/v2/yardstick/images¶

Description: This API is used to do some work related to Yardstick VM images. For Euphrates, it supports:

1. Load Yardstick VM images;

Which API to call will depend on the parameters.

METHOD: POST

Load VM images Example:

{
    'action': 'load_image',
    'args': {
        'name': 'yardstick-image'
    }
}

/api/v2/yardstick/images/<image_id>¶

Description: This API is used to do some work related to Yardstick VM images. For Euphrates, it supports:

1. Get image’s information;
2. Delete images

METHOD: GET

Get image information Example:

http://<SERVER IP>:<PORT>/api/v2/yardstick/images/5g6g3e02-155a-4847-a5f8-154f1b31db8c

METHOD: DELETE

Delete images Example:

http://<SERVER IP>:<PORT>/api/v2/yardstick/images/5g6g3e02-155a-4847-a5f8-154f1b31db8c

/api/v2/yardstick/tasks¶

Description: This API is used to do some work related to yardstick tasks. For Euphrates, it supports:

1. Create a Yardstick task;

Which API to call will depend on the parameters.

METHOD: POST

Create a Yardstick task Example:

{
    'action': 'create_task',
        'args': {
            'name': 'task1',
            'project_id': project_id
        }
}

/api/v2/yardstick/tasks/<task_id>¶

Description: This API is used to do some work related to yardstick tasks. For Euphrates, it supports:

1. Add a environment to a task
2. Add a test case to a task;
3. Add a test suite to a task;
4. run a Yardstick task;
5. Get a tasks’ information;
6. Delete a task.

METHOD: PUT

Add a environment to a task

Example:

{
    'action': 'add_environment',
    'args': {
        'environment_id': 'e3cadbbb-0419-4fed-96f1-a232daa0422a'
    }
}

METHOD: PUT

Add a test case to a task Example:

{
    'action': 'add_case',
    'args': {
        'case_name': 'opnfv_yardstick_tc002',
        'case_content': case_content
    }
}

METHOD: PUT

Add a test suite to a task Example:

{
    'action': 'add_suite',
    'args': {
        'suite_name': 'opnfv_smoke',
        'suite_content': suite_content
    }
}

METHOD: PUT

Run a task

Example:

{
    'action': 'run'
}

METHOD: GET

Get a task’s information Example:

http://<SERVER IP>:<PORT>/api/v2/yardstick/tasks/5g6g3e02-155a-4847-a5f8-154f1b31db8c

METHOD: DELETE

Delete a task

Example:

http://<SERVER IP>:<PORT>/api/v2/yardstick/tasks/5g6g3e02-155a-4847-a5f8-154f1b31db8c

/api/v2/yardstick/testcases¶

Description: This API is used to do some work related to Yardstick testcases. For Euphrates, it supports:

1. Upload a test case;
2. Get all released test cases’ information;

Which API to call will depend on the parameters.

METHOD: POST

Upload a test case Example:

{
    'action': 'upload_case',
    'args': {
        'file': file
    }
}

METHOD: GET

Get all released test cases’ information Example:

http://<SERVER IP>:<PORT>/api/v2/yardstick/testcases

/api/v2/yardstick/testcases/<case_name>¶

Description: This API is used to do some work related to yardstick testcases. For Euphrates, it supports:

1. Get certain released test case’s information;
2. Delete a test case.

METHOD: GET

Get certain released test case’s information Example:

http://<SERVER IP>:<PORT>/api/v2/yardstick/testcases/opnfv_yardstick_tc002

METHOD: DELETE

Delete a certain test case Example:

http://<SERVER IP>:<PORT>/api/v2/yardstick/testcases/opnfv_yardstick_tc002

/api/v2/yardstick/testsuites¶

Description: This API is used to do some work related to yardstick test suites. For Euphrates, it supports:

1. Create a test suite;
2. Get all test suites;

Which API to call will depend on the parameters.

METHOD: POST

Create a test suite Example:

{
    'action': 'create_suite',
    'args': {
        'name': <suite_name>,
        'testcases': [
            'opnfv_yardstick_tc002'
        ]
    }
}

METHOD: GET

Get all test suite Example:

http://<SERVER IP>:<PORT>/api/v2/yardstick/testsuites

/api/v2/yardstick/testsuites¶

Description: This API is used to do some work related to yardstick test suites. For Euphrates, it supports:

1. Get certain test suite’s information;
2. Delete a test case.

METHOD: GET

Get certain test suite’s information Example:

http://<SERVER IP>:<PORT>/api/v2/yardstick/testsuites/<suite_name>

METHOD: DELETE

Delete a certain test suite Example:

http://<SERVER IP>:<PORT>/api/v2/yardstick/testsuites/<suite_name>

/api/v2/yardstick/projects¶

Description: This API is used to do some work related to Yardstick test projects. For Euphrates, it supports:

1. Create a Yardstick project;
2. Get all projects;

Which API to call will depend on the parameters.

METHOD: POST

Create a Yardstick project Example:

{
    'action': 'create_project',
    'args': {
        'name': 'project1'
    }
}

METHOD: GET

Get all projects’ information Example:

http://<SERVER IP>:<PORT>/api/v2/yardstick/projects

/api/v2/yardstick/projects¶

Description: This API is used to do some work related to yardstick test projects. For Euphrates, it supports:

1. Get certain project’s information;
2. Delete a project.

METHOD: GET

Get certain project’s information Example:

http://<SERVER IP>:<PORT>/api/v2/yardstick/projects/<project_id>

METHOD: DELETE

Delete a certain project Example:

http://<SERVER IP>:<PORT>/api/v2/yardstick/projects/<project_id>

/api/v2/yardstick/containers¶

Description: This API is used to do some work related to Docker containers. For Euphrates, it supports:

1. Create a Grafana Docker container;
2. Create an InfluxDB Docker container;

Which API to call will depend on the parameters.

METHOD: POST

Create a Grafana Docker container Example:

{
    'action': 'create_grafana',
    'args': {
        'environment_id': <environment_id>
    }
}

METHOD: POST

Create an InfluxDB Docker container Example:

{
    'action': 'create_influxdb',
    'args': {
        'environment_id': <environment_id>
    }
}

/api/v2/yardstick/containers/<container_id>¶

Description: This API is used to do some work related to Docker containers. For Euphrates, it supports:

1. Get certain container’s information;
2. Delete a container.

METHOD: GET

Get certain container’s information Example:

http://<SERVER IP>:<PORT>/api/v2/yardstick/containers/<container_id>

METHOD: DELETE

Delete a certain container Example:

http://<SERVER IP>:<PORT>/api/v2/yardstick/containers/<container_id>

Components of Network Service¶

• Models for Network Service benchmarking: The Network Service benchmarking requires the proper modelling approach. The NSB provides models using Python files and defining of NSDs and VNFDs.

The benchmark control application being a part of OPNFV yardstick can call that python models to instantiate and configure the VNFs. Depending on infrastructure type (bare-metal or fully virtualized) that calls could be made directly or using MANO system.

• Traffic generators in NSB: Any benchmark application requires a set of traffic generator and traffic profiles defining the method in which traffic is generated.

The Network Service benchmarking model extends the Network Service definition with a set of Traffic Generators (TG) that are treated same way as other VNFs being a part of benchmarked network service. Same as other VNFs the traffic generator are instantiated and terminated.

Every traffic generator has own configuration defined as a traffic profile and a set of KPIs supported. The python models for TG is extended by specific calls to listen and generate traffic.

• The stateless TREX traffic generator: The main traffic generator used as Network Service stimulus is open source TREX tool.

The TREX tool can generate any kind of stateless traffic.

+--------+      +-------+      +--------+
|        |      |       |      |        |
|  Trex  | ---> |  VNF  | ---> |  Trex  |
|        |      |       |      |        |
+--------+      +-------+      +--------+

Supported testcases scenarios:

• Correlated UDP traffic using TREX traffic generator and replay VNF.

  • using different IMIX configuration like pure voice, pure video traffic etc
  • using different number IP flows like 1 flow, 1K, 16K, 64K, 256K, 1M flows
  • Using different number of rules configured like 1 rule, 1K, 10K rules

For UDP correlated traffic following Key Performance Indicators are collected for every combination of test case parameters:

• RFC2544 throughput for various loss rate defined (1% is a default)

VNFs supported for chracterization:¶

1. CGNAPT - Carrier Grade Network Address and port Translation
2. vFW - Virtual Firewall
3. vACL - Access Control List

4. Prox - Packet pROcessing eXecution engine:

   • VNF can act as Drop, Basic Forwarding (no touch), L2 Forwarding (change MAC), GRE encap/decap, Load balance based on packet fields, Symmetric load balancing
   • QinQ encap/decap IPv4/IPv6, ARP, QoS, Routing, Unmpls, Policing, ACL

5. UDP_Replay

Hardware & Software Ingredients¶

SUT requirements:

Item	Description
Memory	Min 20GB
NICs	2 x 10G
OS	Ubuntu 16.04.3 LTS
kernel	4.4.0-34-generic
DPDK	17.02

Boot and BIOS settings:

Boot settings	default_hugepagesz=1G hugepagesz=1G hugepages=16 hugepagesz=2M hugepages=2048 isolcpus=1-11,22-33 nohz_full=1-11,22-33 rcu_nocbs=1-11,22-33 iommu=on iommu=pt intel_iommu=on Note: nohz_full and rcu_nocbs is to disable Linux kernel interrupts
BIOS	CPU Power and Performance Policy <Performance> CPU C-state Disabled CPU P-state Disabled Enhanced Intel® Speedstep® Tech Disabl Hyper-Threading Technology (If supported) Enabled Virtualization Techology Enabled Intel(R) VT for Direct I/O Enabled Coherency Enabled Turbo Boost Disabled

Config yardstick conf¶

If user did not run ‘yardstick env influxdb’ inside the container, which will generate correct yardstick.conf, then create the config file manually (run inside the container):

cp ./etc/yardstick/yardstick.conf.sample /etc/yardstick/yardstick.conf
vi /etc/yardstick/yardstick.conf

Add trex_path, trex_client_lib and bin_path in ‘nsb’ section.

[DEFAULT]
debug = True
dispatcher = file, influxdb

[dispatcher_influxdb]
timeout = 5
target = http://{YOUR_IP_HERE}:8086
db_name = yardstick
username = root
password = root

[nsb]
trex_path=/opt/nsb_bin/trex/scripts
bin_path=/opt/nsb_bin
trex_client_lib=/opt/nsb_bin/trex_client/stl

NS testing - using yardstick CLI¶

See Yardstick Installation

docker exec -it yardstick /bin/bash
source /etc/yardstick/openstack.creds (only for heat TC if nsb_setup.sh was NOT used)
export EXTERNAL_NETWORK="<openstack public network>" (only for heat TC)
yardstick --debug task start yardstick/samples/vnf_samples/nsut/<vnf>/<test case>

Bare-Metal Config pod.yaml describing Topology¶

+----------+              +----------+
|          |              |          |
|          | (0)----->(0) |          |
|    TG1   |              |    DUT   |
|          |              |          |
|          | (n)<-----(n) |          |
+----------+              +----------+
trafficgen_1                   vnf

+----------+              +----------+            +------------+
|          |              |          |            |            |
|          |              |          |            |            |
|          | (0)----->(0) |          |            |    UDP     |
|    TG1   |              |    DUT   |            |   Replay   |
|          |              |          |            |            |
|          |              |          |(1)<---->(0)|            |
+----------+              +----------+            +------------+
trafficgen_1                   vnf                 trafficgen_2

Bare-Metal Config pod.yaml¶

Before executing Yardstick test cases, make sure that pod.yaml reflects the topology and update all the required fields.:

cp /etc/yardstick/nodes/pod.yaml.nsb.sample /etc/yardstick/nodes/pod.yaml

nodes:
-
    name: trafficgen_1
    role: TrafficGen
    ip: 1.1.1.1
    user: root
    password: r00t
    interfaces:
        xe0:  # logical name from topology.yaml and vnfd.yaml
            vpci:      "0000:07:00.0"
            driver:    i40e # default kernel driver
            dpdk_port_num: 0
            local_ip: "152.16.100.20"
            netmask:   "255.255.255.0"
            local_mac: "00:00:00:00:00:01"
        xe1:  # logical name from topology.yaml and vnfd.yaml
            vpci:      "0000:07:00.1"
            driver:    i40e # default kernel driver
            dpdk_port_num: 1
            local_ip: "152.16.40.20"
            netmask:   "255.255.255.0"
            local_mac: "00:00.00:00:00:02"

-
    name: vnf
    role: vnf
    ip: 1.1.1.2
    user: root
    password: r00t
    host: 1.1.1.2 #BM - host == ip, virtualized env - Host - compute node
    interfaces:
        xe0:  # logical name from topology.yaml and vnfd.yaml
            vpci:      "0000:07:00.0"
            driver:    i40e # default kernel driver
            dpdk_port_num: 0
            local_ip: "152.16.100.19"
            netmask:   "255.255.255.0"
            local_mac: "00:00:00:00:00:03"

        xe1:  # logical name from topology.yaml and vnfd.yaml
            vpci:      "0000:07:00.1"
            driver:    i40e # default kernel driver
            dpdk_port_num: 1
            local_ip: "152.16.40.19"
            netmask:   "255.255.255.0"
            local_mac: "00:00:00:00:00:04"
    routing_table:
    - network: "152.16.100.20"
      netmask: "255.255.255.0"
      gateway: "152.16.100.20"
      if: "xe0"
    - network: "152.16.40.20"
      netmask: "255.255.255.0"
      gateway: "152.16.40.20"
      if: "xe1"
    nd_route_tbl:
    - network: "0064:ff9b:0:0:0:0:9810:6414"
      netmask: "112"
      gateway: "0064:ff9b:0:0:0:0:9810:6414"
      if: "xe0"
    - network: "0064:ff9b:0:0:0:0:9810:2814"
      netmask: "112"
      gateway: "0064:ff9b:0:0:0:0:9810:2814"
      if: "xe1"

SR-IOV¶

                             +--------------------+
                             |                    |
                             |                    |
                             |        DUT         |
                             |       (VNF)        |
                             |                    |
                             +--------------------+
                             | VF NIC |  | VF NIC |
                             +--------+  +--------+
                                   ^          ^
                                   |          |
                                   |          |
+----------+               +-------------------------+
|          |               |       ^          ^      |
|          |               |       |          |      |
|          | (0)<----->(0) | ------           |      |
|    TG1   |               |           SUT    |      |
|          |               |                  |      |
|          | (n)<----->(n) |------------------       |
+----------+               +-------------------------+
trafficgen_1                          host

                             +--------------------+
                             |                    |
                             |                    |
                             |        DUT         |
                             |       (VNF)        |
                             |                    |
                             +--------------------+
                             | VF NIC |  | VF NIC |
                             +--------+  +--------+
                                   ^          ^
                                   |          |
                                   |          |
+----------+               +-------------------------+            +--------------+
|          |               |       ^          ^      |            |              |
|          |               |       |          |      |            |              |
|          | (0)<----->(0) | ------           |      |            |     TG2      |
|    TG1   |               |           SUT    |      |            | (UDP Replay) |
|          |               |                  |      |            |              |
|          | (n)<----->(n) |                  ------ | (n)<-->(n) |              |
+----------+               +-------------------------+            +--------------+
trafficgen_1                          host                       trafficgen_2

cp <yardstick>/etc/yardstick/nodes/standalone/trex_bm.yaml.sample /etc/yardstick/nodes/standalone/pod_trex.yaml
cp <yardstick>/etc/yardstick/nodes/standalone/host_sriov.yaml /etc/yardstick/nodes/standalone/host_sriov.yaml

nodes:
-
    name: trafficgen_1
    role: TrafficGen
    ip: 1.1.1.1
    user: root
    password: r00t
    key_filename: /root/.ssh/id_rsa
    interfaces:
        xe0:  # logical name from topology.yaml and vnfd.yaml
            vpci:      "0000:07:00.0"
            driver:    i40e # default kernel driver
            dpdk_port_num: 0
            local_ip: "152.16.100.20"
            netmask:   "255.255.255.0"
            local_mac: "00:00:00:00:00:01"
        xe1:  # logical name from topology.yaml and vnfd.yaml
            vpci:      "0000:07:00.1"
            driver:    i40e # default kernel driver
            dpdk_port_num: 1
            local_ip: "152.16.40.20"
            netmask:   "255.255.255.0"
            local_mac: "00:00.00:00:00:02"

nodes:
-
   name: sriov
   role: Sriov
   ip: 192.168.100.101
   user: ""
   password: ""

contexts:
 - name: yardstick
   type: Node
   file: /etc/yardstick/nodes/standalone/pod_trex.yaml
 - type: StandaloneSriov
   file: /etc/yardstick/nodes/standalone/host_sriov.yaml
   name: yardstick
   vm_deploy: True
   flavor:
     images: "/var/lib/libvirt/images/ubuntu.qcow2"
     ram: 4096
     extra_specs:
       hw:cpu_sockets: 1
       hw:cpu_cores: 6
       hw:cpu_threads: 2
     user: "" # update VM username
     password: "" # update password
   servers:
     vnf:
       network_ports:
         mgmt:
           cidr: '1.1.1.61/24'  # Update VM IP address, if static, <ip>/<mask> or if dynamic, <start of ip>/<mask>
         xe0:
           - uplink_0
         xe1:
           - downlink_0
   networks:
     uplink_0:
       phy_port: "0000:05:00.0"
       vpci: "0000:00:07.0"
       cidr: '152.16.100.10/24'
       gateway_ip: '152.16.100.20'
     downlink_0:
       phy_port: "0000:05:00.1"
       vpci: "0000:00:08.0"
       cidr: '152.16.40.10/24'
       gateway_ip: '152.16.100.20'

OVS-DPDK¶

OVS-DPDK Pre-requisites¶

On Host, where VM is created:

1. Create and configure a bridge named br-int for VM to connect to external network. Currently this can be done using VXLAN tunnel.

   Execute the following on host, where VM is created:

ip link add type vxlan remote <Jumphost IP> local <DUT IP> id <ID: 10> dstport 4789
brctl addbr br-int
brctl addif br-int vxlan0
ip link set dev vxlan0 up
ip addr add <IP#1, like: 172.20.2.1/24> dev br-int
ip link set dev br-int up

Note

May be needed to add extra rules to iptable to forward traffic.

iptables -A FORWARD -i br-int -s <network ip address>/<netmask> -j ACCEPT
iptables -A FORWARD -o br-int -d <network ip address>/<netmask> -j ACCEPT

Execute the following on a jump host:

ip link add type vxlan remote <DUT IP> local <Jumphost IP> id <ID: 10> dstport 4789
ip addr add <IP#2, like: 172.20.2.2/24> dev vxlan0
ip link set dev vxlan0 up

Note

Host and jump host are different baremetal servers.

2. Modify test case management CIDR. IP addresses IP#1, IP#2 and CIDR must be in the same network.

servers:
  vnf:
    network_ports:
      mgmt:
        cidr: '1.1.1.7/24'

3. Build guest image for VNF to run. Most of the sample test cases in Yardstick are using a guest image called yardstick-nsb-image which deviates from an Ubuntu Cloud Server image Yardstick has a tool for building this custom image with SampleVNF. It is necessary to have sudo rights to use this tool.

   Also you may need to install several additional packages to use this tool, by following the commands below:

   sudo apt-get update && sudo apt-get install -y qemu-utils kpartx
   

   This image can be built using the following command in the directory where Yardstick is installed:

   export YARD_IMG_ARCH='amd64'
   sudo echo "Defaults env_keep += \'YARD_IMG_ARCH\'" >> /etc/sudoers
   sudo tools/yardstick-img-dpdk-modify tools/ubuntu-server-cloudimg-samplevnf-modify.sh
   

   for more details refer to chapter Yardstick Installation

   Note

   VM should be build with static IP and should be accessible from yardstick host.

3. OVS & DPDK version.

   • OVS 2.7 and DPDK 16.11.1 above version is supported

4. Setup OVS/DPDK on host.

   Please refer to below link on how to setup OVS-DPDK

                             +--------------------+
                             |                    |
                             |                    |
                             |        DUT         |
                             |       (VNF)        |
                             |                    |
                             +--------------------+
                             | virtio |  | virtio |
                             +--------+  +--------+
                                  ^          ^
                                  |          |
                                  |          |
                             +--------+  +--------+
                             | vHOST0 |  | vHOST1 |
+----------+               +-------------------------+
|          |               |       ^          ^      |
|          |               |       |          |      |
|          | (0)<----->(0) | ------           |      |
|    TG1   |               |          SUT     |      |
|          |               |       (ovs-dpdk) |      |
|          | (n)<----->(n) |------------------       |
+----------+               +-------------------------+
trafficgen_1                          host

                             +--------------------+
                             |                    |
                             |                    |
                             |        DUT         |
                             |       (VNF)        |
                             |                    |
                             +--------------------+
                             | virtio |  | virtio |
                             +--------+  +--------+
                                  ^          ^
                                  |          |
                                  |          |
                             +--------+  +--------+
                             | vHOST0 |  | vHOST1 |
+----------+               +-------------------------+          +------------+
|          |               |       ^          ^      |          |            |
|          |               |       |          |      |          |            |
|          | (0)<----->(0) | ------           |      |          |    TG2     |
|    TG1   |               |          SUT     |      |          |(UDP Replay)|
|          |               |      (ovs-dpdk)  |      |          |            |
|          | (n)<----->(n) |                  ------ |(n)<-->(n)|            |
+----------+               +-------------------------+          +------------+
trafficgen_1                          host                       trafficgen_2

cp <yardstick>/etc/yardstick/nodes/standalone/trex_bm.yaml.sample /etc/yardstick/nodes/standalone/pod_trex.yaml
cp <yardstick>/etc/yardstick/nodes/standalone/host_ovs.yaml /etc/yardstick/nodes/standalone/host_ovs.yaml

nodes:
-
  name: trafficgen_1
  role: TrafficGen
  ip: 1.1.1.1
  user: root
  password: r00t
  interfaces:
      xe0:  # logical name from topology.yaml and vnfd.yaml
          vpci:      "0000:07:00.0"
          driver:    i40e # default kernel driver
          dpdk_port_num: 0
          local_ip: "152.16.100.20"
          netmask:   "255.255.255.0"
          local_mac: "00:00:00:00:00:01"
      xe1:  # logical name from topology.yaml and vnfd.yaml
          vpci:      "0000:07:00.1"
          driver:    i40e # default kernel driver
          dpdk_port_num: 1
          local_ip: "152.16.40.20"
          netmask:   "255.255.255.0"
          local_mac: "00:00.00:00:00:02"

nodes:
-
   name: ovs_dpdk
   role: OvsDpdk
   ip: 192.168.100.101
   user: ""
   password: ""

contexts:
 - name: yardstick
   type: Node
   file: /etc/yardstick/nodes/standalone/pod_trex.yaml
 - type: StandaloneOvsDpdk
   name: yardstick
   file: /etc/yardstick/nodes/standalone/pod_ovs.yaml
   vm_deploy: True
   ovs_properties:
     version:
       ovs: 2.7.0
       dpdk: 16.11.1
     pmd_threads: 2
     ram:
       socket_0: 2048
       socket_1: 2048
     queues: 4
     vpath: "/usr/local"

   flavor:
     images: "/var/lib/libvirt/images/ubuntu.qcow2"
     ram: 4096
     extra_specs:
       hw:cpu_sockets: 1
       hw:cpu_cores: 6
       hw:cpu_threads: 2
     user: "" # update VM username
     password: "" # update password
   servers:
     vnf:
       network_ports:
         mgmt:
           cidr: '1.1.1.61/24'  # Update VM IP address, if static, <ip>/<mask> or if dynamic, <start of ip>/<mask>
         xe0:
           - uplink_0
         xe1:
           - downlink_0
   networks:
     uplink_0:
       phy_port: "0000:05:00.0"
       vpci: "0000:00:07.0"
       cidr: '152.16.100.10/24'
       gateway_ip: '152.16.100.20'
     downlink_0:
       phy_port: "0000:05:00.1"
       vpci: "0000:00:08.0"
       cidr: '152.16.40.10/24'
       gateway_ip: '152.16.100.20'

Single node OpenStack setup with external TG¶

                               +----------------------------+
                               |OpenStack(DevStack)         |
                               |                            |
                               |   +--------------------+   |
                               |   |sample-VNF VM       |   |
                               |   |                    |   |
                               |   |        DUT         |   |
                               |   |       (VNF)        |   |
                               |   |                    |   |
                               |   +--------+  +--------+   |
                               |   | VF NIC |  | VF NIC |   |
                               |   +-----+--+--+----+---+   |
                               |         ^          ^       |
                               |         |          |       |
+----------+                   +---------+----------+-------+
|          |                   |        VF0        VF1      |
|          |                   |         ^          ^       |
|          |                   |         |   SUT    |       |
|    TG    | (PF0)<----->(PF0) +---------+          |       |
|          |                   |                    |       |
|          | (PF1)<----->(PF1) +--------------------+       |
|          |                   |                            |
+----------+                   +----------------------------+
trafficgen_1                                 host

...
GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on"
...

...
GRUB_CMDLINE_LINUX_DEFAULT="amd_iommu=on"
...

sudo update-grub
sudo reboot

sudo journalctl -b 0 | grep -e IOMMU -e DMAR

Feb 06 14:50:14 hostname kernel: ACPI: DMAR 0x000000006C406000 0001E0 (v01 INTEL  S2600WF  00000001 INTL 20091013)
Feb 06 14:50:14 hostname kernel: DMAR: IOMMU enabled
Feb 06 14:50:14 hostname kernel: DMAR: Host address width 46
Feb 06 14:50:14 hostname kernel: DMAR: DRHD base: 0x000000d37fc000 flags: 0x0
Feb 06 14:50:14 hostname kernel: DMAR: dmar0: reg_base_addr d37fc000 ver 1:0 cap 8d2078c106f0466 ecap f020de
Feb 06 14:50:14 hostname kernel: DMAR: DRHD base: 0x000000e0ffc000 flags: 0x0
Feb 06 14:50:14 hostname kernel: DMAR: dmar1: reg_base_addr e0ffc000 ver 1:0 cap 8d2078c106f0466 ecap f020de
Feb 06 14:50:14 hostname kernel: DMAR: DRHD base: 0x000000ee7fc000 flags: 0x0

export http_proxy=http://proxy.company.com:port
export https_proxy=http://proxy.company.com:port
export ftp_proxy=http://proxy.company.com:port
export no_proxy=localhost,127.0.0.1,company.com,<IP-OF-HOST1>,<IP-OF-HOST2>,...
export NO_PROXY=localhost,127.0.0.1,company.com,<IP-OF-HOST1>,<IP-OF-HOST2>,...

sudo -EH apt-get update
sudo -EH apt-get upgrade
sudo -EH apt-get dist-upgrade

sudo -EH apt-get install python
sudo -EH apt-get install python-dev
sudo -EH apt-get install python-pip

sudo ip link set dev enp24s0f0 up
sudo ip link set dev enp24s0f1 up
sudo ip link set dev enp24s0f3 up

# Create VFs on PF
echo 2 | sudo tee /sys/class/net/enp24s0f0/device/sriov_numvfs
echo 2 | sudo tee /sys/class/net/enp24s0f1/device/sriov_numvfs

[[local|localrc]]
HOST_IP=<HOST_IP_ADDRESS>
ADMIN_PASSWORD=password
MYSQL_PASSWORD=$ADMIN_PASSWORD
DATABASE_PASSWORD=$ADMIN_PASSWORD
RABBIT_PASSWORD=$ADMIN_PASSWORD
SERVICE_PASSWORD=$ADMIN_PASSWORD
HORIZON_PASSWORD=$ADMIN_PASSWORD

# Internet access.
RECLONE=False
PIP_UPGRADE=True
IP_VERSION=4

# Services
disable_service n-net
ENABLED_SERVICES+=,q-svc,q-dhcp,q-meta,q-agt,q-sriov-agt

# Heat
enable_plugin heat https://git.openstack.org/openstack/heat stable/pike

# Neutron
enable_plugin neutron https://git.openstack.org/openstack/neutron.git stable/pike

# Neutron Options
FLOATING_RANGE=<RANGE_IN_THE_PUBLIC_INTERFACE_NETWORK>
Q_FLOATING_ALLOCATION_POOL=start=<START_IP_ADDRESS>,end=<END_IP_ADDRESS>
PUBLIC_NETWORK_GATEWAY=<PUBLIC_NETWORK_GATEWAY>
PUBLIC_INTERFACE=<PUBLIC INTERFACE>

# ML2 Configuration
Q_PLUGIN=ml2
Q_ML2_PLUGIN_MECHANISM_DRIVERS=openvswitch,sriovnicswitch
Q_ML2_PLUGIN_TYPE_DRIVERS=vlan,flat,local,vxlan,gre,geneve

# Open vSwitch provider networking configuration
Q_USE_PROVIDERNET_FOR_PUBLIC=True
OVS_PHYSICAL_BRIDGE=br-ex
OVS_BRIDGE_MAPPINGS=public:br-ex
PHYSICAL_DEVICE_MAPPINGS=physnet1:<PF0_IFNAME>,physnet2:<PF1_IFNAME>
PHYSICAL_NETWORK=physnet1,physnet2


[[post-config|$NOVA_CONF]]
[DEFAULT]
scheduler_default_filters=RamFilter,ComputeFilter,AvailabilityZoneFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,PciPassthroughFilter
# Whitelist PCI devices
pci_passthrough_whitelist = {\\"devname\\": \\"<PF0_IFNAME>\\", \\"physical_network\\": \\"physnet1\\" }
pci_passthrough_whitelist = {\\"devname\\": \\"<PF1_IFNAME>\\", \\"physical_network\\": \\"physnet2\\" }

[filter_scheduler]
enabled_filters = RetryFilter,AvailabilityZoneFilter,RamFilter,DiskFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter,SameHostFilter

[libvirt]
cpu_mode = host-model


# ML2 plugin bits for SR-IOV enablement of Intel Corporation XL710/X710 Virtual Function
[[post-config|/$Q_PLUGIN_CONF_FILE]]
[ml2_sriov]
agent_required = True
supported_pci_vendor_devs = <VF_DEV_ID:VF_VEN_ID>

nodes:
-
    name: trafficgen_1
    role: tg__0
    ip: <TG-HOST-IP>
    user: <TG-USER>
    password: <TG-PASS>
    interfaces:
        xe0:  # logical name from topology.yaml and vnfd.yaml
            vpci:      "0000:18:00.0"
            driver:    i40e # default kernel driver
            dpdk_port_num: 0
            local_ip: "10.1.1.150"
            netmask:   "255.255.255.0"
            local_mac: "00:00:00:00:00:01"
        xe1:  # logical name from topology.yaml and vnfd.yaml
            vpci:      "0000:18:00.1"
            driver:    i40e # default kernel driver
            dpdk_port_num: 1
            local_ip: "10.1.1.151"
            netmask:   "255.255.255.0"
            local_mac: "00:00:00:00:00:02"

Multi node OpenStack TG and VNF setup (two nodes)¶

+----------------------------+                   +----------------------------+
|OpenStack(DevStack)         |                   |OpenStack(DevStack)         |
|                            |                   |                            |
|   +--------------------+   |                   |   +--------------------+   |
|   |sample-VNF VM       |   |                   |   |sample-VNF VM       |   |
|   |                    |   |                   |   |                    |   |
|   |         TG         |   |                   |   |        DUT         |   |
|   |    trafficgen_1    |   |                   |   |       (VNF)        |   |
|   |                    |   |                   |   |                    |   |
|   +--------+  +--------+   |                   |   +--------+  +--------+   |
|   | VF NIC |  | VF NIC |   |                   |   | VF NIC |  | VF NIC |   |
|   +----+---+--+----+---+   |                   |   +-----+--+--+----+---+   |
|        ^           ^       |                   |         ^          ^       |
|        |           |       |                   |         |          |       |
+--------+-----------+-------+                   +---------+----------+-------+
|       VF0         VF1      |                   |        VF0        VF1      |
|        ^           ^       |                   |         ^          ^       |
|        |    SUT2   |       |                   |         |   SUT1   |       |
|        |           +-------+ (PF0)<----->(PF0) +---------+          |       |
|        |                   |                   |                    |       |
|        +-------------------+ (PF1)<----->(PF1) +--------------------+       |
|                            |                   |                            |
+----------------------------+                   +----------------------------+
         host2 (compute)                               host1 (controller)

[[local|localrc]]
HOST_IP=<HOST_IP_ADDRESS>
ADMIN_PASSWORD=password
MYSQL_PASSWORD=$ADMIN_PASSWORD
DATABASE_PASSWORD=$ADMIN_PASSWORD
RABBIT_PASSWORD=$ADMIN_PASSWORD
SERVICE_PASSWORD=$ADMIN_PASSWORD
HORIZON_PASSWORD=$ADMIN_PASSWORD
# Controller node
SERVICE_HOST=$HOST_IP
MYSQL_HOST=$SERVICE_HOST
RABBIT_HOST=$SERVICE_HOST
GLANCE_HOSTPORT=$SERVICE_HOST:9292

# Internet access.
RECLONE=False
PIP_UPGRADE=True
IP_VERSION=4

# Services
disable_service n-net
ENABLED_SERVICES+=,q-svc,q-dhcp,q-meta,q-agt,q-sriov-agt

# Heat
enable_plugin heat https://git.openstack.org/openstack/heat stable/pike

# Neutron
enable_plugin neutron https://git.openstack.org/openstack/neutron.git stable/pike

# Neutron Options
FLOATING_RANGE=<RANGE_IN_THE_PUBLIC_INTERFACE_NETWORK>
Q_FLOATING_ALLOCATION_POOL=start=<START_IP_ADDRESS>,end=<END_IP_ADDRESS>
PUBLIC_NETWORK_GATEWAY=<PUBLIC_NETWORK_GATEWAY>
PUBLIC_INTERFACE=<PUBLIC INTERFACE>

# ML2 Configuration
Q_PLUGIN=ml2
Q_ML2_PLUGIN_MECHANISM_DRIVERS=openvswitch,sriovnicswitch
Q_ML2_PLUGIN_TYPE_DRIVERS=vlan,flat,local,vxlan,gre,geneve

# Open vSwitch provider networking configuration
Q_USE_PROVIDERNET_FOR_PUBLIC=True
OVS_PHYSICAL_BRIDGE=br-ex
OVS_BRIDGE_MAPPINGS=public:br-ex
PHYSICAL_DEVICE_MAPPINGS=physnet1:<PF0_IFNAME>,physnet2:<PF1_IFNAME>
PHYSICAL_NETWORK=physnet1,physnet2


[[post-config|$NOVA_CONF]]
[DEFAULT]
scheduler_default_filters=RamFilter,ComputeFilter,AvailabilityZoneFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,PciPassthroughFilter
# Whitelist PCI devices
pci_passthrough_whitelist = {\\"devname\\": \\"<PF0_IFNAME>\\", \\"physical_network\\": \\"physnet1\\" }
pci_passthrough_whitelist = {\\"devname\\": \\"<PF1_IFNAME>\\", \\"physical_network\\": \\"physnet2\\" }

[libvirt]
cpu_mode = host-model


# ML2 plugin bits for SR-IOV enablement of Intel Corporation XL710/X710 Virtual Function
[[post-config|/$Q_PLUGIN_CONF_FILE]]
[ml2_sriov]
agent_required = True
supported_pci_vendor_devs = <VF_DEV_ID:VF_VEN_ID>

[[local|localrc]]
HOST_IP=<HOST_IP_ADDRESS>
MYSQL_PASSWORD=password
DATABASE_PASSWORD=password
RABBIT_PASSWORD=password
ADMIN_PASSWORD=password
SERVICE_PASSWORD=password
HORIZON_PASSWORD=password
# Controller node
SERVICE_HOST=<CONTROLLER_IP_ADDRESS>
MYSQL_HOST=$SERVICE_HOST
RABBIT_HOST=$SERVICE_HOST
GLANCE_HOSTPORT=$SERVICE_HOST:9292

# Internet access.
RECLONE=False
PIP_UPGRADE=True
IP_VERSION=4

# Neutron
enable_plugin neutron https://git.openstack.org/openstack/neutron.git stable/pike

# Services
ENABLED_SERVICES=n-cpu,rabbit,q-agt,placement-api,q-sriov-agt

# Neutron Options
PUBLIC_INTERFACE=<PUBLIC INTERFACE>

# ML2 Configuration
Q_PLUGIN=ml2
Q_ML2_PLUGIN_MECHANISM_DRIVERS=openvswitch,sriovnicswitch
Q_ML2_PLUGIN_TYPE_DRIVERS=vlan,flat,local,vxlan,gre,geneve

# Open vSwitch provider networking configuration
PHYSICAL_DEVICE_MAPPINGS=physnet1:<PF0_IFNAME>,physnet2:<PF1_IFNAME>


[[post-config|$NOVA_CONF]]
[DEFAULT]
scheduler_default_filters=RamFilter,ComputeFilter,AvailabilityZoneFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,PciPassthroughFilter
# Whitelist PCI devices
pci_passthrough_whitelist = {\\"devname\\": \\"<PF0_IFNAME>\\", \\"physical_network\\": \\"physnet1\\" }
pci_passthrough_whitelist = {\\"devname\\": \\"<PF1_IFNAME>\\", \\"physical_network\\": \\"physnet2\\" }

[libvirt]
cpu_mode = host-model


# ML2 plugin bits for SR-IOV enablement of Intel Corporation XL710/X710 Virtual Function
[[post-config|/$Q_PLUGIN_CONF_FILE]]
[ml2_sriov]
agent_required = True
supported_pci_vendor_devs = <VF_DEV_ID:VF_VEN_ID>

yardstick -d task start --task-args='{"provider": "sriov"}' \
samples/vnf_samples/nsut/vfw/tc_heat_rfc2544_ipv4_1rule_1flow_64B_trex.yaml

IxLoad¶

1. Software needed: IxLoadAPI <IxLoadTclApi verson>Linux64.bin.tgz and <IxOS version>Linux64.bin.tar.gz (Download from ixia support site) Install - <IxLoadTclApi verson>Linux64.bin.tgz and <IxOS version>Linux64.bin.tar.gz If the installation was not done inside the container, after installing the IXIA client, check /opt/ixia/ixload/<ver>/bin/ixloadpython and make sure you can run this cmd inside the yardstick container. Usually user is required to copy or link /opt/ixia/python/<ver>/bin/ixiapython to /usr/bin/ixiapython<ver> inside the container.
2. Update pod_ixia.yaml file with ixia details.

cp <repo>/etc/yardstick/nodes/pod.yaml.nsb.sample.ixia etc/yardstick/nodes/pod_ixia.yaml

Config pod_ixia.yaml

nodes:
-
    name: trafficgen_1
    role: IxNet
    ip: 1.2.1.1 #ixia machine ip
    user: user
    password: r00t
    key_filename: /root/.ssh/id_rsa
    tg_config:
        ixchassis: "1.2.1.7" #ixia chassis ip
        tcl_port: "8009" # tcl server port
        lib_path: "/opt/ixia/ixos-api/8.01.0.2/lib/ixTcl1.0"
        root_dir: "/opt/ixia/ixos-api/8.01.0.2/"
        py_bin_path: "/opt/ixia/ixload/8.01.106.3/bin/"
        dut_result_dir: "/mnt/ixia"
        version: 8.1
    interfaces:
        xe0:  # logical name from topology.yaml and vnfd.yaml
            vpci: "2:5" # Card:port
            driver:    "none"
            dpdk_port_num: 0
            local_ip: "152.16.100.20"
            netmask:   "255.255.0.0"
            local_mac: "00:98:10:64:14:00"
        xe1:  # logical name from topology.yaml and vnfd.yaml
            vpci: "2:6" # [(Card, port)]
            driver:    "none"
            dpdk_port_num: 1
            local_ip: "152.40.40.20"
            netmask:   "255.255.0.0"
            local_mac: "00:98:28:28:14:00"

for sriov/ovs_dpdk pod files, please refer to above Standalone Virtualization for ovs-dpdk/sriov configuration

3. Start IxOS TCL Server (Install ‘Ixia IxExplorer IxOS <version>’) You will also need to configure the IxLoad machine to start the IXIA IxosTclServer. This can be started like so:
   • Connect to the IxLoad machine using RDP
   • Go to: Start->Programs->Ixia->IxOS->IxOS 8.01-GA-Patch1->Ixia Tcl Server IxOS 8.01-GA-Patch1 or "C:\Program Files (x86)\Ixia\IxOS\8.01-GA-Patch1\ixTclServer.exe"
4. Create a folder Results in c:and share the folder on the network.
5. Execute testcase in samplevnf folder e.g. <repo>/samples/vnf_samples/nsut/vfw/tc_baremetal_http_ixload_1b_Requests-65000_Concurrency.yaml

IxNetwork¶

IxNetwork testcases use IxNetwork API Python Bindings module, which is installed as part of the requirements of the project.

1. Update pod_ixia.yaml file with ixia details.

cp <repo>/etc/yardstick/nodes/pod.yaml.nsb.sample.ixia etc/yardstick/nodes/pod_ixia.yaml

Config pod_ixia.yaml

nodes:
-
    name: trafficgen_1
    role: IxNet
    ip: 1.2.1.1 #ixia machine ip
    user: user
    password: r00t
    key_filename: /root/.ssh/id_rsa
    tg_config:
        ixchassis: "1.2.1.7" #ixia chassis ip
        tcl_port: "8009" # tcl server port
        lib_path: "/opt/ixia/ixos-api/8.01.0.2/lib/ixTcl1.0"
        root_dir: "/opt/ixia/ixos-api/8.01.0.2/"
        py_bin_path: "/opt/ixia/ixload/8.01.106.3/bin/"
        dut_result_dir: "/mnt/ixia"
        version: 8.1
    interfaces:
        xe0:  # logical name from topology.yaml and vnfd.yaml
            vpci: "2:5" # Card:port
            driver:    "none"
            dpdk_port_num: 0
            local_ip: "152.16.100.20"
            netmask:   "255.255.0.0"
            local_mac: "00:98:10:64:14:00"
        xe1:  # logical name from topology.yaml and vnfd.yaml
            vpci: "2:6" # [(Card, port)]
            driver:    "none"
            dpdk_port_num: 1
            local_ip: "152.40.40.20"
            netmask:   "255.255.0.0"
            local_mac: "00:98:28:28:14:00"

for sriov/ovs_dpdk pod files, please refer to above Standalone Virtualization for ovs-dpdk/sriov configuration

2. Start IxNetwork TCL Server You will also need to configure the IxNetwork machine to start the IXIA IxNetworkTclServer. This can be started like so:

   • Connect to the IxNetwork machine using RDP
   • Go to: Start->Programs->Ixia->IxNetwork->IxNetwork 7.21.893.14 GA->IxNetworkTclServer (or IxNetworkApiServer)

3. Execute testcase in samplevnf folder e.g. <repo>/samples/vnf_samples/nsut/vfw/tc_baremetal_rfc2544_ipv4_1rule_1flow_64B_ixia.yaml

Provider networks¶

The VNFs require a clear L2 connect to the external network in order to generate realistic traffic from multiple address ranges and ports.

In order to prevent Neutron from filtering traffic we have to disable Neutron Port Security. We also disable DHCP on the data ports because we are binding the ports to DPDK and do not need DHCP addresses. We also disable gateways because multiple default gateways can prevent SSH access to the VNF from the floating IP. We only want a gateway on the mgmt network

uplink_0:
  cidr: '10.1.0.0/24'
  gateway_ip: 'null'
  port_security_enabled: False
  enable_dhcp: 'false'

Heat Topologies¶

By default Heat will attach every node to every Neutron network that is created. For scale-out tests we do not want to attach every node to every network.

For each node you can specify which ports are on which network using the network_ports dictionary.

In this example we have TRex xe0 <-> xe0 VNF xe1 <-> xe0 UDP_Replay

vnf_0:
  floating_ip: true
  placement: "pgrp1"
  network_ports:
    mgmt:
      - mgmt
    uplink_0:
      - xe0
    downlink_0:
      - xe1
tg_0:
  floating_ip: true
  placement: "pgrp1"
  network_ports:
    mgmt:
      - mgmt
    uplink_0:
      - xe0
    # Trex always needs two ports
    uplink_1:
      - xe1
tg_1:
  floating_ip: true
  placement: "pgrp1"
  network_ports:
    mgmt:
     - mgmt
    downlink_0:
     - xe0

Availability zone¶

The configuration of the availability zone is requred in cases where location of exact compute host/group of compute hosts needs to be specified for SampleVNF or traffic generator in the heat test case. If this is the case, please follow the instructions below.

1. Create a host aggregate in the OpenStack and add the available compute hosts into the aggregate group.

   Note

   Change the <AZ_NAME> (availability zone name), <AGG_NAME> (host aggregate name) and <HOST> (host name of one of the compute) in the commands below.

   # create host aggregate
   openstack aggregate create --zone <AZ_NAME> --property availability_zone=<AZ_NAME> <AGG_NAME>
   # show available hosts
   openstack compute service list --service nova-compute
   # add selected host into the host aggregate
   openstack aggregate add host <AGG_NAME> <HOST>
   

2. To specify the OpenStack location (the exact compute host or group of the hosts) of SampleVNF or traffic generator in the heat test case, the availability_zone server configuration option should be used. For example:

   Note

   The <AZ_NAME> (availability zone name) should be changed according to the name used during the host aggregate creation steps above.

   context:
     name: yardstick
     image: yardstick-samplevnfs
     ...
     servers:
       vnf__0:
         ...
         availability_zone: <AZ_NAME>
         ...
       tg__0:
         ...
         availability_zone: <AZ_NAME>
         ...
     networks:
       ...
   

There are two example of SampleVNF scale out test case which use the availability zone feature to specify the exact location of scaled VNFs and traffic generators.

Those are:

<repo>/samples/vnf_samples/nsut/prox/tc_prox_heat_context_l2fwd_multiflow-2-scale-out.yaml
<repo>/samples/vnf_samples/nsut/vfw/tc_heat_rfc2544_ipv4_1rule_1flow_64B_trex_scale_out.yaml

1. Before running the scale-out test case, make sure the host aggregates are configured in the OpenStack environment. To check this, run the following command:

   # show configured host aggregates (example)
   openstack aggregate list
   +----+------+-------------------+
   | ID | Name | Availability Zone |
   +----+------+-------------------+
   |  4 | agg0 | AZ_NAME_0         |
   |  5 | agg1 | AZ_NAME_1         |
   +----+------+-------------------+
   

2. If no host aggregates are configured, please use steps above to configure them.

3. Run the SampleVNF PROX scale-out test case, specifying the availability zone of each VNF and traffic generator as a task arguments.

   Note

   The az_0 and az_1 should be changed according to the host aggregates created in the OpenStack.

   yardstick -d task start\
   <repo>/samples/vnf_samples/nsut/prox/tc_prox_heat_context_l2fwd_multiflow-2-scale-out.yaml\
     --task-args='{
       "num_vnfs": 4, "availability_zone": {
         "vnf_0": "az_0", "tg_0": "az_1",
         "vnf_1": "az_0", "tg_1": "az_1",
         "vnf_2": "az_0", "tg_2": "az_1",
         "vnf_3": "az_0", "tg_3": "az_1"
       }
     }'
   

   num_vnfs specifies how many VNFs are going to be deployed in the heat contexts. vnf_X and tg_X arguments configure the availability zone where the VNF and traffic generator is going to be deployed.

Heat¶

For VNF scale-up tests we increase the number for VNF worker threads. In the case of VNFs we also need to increase the number of VCPUs and memory allocated to the VNF.

An example scale-up Heat testcase is:

# Copyright (c) 2016-2018 Intel Corporation
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#      http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
{% set mem = mem or 20480 %}
{% set vcpus = vcpus or 10 %}
{% set vports = vports or 2 %}
---
schema: yardstick:task:0.1
scenarios:
- type: NSPerf
  traffic_profile: ../../traffic_profiles/ipv4_throughput-scale-up.yaml
  extra_args:
    vports: {{ vports }}
  topology: vfw-tg-topology-scale-up.yaml
  nodes:
    tg__0: tg_0.yardstick
    vnf__0: vnf_0.yardstick
  options:
    framesize:
      uplink: {64B: 100}
      downlink: {64B: 100}
    flow:
      src_ip: [
{% for vport in range(0,vports,2|int) %}
       {'tg__0': 'xe{{vport}}'},
{% endfor %}  ]
      dst_ip: [
{% for vport in range(1,vports,2|int) %}
      {'tg__0': 'xe{{vport}}'},
{% endfor %}  ]
      count: 1
    traffic_type: 4
    rfc2544:
      allowed_drop_rate: 0.0001 - 0.0001
    vnf__0:
      rules: acl_1rule.yaml
      vnf_config: {lb_config: 'SW', file: vfw_vnf_pipeline_cores_{{vcpus}}_ports_{{vports}}_lb_1_sw.conf }
  runner:
    type: Iteration
    iterations: 10
    interval: 35
context:
  # put node context first, so we don't HEAT deploy if node has errors
  name: yardstick
  image: yardstick-samplevnfs
  flavor:
    vcpus: {{ vcpus }}
    ram: {{ mem }}
    disk: 6
    extra_specs:
      hw:cpu_sockets: 1
      hw:cpu_cores: {{ vcpus }}
      hw:cpu_threads: 1
  user: ubuntu
  placement_groups:
    pgrp1:
      policy: "availability"
  servers:
    tg_0:
      floating_ip: true
      placement: "pgrp1"
    vnf_0:
      floating_ip: true
      placement: "pgrp1"
  networks:
    mgmt:
      cidr: '10.0.1.0/24'
{% for vport in range(1,vports,2|int) %}
    uplink_{{loop.index0}}:
      cidr: '10.1.{{vport}}.0/24'
      gateway_ip: 'null'
      port_security_enabled: False
      enable_dhcp: 'false'
    downlink_{{loop.index0}}:
      cidr: '10.1.{{vport+1}}.0/24'
      gateway_ip: 'null'
      port_security_enabled: False
      enable_dhcp: 'false'
{% endfor %}

This testcase template requires specifying the number of VCPUs, Memory and Ports. We set the VCPUs and memory using the --task-args options

yardstick task start --task-args='{"mem": 10480, "vcpus": 4, "vports": 2}' \
samples/vnf_samples/nsut/vfw/tc_heat_rfc2544_ipv4_1rule_1flow_64B_trex_scale-up.yaml

In order to support ports scale-up, traffic and topology templates need to be used in testcase.

A example topology template is:

# Copyright (c) 2016-2018 Intel Corporation
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#      http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
---
{% set vports = get(extra_args, 'vports', '2') %}
nsd:nsd-catalog:
    nsd:
    -   id: 3tg-topology
        name: 3tg-topology
        short-name: 3tg-topology
        description: 3tg-topology
        constituent-vnfd:
        -   member-vnf-index: '1'
            vnfd-id-ref: tg__0
            VNF model: ../../vnf_descriptors/tg_rfc2544_tpl.yaml      #VNF type
        -   member-vnf-index: '2'
            vnfd-id-ref: vnf__0
            VNF model: ../../vnf_descriptors/vfw_vnf.yaml      #VNF type

        vld:
{% for vport in range(0,vports,2|int) %}
        -   id: uplink_{{loop.index0}}
            name: tg__0 to vnf__0 link {{vport + 1}}
            type: ELAN
            vnfd-connection-point-ref:
            -   member-vnf-index-ref: '1'
                vnfd-connection-point-ref: xe{{vport}}
                vnfd-id-ref: tg__0
            -   member-vnf-index-ref: '2'
                vnfd-connection-point-ref: xe{{vport}}
                vnfd-id-ref: vnf__0
        -   id: downlink_{{loop.index0}}
            name: vnf__0 to tg__0 link {{vport + 2}}
            type: ELAN
            vnfd-connection-point-ref:
            -   member-vnf-index-ref: '2'
                vnfd-connection-point-ref: xe{{vport+1}}
                vnfd-id-ref: vnf__0
            -   member-vnf-index-ref: '1'
                vnfd-connection-point-ref: xe{{vport+1}}
                vnfd-id-ref: tg__0
{% endfor %}

This template has vports as an argument. To pass this argument it needs to be configured in extra_args scenario definition. Please note that more argument can be defined in that section. All of them will be passed to topology and traffic profile templates

For example:

schema: yardstick:task:0.1
scenarios:
- type: NSPerf
  traffic_profile: ../../traffic_profiles/ipv4_throughput-scale-up.yaml
  extra_args:
    vports: {{ vports }}
  topology: vfw-tg-topology-scale-up.yaml

A example traffic profile template is:

# Copyright (c) 2016-2018 Intel Corporation
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#      http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# flow definition for ACL tests - 1K flows - ipv4 only
#
# the number of flows defines the widest range of parameters
# for example if srcip_range=1.0.0.1-1.0.0.255 and dst_ip_range=10.0.0.1-10.0.1.255
# and it should define only 16 flows
#
# there is assumption that packets generated will have a random sequences of following addresses pairs
# in the packets
# 1. src=1.x.x.x(x.x.x =random from 1..255) dst=10.x.x.x (random from 1..512)
# 2. src=1.x.x.x(x.x.x =random from 1..255) dst=10.x.x.x (random from 1..512)
# ...
# 512. src=1.x.x.x(x.x.x =random from 1..255) dst=10.x.x.x (random from 1..512)
#
# not all combination should be filled
# Any other field with random range will be added to flow definition
#
# the example.yaml provides all possibilities for traffic generation
#
# the profile defines a public and private side to make limited traffic correlation
# between private and public side same way as it is made by IXIA solution.
#
{% set vports = get(extra_args, 'vports', '2') %}
---
schema: "nsb:traffic_profile:0.1"

# This file is a template, it will be filled with values from tc.yaml before passing to the traffic generator

name: rfc2544
description: Traffic profile to run RFC2544 latency
traffic_profile:
  traffic_type: RFC2544Profile # defines traffic behavior - constant or look for highest possible throughput
  frame_rate: 100  # pc of linerate
  duration: {{ duration }}

{% set count = 0 %}
{% for vport in range(vports|int) %}
uplink_{{vport}}:
  ipv4:
    id: {{count + 1 }}
    outer_l2:
      framesize:
        64B: "{{ get(imix, 'imix.uplink.64B', '0') }}"
        128B: "{{ get(imix, 'imix.uplink.128B', '0') }}"
        256B: "{{ get(imix, 'imix.uplink.256B', '0') }}"
        373b: "{{ get(imix, 'imix.uplink.373B', '0') }}"
        512B: "{{ get(imix, 'imix.uplink.512B', '0') }}"
        570B: "{{ get(imix, 'imix.uplink.570B', '0') }}"
        1400B: "{{ get(imix, 'imix.uplink.1400B', '0') }}"
        1500B: "{{ get(imix, 'imix.uplink.1500B', '0') }}"
        1518B: "{{ get(imix, 'imix.uplink.1518B', '0') }}"
    outer_l3v4:
      proto: "udp"
      srcip4: "{{ get(flow, 'flow.src_ip_{{vport}}', '1.1.1.1-1.1.255.255') }}"
      dstip4: "{{ get(flow, 'flow.dst_ip_{{vport}}', '90.90.1.1-90.90.255.255') }}"
      count: "{{ get(flow, 'flow.count', '1') }}"
      ttl: 32
      dscp: 0
    outer_l4:
      srcport: "{{ get(flow, 'flow.src_port_{{vport}}', '1234-4321') }}"
      dstport: "{{ get(flow, 'flow.dst_port_{{vport}}', '2001-4001') }}"
      count: "{{ get(flow, 'flow.count', '1') }}"
downlink_{{vport}}:
  ipv4:
    id: {{count + 2}}
    outer_l2:
      framesize:
        64B: "{{ get(imix, 'imix.downlink.64B', '0') }}"
        128B: "{{ get(imix, 'imix.downlink.128B', '0') }}"
        256B: "{{ get(imix, 'imix.downlink.256B', '0') }}"
        373b: "{{ get(imix, 'imix.downlink.373B', '0') }}"
        512B: "{{ get(imix, 'imix.downlink.512B', '0') }}"
        570B: "{{ get(imix, 'imix.downlink.570B', '0') }}"
        1400B: "{{ get(imix, 'imix.downlink.1400B', '0') }}"
        1500B: "{{ get(imix, 'imix.downlink.1500B', '0') }}"
        1518B: "{{ get(imix, 'imix.downlink.1518B', '0') }}"

    outer_l3v4:
      proto: "udp"
      srcip4: "{{ get(flow, 'flow.dst_ip_{{vport}}', '90.90.1.1-90.90.255.255') }}"
      dstip4: "{{ get(flow, 'flow.src_ip_{{vport}}', '1.1.1.1-1.1.255.255') }}"
      count: "{{ get(flow, 'flow.count', '1') }}"
      ttl: 32
      dscp: 0
    outer_l4:
      srcport: "{{ get(flow, 'flow.dst_port_{{vport}}', '1234-4321') }}"
      dstport: "{{ get(flow, 'flow.src_port_{{vport}}', '2001-4001') }}"
      count: "{{ get(flow, 'flow.count', '1') }}"
{% set count = count + 2 %}
{% endfor %}

There is an option to provide predefined config for SampleVNFs. Path to config file may by specified in vnf_config scenario section.

vnf__0:
   rules: acl_1rule.yaml
   vnf_config: {lb_config: 'SW', file: vfw_vnf_pipeline_cores_4_ports_2_lb_1_sw.conf }

Baremetal¶

1. Follow above traffic generator section to setup.
2. Edit num of threads in <repo>/samples/vnf_samples/nsut/vfw/tc_baremetal_rfc2544_ipv4_1rule_1flow_64B_trex_scale_up.yaml e.g, 6 Threads for given VNF

schema: yardstick:task:0.1
scenarios:
{% for worker_thread in [1, 2 ,3 , 4, 5, 6] %}
- type: NSPerf
  traffic_profile: ../../traffic_profiles/ipv4_throughput.yaml
  topology: vfw-tg-topology.yaml
  nodes:
    tg__0: trafficgen_1.yardstick
    vnf__0: vnf.yardstick
  options:
    framesize:
      uplink: {64B: 100}
      downlink: {64B: 100}
    flow:
      src_ip: [{'tg__0': 'xe0'}]
      dst_ip: [{'tg__0': 'xe1'}]
      count: 1
    traffic_type: 4
    rfc2544:
      allowed_drop_rate: 0.0001 - 0.0001
    vnf__0:
      rules: acl_1rule.yaml
      vnf_config: {lb_config: 'HW', lb_count: 1, worker_config: '1C/1T', worker_threads: {{worker_thread}}}
      nfvi_enable: True
  runner:
    type: Iteration
    iterations: 10
    interval: 35
{% endfor %}
context:
  type: Node
  name: yardstick
  nfvi_type: baremetal
  file: /etc/yardstick/nodes/pod.yaml