The Yardstick framework, the Yardstick test cases are open-source software,

licensed under the terms of the Apache License, Version 2.0.

1.1. Yardstick Release Notes

1.1.1. Abstract

This document compiles the release notes for the Iruya release of OPNFV Yardstick.

1.1.2. Version History

Date

Version

Comment

Jan 10, 2020

9.0.0

Yardstick for Iruya release

1.1.3. Important Notes

The software delivered in the OPNFV Yardstick Project, comprising the Yardstick framework, and the Yardstick test cases is a realization of the methodology in ETSI-ISG NFV-TST001.

The Yardstick framework is installer, infrastructure and application independent.

1.1.4. OPNFV Iruya Release

This Iruya release provides Yardstick as a framework for NFVI testing and OPNFV feature testing, automated in the OPNFV CI pipeline, including:

  • Documentation generated with Sphinx

    • User Guide

    • Developer Guide

    • Release notes (this document)

    • Results

  • Automated Yardstick test suite (daily, weekly)

    • Jenkins Jobs for OPNFV community labs

  • Automated Yardstick test results visualization

    • Dashboard using Grafana (user:opnfv/password: opnfv), influxDB is used as backend

  • Yardstick framework source code

  • Yardstick test cases yaml files

  • Yardstick plug-in configuration yaml files, plug-in install/remove scripts

For Iruya release, the Yardstick framework is used for the following testing:

  • OPNFV platform testing - generic test cases to measure the categories:

    • Compute

    • Network

    • Storage

  • OPNFV platform network service benchmarking (NSB)

    • NSB

  • Test cases for the following OPNFV Projects:

    • Container4NFV

    • High Availability

    • IPv6

    • KVM

    • Parser

    • StorPerf

    • VSperf

The Yardstick framework is developed in the OPNFV community, by the Yardstick team.

Note

The test case description template used for the Yardstick test cases is based on the document ETSI-ISG NFV-TST001; the results report template used for the Yardstick results is based on the IEEE Std 829-2008.

1.1.5. Release Data

Project

Yardstick

Repo/tag

yardstick/opnfv-9.0.0

Yardstick Docker image tag

opnfv-9.0.0

Release designation

Iruya 9.0

Release date

Jan 10, 2020

Purpose of the delivery

OPNFV Iruya 9.0.0

1.1.6. Deliverables

1.1.6.1. Documents

  • User Guide: <yardstick:userguide>

  • Developer Guide: <yardstick:devguide>

1.1.6.2. Software Deliverables

1.1.6.2.1. List of Contexts

Context

Description

Heat

Models orchestration using OpenStack Heat

Node

Models Baremetal, Controller, Compute

Standalone

Models VM running on Non-Managed NFVi

Kubernetes

Models VM running on Non-Managed NFVi

1.1.6.2.2. List of Runners

Runner

Description

Arithmetic

Steps every run arithmetically according to specified input value

Duration

Runs for a specified period of time

Iteration

Runs for a specified number of iterations

IterationIPC

Runs a configurable number of times before it returns. Each iteration has a configurable timeout.

Sequence

Selects input value to a scenario from an input file and runs all entries sequentially

Dynamictp

A runner that searches for the max throughput with binary search

Search

A runner that runs a specific time before it returns

1.1.6.2.3. List of Scenarios

Category

Delivered

Availability

Attacker:

  • baremetal, process

HA tools:

  • check host, openstack, process, service

  • kill process

  • start/stop service

Monitor:

  • command, process

Compute

  • cpuload

  • cyclictest

  • lmbench

  • lmbench_cache

  • perf

  • unixbench

  • ramspeed

  • cachestat

  • memeoryload

  • computecapacity

  • SpecCPU2006

Networking

  • iperf3

  • netperf

  • netperf_node

  • ping

  • ping6

  • pktgen

  • sfc

  • sfc with tacker

  • networkcapacity

  • netutilization

  • nstat

  • pktgenDPDK

Parser

Tosca2Heat

Storage

  • fio

  • bonnie++

  • storagecapacity

StorPerf

storperf

NSB

vFW thoughput test case

1.1.6.3. New Test cases

opnfv_yardstick_tc015: Processing speed with impact on energy consumption and CPU load.

The purpose of TC015 is to evaluate the IaaS compute performance with regards to CPU processing speed with its impact on the energy consumption. It measures score of single cpu running and parallel running. Energy consumption and cpu load are monitored while the cpu test is running. The purpose is also to be able to spot the trends. Test results, graphs and similar shall be stored for comparison reasons and product evolution understanding between different OPNFV versions and/or configurations, different server types.

1.1.7. Version Change

1.1.7.1. Module Version Changes

This is the seventh tracked release of Yardstick. It is based on following upstream versions:

  • OpenStack Stein

1.1.7.2. Document Version Changes

This is the seventh tracked version of the Yardstick framework in OPNFV. It includes the following documentation updates:

  • Yardstick User Guide:

  • Yardstick Developer Guide

  • Yardstick Release Notes for Yardstick: this document

1.1.7.3. Feature additions

1.1.8. Scenario Matrix

1.1.9. Test results

Test results are available in:

1.1.9.1. Known Issues/Faults

1.1.9.2. Corrected Faults

1.1.10. Iruya 9.0.0 known restrictions/issues