# ETSI Plugtests
## NFV API Conformance Results
During the Remote NFV&MEC API Plugtest 2021, Totally 136 API Conformance test sessions were executed for [NFV-SOL003] were executed as well as for [NFV-SOL005].
we had 2 contributions on the NFV Specifications and reported 22 issues on the Robot Test Suites as well as 5 bugs on Tacker.
By the end of September, all the issues reported to ETSI and Tacker in Feb. have been fixed. The latest results show that for VNF LifecycleManagement API, although the total number of test session increased,the pass rate increased from 71.62% to 85.06%. Also for the VNF Package Management API, the pass rate increased from 58.18% to 90.91%.

### Results between Feb. and Oct.
#### VNF LifecycleManagement API
| | Pass | Fail | Totals |
| ---- | ---------- | ---------- | ------ |
| Feb. | 53(71.62%) | 21(28.38%) | 74 |
| Oct. | 69(85.19%) | 12(14.81%) | 81 |
| | +13.57% | -13.57% | |

#### VNF Package Management API
| | Pass | Fail | Totals |
| ---- | ---------- | ---------- | ------ |
| Feb. | 32(58.18%) | 23(41.82%) | 55 |
| Oct. | 50(90.91%) | 5(9.09%) | 55 |
| | +32.73% | -32.73% | |

### Test Results in Detail (Feb.)
The following clauses provide tables and figures which summarize the results for the NFV API Conformance test for the different APIs
#### NFV-SOL003
| | Pass | Fail | Totals |
| --------------------------- | ---------- | ---------- | ------ |
| VNF LifecycleManagement API | 53(71.62%) | 21(28.38%) | 74 |
| VNF Package Management API | 29(70.73%) | 12(29.27%) | 41 |
#### NFV-SOL005
| | Pass | Fail | Totals |
| --------------------------- | ---------- | ---------- | ------ |
| VNF Package Management API | 32(58.18%) | 23(41.82%) | 55 |

#### VNF LifecycleManagement API
The table and figures below summarize the results for VNFLifecycleManagement API per test session for [NFV-SOL003].

| Test session | Total | Pass | Fail |
| ---------------------------------- | ----- | ---------- | ---------- |
| Heal VNF Task | 7 | 4 | 3 |
| IndividualVnfInstance | 8 | 4 | 4 |
| IndividualVnfLcmOperationOccurence | 5 | 4 | 1 |
| InstantiateVNFTask | 6 | 4 | 2 |
| RollbackOerationTask | 7 | 7 | 0 |
| ScaleVNFTask | 7 | 7 | 0 |
| Subscriptions | 15 | 10 | 5 |
| TerminateVNFTask | 6 | 4 | 2 |
| VNFInstances | 13 | 9 | 4 |
| | 74 | 53(71.62%) |21(28.38%) |

#### VNF Package Management API
The table and figures below summarize the results for VNF Package Management API API per test session for [NFV-SOL003] and [NFV-SOL005] .

##### NFV-SOL003
| Test session | Total | Pass | Fail |
| --------------------- | ----- | ---------- | ---------- |
| VNFPackages | 14 | 10 | 4 |
| VNFPackageArtifacts | 11 | 8 | 3 |
| Individual VNFPackage | 6 | 4 | 2 |
| VNFPackageContent | 10 | 7 | 3 |
| | 41 | 29(70.73%) | 12(29.27%) |

##### NFV-SOL005
| Test session | Total | Pass | Fail |
| ----------------------- | ----- | ---------- | ---------- |
| VNFPackages | 16 | 11 | 5 |
| VNFPackageArtifacts | 11 | 5 | 6 |
| Individual VNFPackage | 11 | 7 | 4 |
| VNFPackageContent | 11 | 4 | 7 |
| VNFPackageContentViaURI | 6 | 5 | 1 |
| | 55 | 32(58.18%) | 23(41.82%) |

### Test Results in Detail (Oct.)

The following clauses provide tables and figures which summarize the latest results for the NFV API Conformance test for the different APIs

#### NFV-SOL003
| | Pass | Fail | Totals |
| --------------------------- | ---------- | ---------- | ------ |
| VNF LifecycleManagement API | 69(85.19%) | 12(14.81%) | 81 |

#### NFV-SOL005
| | Pass | Fail | Totals |
| --------------------------- | ---------- | ---------- | ------ |
| VNF Package Management API | 50(90.91%) | 5(9.09%) | 55 |

#### VNF LifecycleManagement API
The table and figures below summarize the results for VNFLifecycleManagement API per test session for [NFV-SOL003].

| Test session | Total | Pass | Fail |
| ---------------------------------- | ----- | ---------- | ---------- |
| Heal VNF Task | 7 | 7 | 0 |
| IndividualVnfInstance | 8 | 6 | 2 |
| IndividualVnfLcmOperationOccurence | 5 | 4 | 1 |
| InstantiateVNFTask | 6 | 6 | 0 |
| RollbackOerationTask | 7 | 7 | 0 |
| ScaleVNFTask | 7 | 7 | 0 |
| IndividualSubscription | 6 | 6 | 0 |
| Subscriptions | 17 | 14 | 3 |
| TerminateVNFTask | 6 | 4 | 2 |
| VNFInstances | 13 | 9 | 4 |
| ChangeExternalVNFConnectivityTask | 5 | 4 | 1 |
| | 87 | 74(85.06%) | 13(14.94%) |

#### VNF Package Management API
The table and figures below summarize the results for VNF Package Management API API per test session for [NFV-SOL003] and [NFV-SOL005] .

| Test session | Total | Pass | Fail |
| ----------------------- | ----- | ---------- | -------- |
| VNFPackages | 16 | 12 | 3 |
| VNFPackageArtifacts | 11 | 11 | 0 |
| Individual VNFPackage | 11 | 11 | 0 |
| VNFPackageContent | 11 | 9 | 2 |
| VNFPackageContentViaURI | 6 | 6 | 0 |
| | 55 | 50(90.91%) | 5(9.09%) |

## Plugtest Outcome
The following clauses shows the detaild of feedback on NFV Specifications and the issues we reported both on Robot Test Suite and Tacker.
### Feedback on NFV Specifications
#### Potential inconsistency on disk and container format
A retrieved inconsistency among the NFV-SOL specifications has been identified related to the definition of disk and
container format permitted values in the VnfPackageSoftwareImageInfo data type in [NFV-SOL003] and [NFVSOL005]. In particular, [NFV-SOL005] in Table 9.5.3.2-1 (and the same in [NFV-SOL003] Table 10.5.3.2-1) specifies.the permitted values disk and container format attributes as all uppercase
However, two notes in the [NFV-SOL005] Table 9.5.3.2-1 (and the same in [NFV-SOL003] Table 10.5.3.2-1) state that the list of permitted values was taken from “Container formats” and “Disk Formats” defined by Openstack in:https://docs.openstack.org/glance/pike/user/formats.html
In these Openstack reference definitions, the disk and container format permitted values are defined as all lowercase, so not aligned with [NFV-SOL005] Table 9.5.3.2-1 and [NFV-SOL003] Table 10.5.3.2-1 specifications.

Issue 8008 has been reported to NFV SOL Working Group to notify this potential inconsistency.

#### Usability issue on hardcoded authorization header name
From the specification perspective, the [NFV-SOL013] specifies the header name for authorization to be required, this being the name “Authorization”. However, as part of OpenStack, Tacker use the header "X-Auth-Token" which avoiding us to run the NFV API Conformance tests at all
in order to achieve conformance to the specifications, Tacker shall work to allow their products and implementation to utilize the standard authorization header name ( i.e. “Authorization”).

### Feedback on the NFV Robot Test Suite
The issues we reported during the tests are as followed.
- Bug#133 The parameter ${vnfLcmOpOccId} is defined repetitiously in the variables.txt file and the configuration.txt file
- Bug#140 Wrong Keywords in SOL003 VNFPackageManagement-API VNFPackageContent.robot
- Bug#143 Wrong parameter name in 'IndividualVnfLcmOperationOccurence.robot' SOL003 VNFLifecycleManagement-API v2.6.1
- Bug#144 Wrong method in Keywords 'GET Scale vnfInstance' SOL003 VNFLifecycleManagement-API v2.6.1
- Bug#145 GET APIs with Range request in v2.6.1 VNFPackageManagement-API
- Bug#147 Wrong parameter vlalue in 'Get all VNF Packages with malformed authorization token' SOL005 VNFPackges.robot V2.6.1
- Bug#150 Missing Set Suite Variable in keyword 'Check Postcondition VNF Package Artifact Exist' v2.6.1 SOL003&SOL005 VNFPackageManagement-API
- Bug#152 Definition of disk and container format conflicted with Specification v2.6.1 SOL003&SOL005 VNFPackageManagementAPI
- Bug#158 \"=\" should not be included in Content range SOL003/SOL005 VNFPackageManagementAPI V2.6.1
- Bug#159 FileNotFoundError: No such file or directory:\"schema//Subscriptions.schema.json\" SOL003 VNFLifecycleMangagementAPI v2.6.1
- Bug#151 - When ${NFVO_RANGE_OK} =1 ， test case 'XXX and NFVO not supporting Range Requests' should be skipped SOL003 v2.6.1 VNFPackage ManagementAPI (opened)
- Bug#165 - Using the same parameter when checking full size and Content-length SOL003 v2.6.1 VNFPackageManagementAPI (opened)
- Bug#166 - Wrong data type for links SOL005 VNFPackageManagementAPI v2.6.1 (opened)
- Bug#167 - POST Terminate a vnfInstance Conflict returned http code 202 SOL003 VNFLifecycleManagementAPI v2.6.1 (opened)
- Bug#135 – SOL003/VNF LCM – conflicting operations on vnf instances
- Bug#153 – SOL003/VNFPackageArtifacts/2.6.1 Range is kept all along the test cases
- Bug#146 – Mispalced referenced json requests in VNFPackageManagementKeywords.robot
- Bug#193-unexpected GET method runs after http code 303 returned in Subscriptions.robot SOL003 VNFLifecycleManagementAPI
- Bug#190-Wrong Problem Details JSON Object set in Keywords Check HTTP Response Body Json Schema Is ProblemDetails SOL005 VNFPackageManagementAPI v2.6.1
- Bug#192-Wrong Parameter name in IndividualVNFInstance.robot SOL003 VNFLifecysleManagementAPI
- Bug#157-Unknown parameter value appeared when testing SOL003 VNFPackageManagementAPI V2.6.1
- Bug#194-Keywords not running when testing Individual VNFInstances SOL003 VNFLifecycleMangementAPI V2.6.1

### Feedback on Tacker
The bugs we reported during the tests are as followed.
- #1945387 Missing Location in http header when changing external connectivity  (Oct.)
- #1946412 VNF Instance can be terminated and patched at the same time (Oct.)
- #1915100 Accept unsupported Function 'Patch Individual VNFInstance‘
- #1915107 Opeartions with VNFInstance doesn't return a "Location" http header which includes vnfLcmOpOccid
- #1915109 Missing Location in http header when creating new VNF Package Resource
- #1915119 Wrong http code returned when getting all vnf packages with malformed authorization token