Regression Summary Result

# Regression Summary Result ## Result of p1 TestNo | Reason for failure -------|-------------------- Case.2-2 | nso NG due to acl is found to be disabled Case.6-1 | possibly a bug from the nso (6-1 and 7-1 is exactly the same but 7-1 passed); nso NG Case.11-1 | nso NG due to acl is found to be disabled (same as test 2-2) **NOTE** - newer csr default is disabled, must enable it before running the test ``` admin@ncs(config)# megapop l2nid-mpls-csr mpop-bgp-1 access-list acl-feature [disable,enable] (enable): enable ``` ## Result of p2 (covers mainly the bgp service) TestNo | Reason for failure -------|-------------------- Case.26-1 onwards | cbd database locked issue when rollback **ISSUE:** cbd database locked issue when rollback ``` I observed sth, with a fresh new service, no.26-1 will be successful. BUT it will end with the following ---------------------------------------------------------------------- [2021-02-26 15:38:18]: 12. Validate network deletion(if required) [2021-02-26 15:38:18]: 13. Rollback NSO (NSO API or NSO CLI or CURL) run remote cmd <--rollback command triggered start ssh 40473 end ssh revision: 40473 run remote cmd start ssh Aborted: the configuration database is locked <-- 409 conflict response ---------------------------------------------------------------------- ``` with this problem, the rollback did not execute properly. Hence, the old config from the previous testcase is still in the router. triggering the problem of esc out of sync.. I tried the manual way on cli, it was ok ``` admin@ncs(config)# rollback configuration 40473 admin@ncs(config)# commit dry-run admin@ncs#commit <--ok, no problem normal way in script should be ---------------------------------------------------------------------- [2021-02-26 15:40:18]: 13. Rollback NSO (NSO API or NSO CLI or CURL) run remote cmd start ssh 40474 end ssh -> OK ---------------------------------------------------------------------- ``` Then i saw this: ``` Aborted: RPC error towards sngtp-esc: operation_failed: Unable to process Update request for deployment id: SystemAdminTenantIdmpop-bgp-1_L2M_CSR_sngtp-esc, a previous configuration request for this deployment is still in progress. ``` So I figured, our esc needs to take a longer time to handle the request before taking new request. i tweaked the code (workflow/workflow13.py) to add time delay for executing rollback. ``` [2021-02-28 15:38:31]: 13. Rollback NSO (NSO API or NSO CLI or CURL) run remote cmd start ssh 40556 end ssh revision: 40556 run remote cmd <--i added a time delay before running the 2nd rollback start ssh Commit complete. end ssh -> OK TestNo.26-2 Success ``` ### Additional findings for the CDB locked issue I suppose actually the script runs a rollback only one time like other test cases, as we can see the output from line 67 and 70 of `workflow/workflow13.py` only one time. Seems like somebody is locking the cdb when rollback is happening. So, the script just have to to wait for more time before running the rollback. Then our workaround is adding `time.sleep(90)` before `top_revision = nso.nso_retrieve_revision()` to solve the issue. workflow/workflow13.py: ```python 57 elif cmd_type == "ROLLBACK": 58 time.sleep(90) 59 top_revision = nso.nso_retrieve_revision() 60 print('LOOK HERE FOR TOP_REVISION' + str(top_revision)) 61 62 if top_revision > revision: 63 int_rev = int(revision) + 1 64 revision = str(int_rev) 65 66 logger.write_value("LOOK HERE FOR revision: " + revision) 67 print("=============ROLLBACK STARTS============") 68 message = nso.rollback(revision) 69 print('LOOK HERE FOR MESSAGE' + str(message)) 70 print("=============ROLLBACK ENDS==============") 71 if not util.isNull(message) and message.lower().find("error") > 0: 72 print("=============ROLLBACK ERROR MSG================") 73 ret_message = ret_message + "\n" + message + "\n***********************" 74 result = "NG" 75 print("============END OF ROLLBACK ERROR MSG===============") ``` The result before adding the delay: ``` LOOK HERE FOR CMD Number#0: <-- start of loop#1 (hit the condition on line 62) run remote cmd start ssh 40862 end ssh LOOK HERE FOR TOP_REVISION40862 LOOK HERE FOR revision: 40862 =============ROLLBACK STARTS============ <-- output from line 67 run remote cmd start ssh Aborted: the configuration database is locked <-- an error message from NSO cli end ssh LOOK HERE FOR MESSAGEAborted: the configuration database is locked =============ROLLBACK ENDS============== <-- output from line 70 LOOK HERE FOR CMD Number#1: <-- start of loop#2 (doesn't hit the condition on line 62) LOOK HERE FOR CMD Number#2: <-- start of loop#3 (doesn't hit the condition on line 62) -> OK TestNo.26-1 Success ``` The result after adding the delay (on line 58): ``` LOOK HERE FOR CMD Number#0: run remote cmd start ssh 40877 end ssh LOOK HERE FOR TOP_REVISION40877 LOOK HERE FOR revision: 40876 =============ROLLBACK STARTS============ run remote cmd start ssh Commit complete. <-- message returned from NSO cli (meant rollback successfully) end ssh LOOK HERE FOR MESSAGECommit complete. =============ROLLBACK ENDS============== LOOK HERE FOR CMD Number#1: LOOK HERE FOR CMD Number#2: -> OK TestNo.26-1 Success ``` Moreover, most of test cases have 3 times looping for rollback (according to a number of rows in the Excel sheet per test case, i.e. 3 rows per test case), but just enter the condition `if top_revision > revision:` only one time, hence there is only one time rollback. ## Result of p3 (covers mainly the static service) TestNo | Reason for failure -------|-------------------- Case.26-1 onwards | cbd database locked issue when rollback ## Result of p4 TestNo | Reason for failure -------|-------------------- Case.51-1-5 | dashboard testcase Case.51-16-219 | utm testcase, can ignore for now Case.64-1 onwards | cbd database locked issue when rollback **NOTE** - fail from testcase no. 64-1, same issue as p2, p3 ## Result of p5 TestNo | Reason for failure -------|-------------------- Case.69-1-198| security map issue Case.70-1-79 | security map issue Case.71-1-50 | security map issue Case.84-1-32,53-54 | is due to "sequence 100 deny ipv6 any any by default" added at the end ipv6 acl for this newer csr **ISSUE:** security map config cannot be added from nso cli - configuration can manually added directly in the csr in telepark, configuration cannot be added ``` admin@ncs(config)# megapop l2nid-mpls-csr mpop-bgp-1 admin@ncs(config-l2nid-mpls-csr-mpop-bgp-1)# security class-map SEC-ALLOWED-PROTOCOLS match-type match-any protocol 802-11-iapp admin@ncs(config-class-map-SEC-ALLOWED-PROTOCOLS)# commit dry-run cli <-- this is the abnormal response we get ``` However, in preprod, configuration can be added ``` + security { + class-map SEC-ALLOWED-PROTOCOLS { + match-type match-any; + protocol 802-11-iapp; + } + } ``` We checked the packages in both sites, we are not missing any packages from the preprod ##### telepark ``` admin@ncs# show packages package package-version PACKAGE NAME VERSION ------------------------------------- Sites 1.2 cisco-ios 5.4.6 cisco-iosxr 4.6.2 csr-license 0.0.2 dashboard 1.1 dashboard-csr 2.1 dashboard-utm 0.3.14 dd-resource-manager-shared 0.3.14 esc 2.3.1.5 fortinet-fortios 4.0.7 hostname 0.0.1 ipv6 1.0.2 nfv 3.6.1 nfv-rpc 0.3.14 nso-util 1.0 quagga-bgp 3.4.1 sites-common 0.3.14 statefull-firewall 1.0.0 tailf-etsi-rel2-nfvo 1.3.0 tailf-etsi-rel2-nfvo-esc 1.3.0 tailf-hcc 4.2.0 utm 0.3.14 utm-service 0.3.14 ``` ##### preprod ``` admin@lab-nfv-nso-1# show packages package package-version PACKAGE NAME VERSION ------------------------------------- Sites 1.2 cisco-ios 5.4.6 cisco-iosxr 4.6.2 csr-license 0.0.2 dashboard 1.1 dashboard-csr 2.1 dashboard-utm 0.3.14 dd-resource-manager-shared 0.3.14 esc 2.3.1.5 fortinet-fortios 4.0.7 hostname 0.0.1 ipv6 1.0.2 nfv 3.6.1 nfv-rpc 0.3.14 nso-util 1.0 sites-common 0.3.14 statefull-firewall 1.0.0 tailf-etsi-rel2-nfvo 1.3.0 tailf-etsi-rel2-nfvo-esc 1.3.0 utm 0.3.14 utm-service 0.3.14 admin@lab-nfv-nso-1# ``` ## Result of p6 (service model testcases) TestNo | Reason for failure -------|-------------------- Case 2: 10-11 | dhcp-param issue for multiple value on dns-server *\*preprod has the same issue\** Case 2: 12-13 | dhcp-param issue for multiple value on dns-server *\*preprod has the same issue\** Case 3: 1,3 | static config is fine. it is pushed and matched what is expected, testcase is inaccurate Case 6: 3-4,6 | test case error with meaningless "test" in the excel **ISSUE:** to update dns-server to both 192.168.1.2, 192.168.1.3. manual (nso cli) works but from API only 192.168.1.3 is committed. *(\*\*to flag up!)* ``` admin@ncs(config)# devices device admin-mpop-bgp-1_L2M_CSR_sngtp-esc-CSR1kv-CSR-sngtp-esc-1 config admin@ncs(config-config)# ios:ip dhcp pool Pool3 admin@ncs(dhcp-config)# dns-server 192.168.1.2 192.168.1.3 admin@ncs(dhcp-config)# commit dry-run cli devices { device admin-mpop-bgp-1_L2M_CSR_sngtp-esc-CSR1kv-CSR-sngtp-esc-1 { config { ios:ip { dhcp { pool { + dns-server 192.168.1.2 192.168.1.3; } } } } } ``` ## Result of p7 (service model testcases) TestNo | Reason for failure -------|-------------------- Case 2: 5-7 | test case error with meaningless "test" in the excel Case 8: 8 | test case error with meaningless "test" in the excel Case 8: 10 | not the test test, but it failed on the preprod as well Case 8: 11-12 | test case error with meaningless "test" in the excel Case 9: 1-16 | test case 9-1 to 9-16 is due to the config changes in the newer csr, except 9-9 is test case error with meaningless "test" in the excel Case 13: 11,13 | 11: test case error with meaningless "test" in the excel, 13: test case structure error Case 15: 2 | newer csr only accepts hold time from the range between 900 and 3000 Case 16: 1-12 | security map issue, same issue as p5 Case 17: 3,5-13 | 3: test case structure error, preprod has the same error, 5-7 some config diff because of the new infra config (preprod has the same error too), 8-13: test case error with meaningless "test" in the excel Case 18: 2,3,5-8 | same preprod error with additional "prefix-delegation pool LocalV6PrefixPool3 lifetime 1800 600" **ISSUE: case 9-1 to 9-16** These 2 commands are default in the newer csr. So, it won't show. Older version will show the config with default value. ``` graceful-restart restart-time 120 graceful-restart stale-path-time 360 ``` **ISSUE: case 15-2** Error observed: ``` {"errors": {"error": [{"error-message": "Network Element Driver error ned_external_error for device admin-mpop-bgp-1_L2M_CSR_sngtp-esc-CSR1kv-CSR-sngtp-esc-1: standby 20 timers msec 300 msec 800: standby 20 timers msec 300 msec 800 ``` Manual configuration on the csr shows the problem: ``` MPOPBGP000001(config-if)# standby 20 timers msec 300 msec 800 ^ % Invalid input detected at '^' marker. MPOPBGP000001(config-if)# standby 20 timers msec 300 msec ? <900-3000> Hold time in milliseconds MPOPBGP000001(config-if)# standby 20 timers msec 300 msec MPOPBGP000001(config-if)# standby 20 timers msec 300 msec 900 ```

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.