owned this note
owned this note
Published
Linked with GitHub
# Bug [1765839](https://bugzilla.redhat.com/show_bug.cgi?id=1765839) investigation notes
I'm tracking my investigation into an issue where the `openstack undercloud install` command does not update the certificates.
I'm using the email attached to the certificate to quickly verify that the certificate has been updated. The original certificate has the email `original-cert@example.com` and the replacement certificate has the email `new-cert@example.com`.
## Prior to `openstack undercloud install`
### All of the certificates match
#### endpoint certificate
```
[root@undercloud private]# openssl s_client -connect 192.168.24.2:13000
CONNECTED(00000003)
depth=1 C = US, ST = Texas, L = San Antonio, O = Red Hat, OU = SEC-DFG, CN = undercloud.ooo.test, emailAddress = dwilde@redhat.com
verify return:1
depth=0 C = US, ST = Texas, O = Red Hat, OU = SECDFG, CN = 192.168.24.2, emailAddress = original-cert@example.com
verify return:1
---
Certificate chain
0 s:/C=US/ST=Texas/O=Red Hat/OU=SECDFG/CN=192.168.24.2/emailAddress=original-cert@example.com
i:/C=US/ST=Texas/L=San Antonio/O=Red Hat/OU=SEC-DFG/CN=undercloud.ooo.test/emailAddress=dwilde@redhat.com
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIEsDCCApigAwIBAgICEAUwDQYJKoZIhvcNAQELBQAwgZcxCzAJBgNVBAYTAlVT
MQ4wDAYDVQQIDAVUZXhhczEUMBIGA1UEBwwLU2FuIEFudG9uaW8xEDAOBgNVBAoM
B1JlZCBIYXQxEDAOBgNVBAsMB1NFQy1ERkcxHDAaBgNVBAMME3VuZGVyY2xvdWQu
b29vLnRlc3QxIDAeBgkqhkiG9w0BCQEWEWR3aWxkZUByZWRoYXQuY29tMB4XDTIw
MDQxNTE0MzkwOVoXDTMwMDQxMzE0MzkwOVowgYExCzAJBgNVBAYTAlVTMQ4wDAYD
VQQIDAVUZXhhczEQMA4GA1UECgwHUmVkIEhhdDEPMA0GA1UECwwGU0VDREZHMRUw
EwYDVQQDDAwxOTIuMTY4LjI0LjIxKDAmBgkqhkiG9w0BCQEWGW9yaWdpbmFsLWNl
cnRAZXhhbXBsZS5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCg
8z7DOKXZFsBfIW7Oqh0kZHdmgGuVkjHTkTv7JtfD21MCDhvpzsONIz89Sl15Woeo
YKxu4jvpmXkuo/tNBfDO7zziEOkcxT5TvSEhVD0BWBb1Tqh+bkZAi/PhfwTd74xn
E4Y1HtBcGo0nAoI39EpELcN5ZzxnFjrUYOk5GHDh2pjnvh0qttpOaIno148mjxLy
jMVaWKKf06iDWOwL/xn2PjUKEwtfXI0VxsMxAC4fBIgqw3z1jQ43AYOp/vnPd1s4
YxIgoZvj2rLwBfWH9c2KhYSSHRnc06U9LCmlTV7gxGO4wAuErHs0elvTZV0RA7TQ
wG/aCNiQQ5UN/Lv62nXNAgMBAAGjGjAYMAkGA1UdEwQCMAAwCwYDVR0PBAQDAgXg
MA0GCSqGSIb3DQEBCwUAA4ICAQBXeVYxNfU//sPLcBbC7Kt+bdgS2pZkaRrNrnbt
NWql2i0tCR9o+FOCqXH4g+gVmF44dV5BLbIk6YtvbsI00le3gKAwR09CeVfF33B4
h5s31wtL6LZJLSNcA3ttwz2L0yVO7Tb7BxxIqA28aHdnQpRMCd9hmq29qmbry4Ww
pm6CcGo/BUNHy2O2mNPxxWooUzq+s3CEvqj7kuoJ7W10mdoSQyVZIypZs/mH1lXf
GVyj+FxCeTBgS2s0i29WFXjj4PbR7L2D3e9PolsVfjevBekBrk51zcTwJELJnIhb
1+xLlzgtKbq+DW4Ojn7JYcWTPfZGLZpzbi0qPUZx9YSwnPITn6q/mEWRNlbgx9j4
wySmtRgjB4FgfXITm7kGadHtMbrIWRXkDzAggnWHHA/eQGluoeN5V6eU2LfSIQkC
+BnNVDHEqZWGqj0n1gKKrHX5sjn2JZUOYLIrcnxVkN72GAsZVKcxphvKuk0b1znj
D+W1LQDzU4vLzmLP864qJ70viZwhB2j1SVTI79HrV/GPfRt6uF/CA433RHwDmVsr
eLSc/t8WD2EucKupEhzFjktlOtXgs59e6xS4nxD+wbVeCq8L3lWisLfvvLX5koMX
s5skWz/69VOPc3BtBTML2hWAViBAZxxCdhkcgMpt1V2UBbC9v3Fs4yTuxNP5OE3E
QXrCPA==
-----END CERTIFICATE-----
subject=/C=US/ST=Texas/O=Red Hat/OU=SECDFG/CN=192.168.24.2/emailAddress=original-cert@example.com
issuer=/C=US/ST=Texas/L=San Antonio/O=Red Hat/OU=SEC-DFG/CN=undercloud.ooo.test/emailAddress=dwilde@redhat.com
---
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 1863 bytes and written 415 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
Session-ID: 69334620EF5B0DB3F77D442E095C18C9BA5A052632C88D8F3595FB8BBE8A58A5
Session-ID-ctx:
Master-Key: 4AEEE5FC255632ED75ED01E7EFF1C2FA2DCBE998672D5394C350288BD0F1A37D14AB847B1D631F6E655BBBE20EEEF1B5
Key-Arg : None
Krb5 Principal: None
PSK identity: None
PSK identity hint: None
TLS session ticket lifetime hint: 300 (seconds)
TLS session ticket:
0000 - 25 61 9b e2 f0 ab e1 fd-40 66 b6 16 e5 2d 2e 72 %a......@f...-.r
0010 - 8d 32 82 d7 bf 2b 77 45-30 ea dd 5d 2e 09 83 57 .2...+wE0..]...W
0020 - 47 fd 13 0b 5a 66 10 0b-25 5d 16 04 32 4c 42 a1 G...Zf..%]..2LB.
0030 - 01 91 bb 6d 9e 64 d7 5e-2a ac 26 4b 27 1d bc 8b ...m.d.^*.&K'...
0040 - e9 08 51 d9 a2 db dc d8-22 f7 49 d0 24 e2 71 54 ..Q.....".I.$.qT
0050 - 71 8a 09 75 31 1b ee b2-73 d8 b6 0c 8d e6 0e a8 q..u1...s.......
0060 - be 21 fa 13 d8 2c 3d dd-b6 0f d9 1d 21 7f 9c 9b .!...,=.....!...
0070 - d3 30 64 e4 03 ca 60 97-1a b2 2a 05 cb 05 cf 37 .0d...`...*....7
0080 - a3 0e 3a 2e 18 a3 3a 48-ca dc 02 0c 88 7e 40 f0 ..:...:H.....~@.
0090 - a6 f6 3f a4 c2 e1 ef 7f-ea 5d 42 a6 da 54 04 8b ..?......]B..T..
Start Time: 1586976341
Timeout : 300 (sec)
Verify return code: 0 (ok)
---
HTTP/1.0 408 Request Time-out
Cache-Control: no-cache
Connection: close
Content-Type: text/html
<html><body><h1>408 Request Time-out</h1>
Your browser didn't send a complete request in time.
</body></html>
closed
```
#### undercloud filesystem certificate
```
[root@undercloud private]# openssl x509 -in /etc/pki/tls/private/overcloud_endpoint.pem -text
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 4101 (0x1005)
Signature Algorithm: sha256WithRSAEncryption
Issuer: C=US, ST=Texas, L=San Antonio, O=Red Hat, OU=SEC-DFG, CN=undercloud.ooo.test/emailAddress=dwilde@redhat.com
Validity
Not Before: Apr 15 14:39:09 2020 GMT
Not After : Apr 13 14:39:09 2030 GMT
Subject: C=US, ST=Texas, O=Red Hat, OU=SECDFG, CN=192.168.24.2/emailAddress=original-cert@example.com
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
00:a0:f3:3e:c3:38:a5:d9:16:c0:5f:21:6e:ce:aa:
1d:24:64:77:66:80:6b:95:92:31:d3:91:3b:fb:26:
d7:c3:db:53:02:0e:1b:e9:ce:c3:8d:23:3f:3d:4a:
5d:79:5a:87:a8:60:ac:6e:e2:3b:e9:99:79:2e:a3:
fb:4d:05:f0:ce:ef:3c:e2:10:e9:1c:c5:3e:53:bd:
21:21:54:3d:01:58:16:f5:4e:a8:7e:6e:46:40:8b:
f3:e1:7f:04:dd:ef:8c:67:13:86:35:1e:d0:5c:1a:
8d:27:02:82:37:f4:4a:44:2d:c3:79:67:3c:67:16:
3a:d4:60:e9:39:18:70:e1:da:98:e7:be:1d:2a:b6:
da:4e:68:89:e8:d7:8f:26:8f:12:f2:8c:c5:5a:58:
a2:9f:d3:a8:83:58:ec:0b:ff:19:f6:3e:35:0a:13:
0b:5f:5c:8d:15:c6:c3:31:00:2e:1f:04:88:2a:c3:
7c:f5:8d:0e:37:01:83:a9:fe:f9:cf:77:5b:38:63:
12:20:a1:9b:e3:da:b2:f0:05:f5:87:f5:cd:8a:85:
84:92:1d:19:dc:d3:a5:3d:2c:29:a5:4d:5e:e0:c4:
63:b8:c0:0b:84:ac:7b:34:7a:5b:d3:65:5d:11:03:
b4:d0:c0:6f:da:08:d8:90:43:95:0d:fc:bb:fa:da:
75:cd
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Basic Constraints:
CA:FALSE
X509v3 Key Usage:
Digital Signature, Non Repudiation, Key Encipherment
Signature Algorithm: sha256WithRSAEncryption
57:79:56:31:35:f5:3f:fe:c3:cb:70:16:c2:ec:ab:7e:6d:d8:
12:da:96:64:69:1a:cd:ae:76:ed:35:6a:a5:da:2d:2d:09:1f:
68:f8:53:82:a9:71:f8:83:e8:15:98:5e:38:75:5e:41:2d:b2:
24:e9:8b:6f:6e:c2:34:d2:57:b7:80:a0:30:47:4f:42:79:57:
c5:df:70:78:87:9b:37:d7:0b:4b:e8:b6:49:2d:23:5c:03:7b:
6d:c3:3d:8b:d3:25:4e:ed:36:fb:07:1c:48:a8:0d:bc:68:77:
67:42:94:4c:09:df:61:9a:ad:bd:aa:66:eb:cb:85:b0:a6:6e:
82:70:6a:3f:05:43:47:cb:63:b6:98:d3:f1:c5:6a:28:53:3a:
be:b3:70:84:be:a8:fb:92:ea:09:ed:6d:74:99:da:12:43:25:
59:23:2a:59:b3:f9:87:d6:55:df:19:5c:a3:f8:5c:42:79:30:
60:4b:6b:34:8b:6f:56:15:78:e3:e0:f6:d1:ec:bd:83:dd:ef:
4f:a2:5b:15:7e:37:af:05:e9:01:ae:4e:75:cd:c4:f0:24:42:
c9:9c:88:5b:d7:ec:4b:97:38:2d:29:ba:be:0d:6e:0e:8e:7e:
c9:61:c5:93:3d:f6:46:2d:9a:73:6e:2d:2a:3d:46:71:f5:84:
b0:9c:f2:13:9f:aa:bf:98:45:91:36:56:e0:c7:d8:f8:c3:24:
a6:b5:18:23:07:81:60:7d:72:13:9b:b9:06:69:d1:ed:31:ba:
c8:59:15:e4:0f:30:20:82:75:87:1c:0f:de:40:69:6e:a1:e3:
79:57:a7:94:d8:b7:d2:21:09:02:f8:19:cd:54:31:c4:a9:95:
86:aa:3d:27:d6:02:8a:ac:75:f9:b2:39:f6:25:95:0e:60:b2:
2b:72:7c:55:90:de:f6:18:0b:19:54:a7:31:a6:1b:ca:ba:4d:
1b:d7:39:e3:0f:e5:b5:2d:00:f3:53:8b:cb:ce:62:cf:f3:ae:
2a:27:bd:2f:89:9c:21:07:68:f5:49:54:c8:ef:d1:eb:57:f1:
8f:7d:1b:7a:b8:5f:c2:03:8d:f7:44:7c:03:99:5b:2b:78:b4:
9c:fe:df:16:0f:61:2e:70:ab:a9:12:1c:c5:8e:4b:65:3a:d5:
e0:b3:9f:5e:eb:14:b8:9f:10:fe:c1:b5:5e:0a:af:0b:de:55:
a2:b0:b7:ef:bc:b5:f9:92:83:17:b3:9b:24:5b:3f:fa:f5:53:
8f:73:70:6d:05:33:0b:da:15:80:56:20:40:67:1c:42:76:19:
1c:80:ca:6d:d5:5d:94:05:b0:bd:bf:71:6c:e3:24:ee:c4:d3:
f9:38:4d:c4:41:7a:c2:3c
-----BEGIN CERTIFICATE-----
MIIEsDCCApigAwIBAgICEAUwDQYJKoZIhvcNAQELBQAwgZcxCzAJBgNVBAYTAlVT
MQ4wDAYDVQQIDAVUZXhhczEUMBIGA1UEBwwLU2FuIEFudG9uaW8xEDAOBgNVBAoM
B1JlZCBIYXQxEDAOBgNVBAsMB1NFQy1ERkcxHDAaBgNVBAMME3VuZGVyY2xvdWQu
b29vLnRlc3QxIDAeBgkqhkiG9w0BCQEWEWR3aWxkZUByZWRoYXQuY29tMB4XDTIw
MDQxNTE0MzkwOVoXDTMwMDQxMzE0MzkwOVowgYExCzAJBgNVBAYTAlVTMQ4wDAYD
VQQIDAVUZXhhczEQMA4GA1UECgwHUmVkIEhhdDEPMA0GA1UECwwGU0VDREZHMRUw
EwYDVQQDDAwxOTIuMTY4LjI0LjIxKDAmBgkqhkiG9w0BCQEWGW9yaWdpbmFsLWNl
cnRAZXhhbXBsZS5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCg
8z7DOKXZFsBfIW7Oqh0kZHdmgGuVkjHTkTv7JtfD21MCDhvpzsONIz89Sl15Woeo
YKxu4jvpmXkuo/tNBfDO7zziEOkcxT5TvSEhVD0BWBb1Tqh+bkZAi/PhfwTd74xn
E4Y1HtBcGo0nAoI39EpELcN5ZzxnFjrUYOk5GHDh2pjnvh0qttpOaIno148mjxLy
jMVaWKKf06iDWOwL/xn2PjUKEwtfXI0VxsMxAC4fBIgqw3z1jQ43AYOp/vnPd1s4
YxIgoZvj2rLwBfWH9c2KhYSSHRnc06U9LCmlTV7gxGO4wAuErHs0elvTZV0RA7TQ
wG/aCNiQQ5UN/Lv62nXNAgMBAAGjGjAYMAkGA1UdEwQCMAAwCwYDVR0PBAQDAgXg
MA0GCSqGSIb3DQEBCwUAA4ICAQBXeVYxNfU//sPLcBbC7Kt+bdgS2pZkaRrNrnbt
NWql2i0tCR9o+FOCqXH4g+gVmF44dV5BLbIk6YtvbsI00le3gKAwR09CeVfF33B4
h5s31wtL6LZJLSNcA3ttwz2L0yVO7Tb7BxxIqA28aHdnQpRMCd9hmq29qmbry4Ww
pm6CcGo/BUNHy2O2mNPxxWooUzq+s3CEvqj7kuoJ7W10mdoSQyVZIypZs/mH1lXf
GVyj+FxCeTBgS2s0i29WFXjj4PbR7L2D3e9PolsVfjevBekBrk51zcTwJELJnIhb
1+xLlzgtKbq+DW4Ojn7JYcWTPfZGLZpzbi0qPUZx9YSwnPITn6q/mEWRNlbgx9j4
wySmtRgjB4FgfXITm7kGadHtMbrIWRXkDzAggnWHHA/eQGluoeN5V6eU2LfSIQkC
+BnNVDHEqZWGqj0n1gKKrHX5sjn2JZUOYLIrcnxVkN72GAsZVKcxphvKuk0b1znj
D+W1LQDzU4vLzmLP864qJ70viZwhB2j1SVTI79HrV/GPfRt6uF/CA433RHwDmVsr
eLSc/t8WD2EucKupEhzFjktlOtXgs59e6xS4nxD+wbVeCq8L3lWisLfvvLX5koMX
s5skWz/69VOPc3BtBTML2hWAViBAZxxCdhkcgMpt1V2UBbC9v3Fs4yTuxNP5OE3E
QXrCPA==
-----END CERTIFICATE-----
```
##### container bind mounted certificate
```
[root@undercloud private]# podman exec -it 8d8991bb489f openssl x509 -in /var/lib/kolla/config_files/src-tls/etc/pki/tls/private/overcloud_endpoint.pem -text
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 4101 (0x1005)
Signature Algorithm: sha256WithRSAEncryption
Issuer: C=US, ST=Texas, L=San Antonio, O=Red Hat, OU=SEC-DFG, CN=undercloud.ooo.test/emailAddress=dwilde@redhat.com
Validity
Not Before: Apr 15 14:39:09 2020 GMT
Not After : Apr 13 14:39:09 2030 GMT
Subject: C=US, ST=Texas, O=Red Hat, OU=SECDFG, CN=192.168.24.2/emailAddress=original-cert@example.com
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
00:a0:f3:3e:c3:38:a5:d9:16:c0:5f:21:6e:ce:aa:
1d:24:64:77:66:80:6b:95:92:31:d3:91:3b:fb:26:
d7:c3:db:53:02:0e:1b:e9:ce:c3:8d:23:3f:3d:4a:
5d:79:5a:87:a8:60:ac:6e:e2:3b:e9:99:79:2e:a3:
fb:4d:05:f0:ce:ef:3c:e2:10:e9:1c:c5:3e:53:bd:
21:21:54:3d:01:58:16:f5:4e:a8:7e:6e:46:40:8b:
f3:e1:7f:04:dd:ef:8c:67:13:86:35:1e:d0:5c:1a:
8d:27:02:82:37:f4:4a:44:2d:c3:79:67:3c:67:16:
3a:d4:60:e9:39:18:70:e1:da:98:e7:be:1d:2a:b6:
da:4e:68:89:e8:d7:8f:26:8f:12:f2:8c:c5:5a:58:
a2:9f:d3:a8:83:58:ec:0b:ff:19:f6:3e:35:0a:13:
0b:5f:5c:8d:15:c6:c3:31:00:2e:1f:04:88:2a:c3:
7c:f5:8d:0e:37:01:83:a9:fe:f9:cf:77:5b:38:63:
12:20:a1:9b:e3:da:b2:f0:05:f5:87:f5:cd:8a:85:
84:92:1d:19:dc:d3:a5:3d:2c:29:a5:4d:5e:e0:c4:
63:b8:c0:0b:84:ac:7b:34:7a:5b:d3:65:5d:11:03:
b4:d0:c0:6f:da:08:d8:90:43:95:0d:fc:bb:fa:da:
75:cd
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Basic Constraints:
CA:FALSE
X509v3 Key Usage:
Digital Signature, Non Repudiation, Key Encipherment
Signature Algorithm: sha256WithRSAEncryption
57:79:56:31:35:f5:3f:fe:c3:cb:70:16:c2:ec:ab:7e:6d:d8:
12:da:96:64:69:1a:cd:ae:76:ed:35:6a:a5:da:2d:2d:09:1f:
68:f8:53:82:a9:71:f8:83:e8:15:98:5e:38:75:5e:41:2d:b2:
24:e9:8b:6f:6e:c2:34:d2:57:b7:80:a0:30:47:4f:42:79:57:
c5:df:70:78:87:9b:37:d7:0b:4b:e8:b6:49:2d:23:5c:03:7b:
6d:c3:3d:8b:d3:25:4e:ed:36:fb:07:1c:48:a8:0d:bc:68:77:
67:42:94:4c:09:df:61:9a:ad:bd:aa:66:eb:cb:85:b0:a6:6e:
82:70:6a:3f:05:43:47:cb:63:b6:98:d3:f1:c5:6a:28:53:3a:
be:b3:70:84:be:a8:fb:92:ea:09:ed:6d:74:99:da:12:43:25:
59:23:2a:59:b3:f9:87:d6:55:df:19:5c:a3:f8:5c:42:79:30:
60:4b:6b:34:8b:6f:56:15:78:e3:e0:f6:d1:ec:bd:83:dd:ef:
4f:a2:5b:15:7e:37:af:05:e9:01:ae:4e:75:cd:c4:f0:24:42:
c9:9c:88:5b:d7:ec:4b:97:38:2d:29:ba:be:0d:6e:0e:8e:7e:
c9:61:c5:93:3d:f6:46:2d:9a:73:6e:2d:2a:3d:46:71:f5:84:
b0:9c:f2:13:9f:aa:bf:98:45:91:36:56:e0:c7:d8:f8:c3:24:
a6:b5:18:23:07:81:60:7d:72:13:9b:b9:06:69:d1:ed:31:ba:
c8:59:15:e4:0f:30:20:82:75:87:1c:0f:de:40:69:6e:a1:e3:
79:57:a7:94:d8:b7:d2:21:09:02:f8:19:cd:54:31:c4:a9:95:
86:aa:3d:27:d6:02:8a:ac:75:f9:b2:39:f6:25:95:0e:60:b2:
2b:72:7c:55:90:de:f6:18:0b:19:54:a7:31:a6:1b:ca:ba:4d:
1b:d7:39:e3:0f:e5:b5:2d:00:f3:53:8b:cb:ce:62:cf:f3:ae:
2a:27:bd:2f:89:9c:21:07:68:f5:49:54:c8:ef:d1:eb:57:f1:
8f:7d:1b:7a:b8:5f:c2:03:8d:f7:44:7c:03:99:5b:2b:78:b4:
9c:fe:df:16:0f:61:2e:70:ab:a9:12:1c:c5:8e:4b:65:3a:d5:
e0:b3:9f:5e:eb:14:b8:9f:10:fe:c1:b5:5e:0a:af:0b:de:55:
a2:b0:b7:ef:bc:b5:f9:92:83:17:b3:9b:24:5b:3f:fa:f5:53:
8f:73:70:6d:05:33:0b:da:15:80:56:20:40:67:1c:42:76:19:
1c:80:ca:6d:d5:5d:94:05:b0:bd:bf:71:6c:e3:24:ee:c4:d3:
f9:38:4d:c4:41:7a:c2:3c
-----BEGIN CERTIFICATE-----
MIIEsDCCApigAwIBAgICEAUwDQYJKoZIhvcNAQELBQAwgZcxCzAJBgNVBAYTAlVT
MQ4wDAYDVQQIDAVUZXhhczEUMBIGA1UEBwwLU2FuIEFudG9uaW8xEDAOBgNVBAoM
B1JlZCBIYXQxEDAOBgNVBAsMB1NFQy1ERkcxHDAaBgNVBAMME3VuZGVyY2xvdWQu
b29vLnRlc3QxIDAeBgkqhkiG9w0BCQEWEWR3aWxkZUByZWRoYXQuY29tMB4XDTIw
MDQxNTE0MzkwOVoXDTMwMDQxMzE0MzkwOVowgYExCzAJBgNVBAYTAlVTMQ4wDAYD
VQQIDAVUZXhhczEQMA4GA1UECgwHUmVkIEhhdDEPMA0GA1UECwwGU0VDREZHMRUw
EwYDVQQDDAwxOTIuMTY4LjI0LjIxKDAmBgkqhkiG9w0BCQEWGW9yaWdpbmFsLWNl
cnRAZXhhbXBsZS5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCg
8z7DOKXZFsBfIW7Oqh0kZHdmgGuVkjHTkTv7JtfD21MCDhvpzsONIz89Sl15Woeo
YKxu4jvpmXkuo/tNBfDO7zziEOkcxT5TvSEhVD0BWBb1Tqh+bkZAi/PhfwTd74xn
E4Y1HtBcGo0nAoI39EpELcN5ZzxnFjrUYOk5GHDh2pjnvh0qttpOaIno148mjxLy
jMVaWKKf06iDWOwL/xn2PjUKEwtfXI0VxsMxAC4fBIgqw3z1jQ43AYOp/vnPd1s4
YxIgoZvj2rLwBfWH9c2KhYSSHRnc06U9LCmlTV7gxGO4wAuErHs0elvTZV0RA7TQ
wG/aCNiQQ5UN/Lv62nXNAgMBAAGjGjAYMAkGA1UdEwQCMAAwCwYDVR0PBAQDAgXg
MA0GCSqGSIb3DQEBCwUAA4ICAQBXeVYxNfU//sPLcBbC7Kt+bdgS2pZkaRrNrnbt
NWql2i0tCR9o+FOCqXH4g+gVmF44dV5BLbIk6YtvbsI00le3gKAwR09CeVfF33B4
h5s31wtL6LZJLSNcA3ttwz2L0yVO7Tb7BxxIqA28aHdnQpRMCd9hmq29qmbry4Ww
pm6CcGo/BUNHy2O2mNPxxWooUzq+s3CEvqj7kuoJ7W10mdoSQyVZIypZs/mH1lXf
GVyj+FxCeTBgS2s0i29WFXjj4PbR7L2D3e9PolsVfjevBekBrk51zcTwJELJnIhb
1+xLlzgtKbq+DW4Ojn7JYcWTPfZGLZpzbi0qPUZx9YSwnPITn6q/mEWRNlbgx9j4
wySmtRgjB4FgfXITm7kGadHtMbrIWRXkDzAggnWHHA/eQGluoeN5V6eU2LfSIQkC
+BnNVDHEqZWGqj0n1gKKrHX5sjn2JZUOYLIrcnxVkN72GAsZVKcxphvKuk0b1znj
D+W1LQDzU4vLzmLP864qJ70viZwhB2j1SVTI79HrV/GPfRt6uF/CA433RHwDmVsr
eLSc/t8WD2EucKupEhzFjktlOtXgs59e6xS4nxD+wbVeCq8L3lWisLfvvLX5koMX
s5skWz/69VOPc3BtBTML2hWAViBAZxxCdhkcgMpt1V2UBbC9v3Fs4yTuxNP5OE3E
QXrCPA==
-----END CERTIFICATE-----
```
### Generate a new certificate
This is based on the steps outlined in Appendix A[^1] of the Director Installation and Usage guide.
```
(undercloud) [stack@undercloud ~]$ ./newcert.sh
+ openssl req -config openssl.cnf -key server.key.pem -new -out server.csr.pem
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [US]:
State or Province Name (full name) [Texas]:
Locality Name (eg, city) [San Antonio]:
Organization Name (eg, company) [Red Hat]:
Organizational Unit Name (eg, section) [SECDFG]:
Common Name (eg, your name or your server's hostname) [192.168.1.14]:192.168.24.2
Email Address []:new-cert@example.com
Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:
+ sudo openssl ca -config openssl.cnf -extensions v3_req -days 3650 -in server.csr.pem -out server.crt.pem -cert ca.crt.pem -keyfile ca.key.pem
Using configuration from openssl.cnf
Check that the request matches the signature
Signature ok
Certificate Details:
Serial Number: 4102 (0x1006)
Validity
Not Before: Apr 15 19:05:47 2020 GMT
Not After : Apr 13 19:05:47 2030 GMT
Subject:
countryName = US
stateOrProvinceName = Texas
organizationName = Red Hat
organizationalUnitName = SECDFG
commonName = 192.168.24.2
emailAddress = new-cert@example.com
X509v3 extensions:
X509v3 Basic Constraints:
CA:FALSE
X509v3 Key Usage:
Digital Signature, Non Repudiation, Key Encipherment
Certificate is to be certified until Apr 13 19:05:47 2030 GMT (3650 days)
Sign the certificate? [y/n]:y
1 out of 1 certificate requests certified, commit? [y/n]y
Write out database with 1 new entries
Data Base Updated
+ cat server.crt.pem server.key.pem
+ sudo cp undercloud.pem /etc/pki/instack-certs/
+ sudo semanage fcontext -a -t etc_t '/etc/pki/instack-certs(/.*)?'
+ sudo restorecon -R /etc/pki/instack-certs
```
## Run `openstack undercloud install`
```
===============================================================================
Wait for containers to start for step 4 using paunch ------------------------------------------------------------------------------------------------- 178.40s
Wait for containers to start for step 3 using paunch ------------------------------------------------------------------------------------------------- 172.10s
Wait for puppet host configuration to finish --------------------------------------------------------------------------------------------------------- 134.77s
Wait for puppet host configuration to finish --------------------------------------------------------------------------------------------------------- 134.76s
Wait for container-puppet tasks (generate config) to finish ------------------------------------------------------------------------------------------ 119.88s
Run puppet on the host to apply IPtables rules -------------------------------------------------------------------------------------------------------- 79.89s
Wait for puppet host configuration to finish ---------------------------------------------------------------------------------------------------------- 59.72s
Wait for puppet host configuration to finish ---------------------------------------------------------------------------------------------------------- 59.67s
Wait for puppet host configuration to finish ---------------------------------------------------------------------------------------------------------- 59.58s
Run deployment UndercloudPostPyDeployment ------------------------------------------------------------------------------------------------------------- 42.26s
Wait for container-puppet tasks (bootstrap tasks) for step 4 to finish -------------------------------------------------------------------------------- 41.01s
Wait for container-puppet tasks (bootstrap tasks) for step 5 to finish -------------------------------------------------------------------------------- 37.83s
Wait for container-puppet tasks (bootstrap tasks) for step 3 to finish -------------------------------------------------------------------------------- 37.78s
Wait for container-puppet tasks (bootstrap tasks) for step 2 to finish -------------------------------------------------------------------------------- 34.63s
Wait for containers to start for step 5 using paunch -------------------------------------------------------------------------------------------------- 31.42s
Write kolla config json files ------------------------------------------------------------------------------------------------------------------------- 18.03s
Creating container startup configs for step_4 --------------------------------------------------------------------------------------------------------- 13.58s
Pre-fetch all the containers -------------------------------------------------------------------------------------------------------------------------- 12.79s
Wait for containers to start for step 2 using paunch -------------------------------------------------------------------------------------------------- 12.75s
Wait for containers to start for step 1 using paunch -------------------------------------------------------------------------------------------------- 12.73s
Install artifact is located at /home/stack/undercloud-install-20200415194554.tar.bzip2
########################################################
Deployment successful!
########################################################
Writing the stack virtual update mark file /var/lib/tripleo-heat-installer/update_mark_undercloud
reset failed: reset: standard error: Inappropriate ioctl for device
Unable to reset command line. Try manually running "reset" if the command line is broken.
##########################################################
The Undercloud has been successfully installed.
Useful files:
Password file is at /home/stack/undercloud-passwords.conf
The stackrc file is at ~/stackrc
Use these files to interact with OpenStack services, and
ensure they are secured.
##########################################################
```
### Check certificates again
#### endpoint certificate
```
[root@undercloud private]# openssl s_client -connect 192.168.24.2:13000
CONNECTED(00000003)
depth=1 C = US, ST = Texas, L = San Antonio, O = Red Hat, OU = SEC-DFG, CN = undercloud.ooo.test, emailAddress = dwilde@redhat.com
verify return:1
depth=0 C = US, ST = Texas, O = Red Hat, OU = SECDFG, CN = 192.168.24.2, emailAddress = original-cert@example.com
verify return:1
---
Certificate chain
0 s:/C=US/ST=Texas/O=Red Hat/OU=SECDFG/CN=192.168.24.2/emailAddress=original-cert@example.com
i:/C=US/ST=Texas/L=San Antonio/O=Red Hat/OU=SEC-DFG/CN=undercloud.ooo.test/emailAddress=dwilde@redhat.com
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIEsDCCApigAwIBAgICEAUwDQYJKoZIhvcNAQELBQAwgZcxCzAJBgNVBAYTAlVT
MQ4wDAYDVQQIDAVUZXhhczEUMBIGA1UEBwwLU2FuIEFudG9uaW8xEDAOBgNVBAoM
B1JlZCBIYXQxEDAOBgNVBAsMB1NFQy1ERkcxHDAaBgNVBAMME3VuZGVyY2xvdWQu
b29vLnRlc3QxIDAeBgkqhkiG9w0BCQEWEWR3aWxkZUByZWRoYXQuY29tMB4XDTIw
MDQxNTE0MzkwOVoXDTMwMDQxMzE0MzkwOVowgYExCzAJBgNVBAYTAlVTMQ4wDAYD
VQQIDAVUZXhhczEQMA4GA1UECgwHUmVkIEhhdDEPMA0GA1UECwwGU0VDREZHMRUw
EwYDVQQDDAwxOTIuMTY4LjI0LjIxKDAmBgkqhkiG9w0BCQEWGW9yaWdpbmFsLWNl
cnRAZXhhbXBsZS5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCg
8z7DOKXZFsBfIW7Oqh0kZHdmgGuVkjHTkTv7JtfD21MCDhvpzsONIz89Sl15Woeo
YKxu4jvpmXkuo/tNBfDO7zziEOkcxT5TvSEhVD0BWBb1Tqh+bkZAi/PhfwTd74xn
E4Y1HtBcGo0nAoI39EpELcN5ZzxnFjrUYOk5GHDh2pjnvh0qttpOaIno148mjxLy
jMVaWKKf06iDWOwL/xn2PjUKEwtfXI0VxsMxAC4fBIgqw3z1jQ43AYOp/vnPd1s4
YxIgoZvj2rLwBfWH9c2KhYSSHRnc06U9LCmlTV7gxGO4wAuErHs0elvTZV0RA7TQ
wG/aCNiQQ5UN/Lv62nXNAgMBAAGjGjAYMAkGA1UdEwQCMAAwCwYDVR0PBAQDAgXg
MA0GCSqGSIb3DQEBCwUAA4ICAQBXeVYxNfU//sPLcBbC7Kt+bdgS2pZkaRrNrnbt
NWql2i0tCR9o+FOCqXH4g+gVmF44dV5BLbIk6YtvbsI00le3gKAwR09CeVfF33B4
h5s31wtL6LZJLSNcA3ttwz2L0yVO7Tb7BxxIqA28aHdnQpRMCd9hmq29qmbry4Ww
pm6CcGo/BUNHy2O2mNPxxWooUzq+s3CEvqj7kuoJ7W10mdoSQyVZIypZs/mH1lXf
GVyj+FxCeTBgS2s0i29WFXjj4PbR7L2D3e9PolsVfjevBekBrk51zcTwJELJnIhb
1+xLlzgtKbq+DW4Ojn7JYcWTPfZGLZpzbi0qPUZx9YSwnPITn6q/mEWRNlbgx9j4
wySmtRgjB4FgfXITm7kGadHtMbrIWRXkDzAggnWHHA/eQGluoeN5V6eU2LfSIQkC
+BnNVDHEqZWGqj0n1gKKrHX5sjn2JZUOYLIrcnxVkN72GAsZVKcxphvKuk0b1znj
D+W1LQDzU4vLzmLP864qJ70viZwhB2j1SVTI79HrV/GPfRt6uF/CA433RHwDmVsr
eLSc/t8WD2EucKupEhzFjktlOtXgs59e6xS4nxD+wbVeCq8L3lWisLfvvLX5koMX
s5skWz/69VOPc3BtBTML2hWAViBAZxxCdhkcgMpt1V2UBbC9v3Fs4yTuxNP5OE3E
QXrCPA==
-----END CERTIFICATE-----
subject=/C=US/ST=Texas/O=Red Hat/OU=SECDFG/CN=192.168.24.2/emailAddress=original-cert@example.com
issuer=/C=US/ST=Texas/L=San Antonio/O=Red Hat/OU=SEC-DFG/CN=undercloud.ooo.test/emailAddress=dwilde@redhat.com
---
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 1863 bytes and written 415 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
Session-ID: 6A0FDC11A30AAF987EEA189C98B0EC6A6B5748576AE7CCCF7260E3970C2FF457
Session-ID-ctx:
Master-Key: 084141C65C9F9551A5ACAF9A629E8C1962A48ED86FADBF0BC77A6D50DC73E2F067895D961A0BD8B7691292933D3FB130
Key-Arg : None
Krb5 Principal: None
PSK identity: None
PSK identity hint: None
TLS session ticket lifetime hint: 300 (seconds)
TLS session ticket:
0000 - 25 61 9b e2 f0 ab e1 fd-40 66 b6 16 e5 2d 2e 72 %a......@f...-.r
0010 - 38 1a 8e d2 1d 2b 4d 39-f5 f7 d9 c5 36 65 b5 bf 8....+M9....6e..
0020 - a3 84 ce a0 1d b3 ff 0c-dc d5 f4 75 5d 54 b2 2a ...........u]T.*
0030 - ac 26 cd 6d 3b 79 0b d3-c3 05 a6 7a 8e d8 42 94 .&.m;y.....z..B.
0040 - 7a 91 7b ba 0a 4d 27 08-54 cd db 60 45 f2 a3 77 z.{..M'.T..`E..w
0050 - 7a 87 26 42 ea ec 7d 5b-14 d2 f0 0e 97 f2 27 e8 z.&B..}[......'.
0060 - 21 60 44 75 1e 3a 17 c4-7b e5 c3 0a 2b fc e4 a9 !`Du.:..{...+...
0070 - b7 51 e2 f3 49 2f 8e b8-ee e7 11 f8 4b e0 df 6f .Q..I/......K..o
0080 - d9 ad 8d 04 9f e2 15 ea-c5 98 7c 1a d8 c1 9c cf ..........|.....
0090 - 5f d8 50 7a 3a db ab 77-e7 bb 44 2c fb 16 0e 90 _.Pz:..w..D,....
Start Time: 1586985233
Timeout : 300 (sec)
Verify return code: 0 (ok)
---
HTTP/1.0 408 Request Time-out
Cache-Control: no-cache
Connection: close
Content-Type: text/html
<html><body><h1>408 Request Time-out</h1>
Your browser didn't send a complete request in time.
</body></html>
closed
```
#### undercloud filesystem certificate
```
[root@undercloud private]# openssl x509 -in /etc/pki/tls/private/overcloud_endpoint.pem -text
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 4102 (0x1006)
Signature Algorithm: sha256WithRSAEncryption
Issuer: C=US, ST=Texas, L=San Antonio, O=Red Hat, OU=SEC-DFG, CN=undercloud.ooo.test/emailAddress=dwilde@redhat.com
Validity
Not Before: Apr 15 19:05:47 2020 GMT
Not After : Apr 13 19:05:47 2030 GMT
Subject: C=US, ST=Texas, O=Red Hat, OU=SECDFG, CN=192.168.24.2/emailAddress=new-cert@example.com
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
00:a0:f3:3e:c3:38:a5:d9:16:c0:5f:21:6e:ce:aa:
1d:24:64:77:66:80:6b:95:92:31:d3:91:3b:fb:26:
d7:c3:db:53:02:0e:1b:e9:ce:c3:8d:23:3f:3d:4a:
5d:79:5a:87:a8:60:ac:6e:e2:3b:e9:99:79:2e:a3:
fb:4d:05:f0:ce:ef:3c:e2:10:e9:1c:c5:3e:53:bd:
21:21:54:3d:01:58:16:f5:4e:a8:7e:6e:46:40:8b:
f3:e1:7f:04:dd:ef:8c:67:13:86:35:1e:d0:5c:1a:
8d:27:02:82:37:f4:4a:44:2d:c3:79:67:3c:67:16:
3a:d4:60:e9:39:18:70:e1:da:98:e7:be:1d:2a:b6:
da:4e:68:89:e8:d7:8f:26:8f:12:f2:8c:c5:5a:58:
a2:9f:d3:a8:83:58:ec:0b:ff:19:f6:3e:35:0a:13:
0b:5f:5c:8d:15:c6:c3:31:00:2e:1f:04:88:2a:c3:
7c:f5:8d:0e:37:01:83:a9:fe:f9:cf:77:5b:38:63:
12:20:a1:9b:e3:da:b2:f0:05:f5:87:f5:cd:8a:85:
84:92:1d:19:dc:d3:a5:3d:2c:29:a5:4d:5e:e0:c4:
63:b8:c0:0b:84:ac:7b:34:7a:5b:d3:65:5d:11:03:
b4:d0:c0:6f:da:08:d8:90:43:95:0d:fc:bb:fa:da:
75:cd
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Basic Constraints:
CA:FALSE
X509v3 Key Usage:
Digital Signature, Non Repudiation, Key Encipherment
Signature Algorithm: sha256WithRSAEncryption
6d:6e:8f:9f:84:e0:9e:69:29:9b:38:59:fe:bb:20:8b:74:17:
b5:7f:de:04:7e:91:38:12:89:30:3b:ed:c9:0a:b0:a3:c2:d0:
d4:25:8a:4d:a2:33:94:47:fe:97:06:3b:26:eb:bd:6f:68:ab:
f9:f2:51:c4:ca:ee:a7:9b:a6:d4:23:6a:70:37:fe:db:30:9e:
b0:f9:8a:ea:95:01:91:e0:04:7f:4e:f5:4c:2c:a2:34:ba:91:
a7:7d:94:7c:54:7a:cf:cb:85:59:14:3c:11:84:3c:44:b4:19:
4b:5c:ca:85:34:e9:6f:d0:32:77:88:92:4f:1f:2e:ca:7c:d0:
80:fa:f2:cd:d7:f2:6b:71:e4:01:81:d1:47:ee:3e:df:0e:1e:
87:ec:a8:08:cc:d6:1e:14:7f:9f:57:8e:cb:04:27:0f:cf:ce:
30:a2:41:20:28:41:ba:59:57:aa:55:d3:95:74:9d:09:7c:80:
e7:0a:5f:61:d4:ab:5d:45:32:72:31:2e:96:a0:a5:01:cc:55:
00:c3:ec:a8:9f:0e:e4:3c:f4:9d:b2:c1:27:81:44:2e:a5:0d:
f5:07:a8:9d:7e:0e:55:38:a7:cc:44:3c:c0:20:3b:1d:d3:81:
e1:18:c2:5e:a7:8a:6a:0e:3c:a5:96:18:56:31:3b:a1:ad:1b:
48:f7:8d:b8:86:d1:8e:73:d0:82:cd:99:94:95:ba:57:f7:23:
8d:fc:07:55:c8:c6:ba:5d:bf:3a:97:2c:d8:6d:de:2e:52:71:
77:51:df:87:f0:34:e5:6a:bc:47:a3:26:1a:d3:5a:83:2e:bd:
44:25:ff:c8:5f:f1:6f:9c:44:ce:82:27:ad:c2:9d:af:7f:7a:
7d:cb:9b:ac:ee:ad:9e:dc:68:6c:e4:25:23:c0:4a:ad:55:35:
b6:2e:ba:63:51:3b:eb:8d:70:21:38:7d:f5:3b:70:bc:1c:43:
e1:23:87:c0:d7:4b:47:ed:16:57:b1:3f:dc:df:e3:d7:f9:ad:
25:1d:f9:7a:92:21:f2:ee:30:63:b1:fc:e0:2f:bb:86:34:91:
e8:c2:77:5d:eb:d6:90:75:64:c3:a8:5a:32:1b:0f:15:99:de:
d5:5e:7d:58:b9:9b:86:ed:78:13:26:a9:42:6e:99:d4:09:ee:
7e:19:62:9e:7a:82:13:6c:e8:8b:76:1f:ee:28:4a:fd:5e:6f:
28:45:48:39:c8:cf:6a:00:75:c9:b2:3e:32:64:32:f4:79:f8:
5c:14:73:b5:0d:ef:e7:c1:6f:5b:7b:48:75:d9:f9:bb:2e:6f:
66:4e:42:2a:ef:28:0b:f0:37:f3:a2:54:d6:cc:ef:94:ad:91:
05:64:33:ee:23:32:6a:b6
-----BEGIN CERTIFICATE-----
MIIEqjCCApKgAwIBAgICEAYwDQYJKoZIhvcNAQELBQAwgZcxCzAJBgNVBAYTAlVT
MQ4wDAYDVQQIDAVUZXhhczEUMBIGA1UEBwwLU2FuIEFudG9uaW8xEDAOBgNVBAoM
B1JlZCBIYXQxEDAOBgNVBAsMB1NFQy1ERkcxHDAaBgNVBAMME3VuZGVyY2xvdWQu
b29vLnRlc3QxIDAeBgkqhkiG9w0BCQEWEWR3aWxkZUByZWRoYXQuY29tMB4XDTIw
MDQxNTE5MDU0N1oXDTMwMDQxMzE5MDU0N1owfDELMAkGA1UEBhMCVVMxDjAMBgNV
BAgMBVRleGFzMRAwDgYDVQQKDAdSZWQgSGF0MQ8wDQYDVQQLDAZTRUNERkcxFTAT
BgNVBAMMDDE5Mi4xNjguMjQuMjEjMCEGCSqGSIb3DQEJARYUbmV3LWNlcnRAZXhh
bXBsZS5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCg8z7DOKXZ
FsBfIW7Oqh0kZHdmgGuVkjHTkTv7JtfD21MCDhvpzsONIz89Sl15WoeoYKxu4jvp
mXkuo/tNBfDO7zziEOkcxT5TvSEhVD0BWBb1Tqh+bkZAi/PhfwTd74xnE4Y1HtBc
Go0nAoI39EpELcN5ZzxnFjrUYOk5GHDh2pjnvh0qttpOaIno148mjxLyjMVaWKKf
06iDWOwL/xn2PjUKEwtfXI0VxsMxAC4fBIgqw3z1jQ43AYOp/vnPd1s4YxIgoZvj
2rLwBfWH9c2KhYSSHRnc06U9LCmlTV7gxGO4wAuErHs0elvTZV0RA7TQwG/aCNiQ
Q5UN/Lv62nXNAgMBAAGjGjAYMAkGA1UdEwQCMAAwCwYDVR0PBAQDAgXgMA0GCSqG
SIb3DQEBCwUAA4ICAQBtbo+fhOCeaSmbOFn+uyCLdBe1f94EfpE4EokwO+3JCrCj
wtDUJYpNojOUR/6XBjsm671vaKv58lHEyu6nm6bUI2pwN/7bMJ6w+YrqlQGR4AR/
TvVMLKI0upGnfZR8VHrPy4VZFDwRhDxEtBlLXMqFNOlv0DJ3iJJPHy7KfNCA+vLN
1/JrceQBgdFH7j7fDh6H7KgIzNYeFH+fV47LBCcPz84wokEgKEG6WVeqVdOVdJ0J
fIDnCl9h1KtdRTJyMS6WoKUBzFUAw+yonw7kPPSdssEngUQupQ31B6idfg5VOKfM
RDzAIDsd04HhGMJep4pqDjyllhhWMTuhrRtI9424htGOc9CCzZmUlbpX9yON/AdV
yMa6Xb86lyzYbd4uUnF3Ud+H8DTlarxHoyYa01qDLr1EJf/IX/FvnETOgietwp2v
f3p9y5us7q2e3Ghs5CUjwEqtVTW2LrpjUTvrjXAhOH31O3C8HEPhI4fA10tH7RZX
sT/c3+PX+a0lHfl6kiHy7jBjsfzgL7uGNJHowndd69aQdWTDqFoyGw8Vmd7VXn1Y
uZuG7XgTJqlCbpnUCe5+GWKeeoITbOiLdh/uKEr9Xm8oRUg5yM9qAHXJsj4yZDL0
efhcFHO1De/nwW9be0h12fm7Lm9mTkIq7ygL8DfzolTWzO+UrZEFZDPuIzJqtg==
-----END CERTIFICATE-----
```
#### container bind mounted certificate
```
[root@undercloud private]# podman exec -it 8d8991bb489f openssl x509 -in /var/lib/kolla/config_files/src-tls/etc/pki/tls/private/overcloud_endpoint.pem -text
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 4101 (0x1005)
Signature Algorithm: sha256WithRSAEncryption
Issuer: C=US, ST=Texas, L=San Antonio, O=Red Hat, OU=SEC-DFG, CN=undercloud.ooo.test/emailAddress=dwilde@redhat.com
Validity
Not Before: Apr 15 14:39:09 2020 GMT
Not After : Apr 13 14:39:09 2030 GMT
Subject: C=US, ST=Texas, O=Red Hat, OU=SECDFG, CN=192.168.24.2/emailAddress=original-cert@example.com
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
00:a0:f3:3e:c3:38:a5:d9:16:c0:5f:21:6e:ce:aa:
1d:24:64:77:66:80:6b:95:92:31:d3:91:3b:fb:26:
d7:c3:db:53:02:0e:1b:e9:ce:c3:8d:23:3f:3d:4a:
5d:79:5a:87:a8:60:ac:6e:e2:3b:e9:99:79:2e:a3:
fb:4d:05:f0:ce:ef:3c:e2:10:e9:1c:c5:3e:53:bd:
21:21:54:3d:01:58:16:f5:4e:a8:7e:6e:46:40:8b:
f3:e1:7f:04:dd:ef:8c:67:13:86:35:1e:d0:5c:1a:
8d:27:02:82:37:f4:4a:44:2d:c3:79:67:3c:67:16:
3a:d4:60:e9:39:18:70:e1:da:98:e7:be:1d:2a:b6:
da:4e:68:89:e8:d7:8f:26:8f:12:f2:8c:c5:5a:58:
a2:9f:d3:a8:83:58:ec:0b:ff:19:f6:3e:35:0a:13:
0b:5f:5c:8d:15:c6:c3:31:00:2e:1f:04:88:2a:c3:
7c:f5:8d:0e:37:01:83:a9:fe:f9:cf:77:5b:38:63:
12:20:a1:9b:e3:da:b2:f0:05:f5:87:f5:cd:8a:85:
84:92:1d:19:dc:d3:a5:3d:2c:29:a5:4d:5e:e0:c4:
63:b8:c0:0b:84:ac:7b:34:7a:5b:d3:65:5d:11:03:
b4:d0:c0:6f:da:08:d8:90:43:95:0d:fc:bb:fa:da:
75:cd
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Basic Constraints:
CA:FALSE
X509v3 Key Usage:
Digital Signature, Non Repudiation, Key Encipherment
Signature Algorithm: sha256WithRSAEncryption
57:79:56:31:35:f5:3f:fe:c3:cb:70:16:c2:ec:ab:7e:6d:d8:
12:da:96:64:69:1a:cd:ae:76:ed:35:6a:a5:da:2d:2d:09:1f:
68:f8:53:82:a9:71:f8:83:e8:15:98:5e:38:75:5e:41:2d:b2:
24:e9:8b:6f:6e:c2:34:d2:57:b7:80:a0:30:47:4f:42:79:57:
c5:df:70:78:87:9b:37:d7:0b:4b:e8:b6:49:2d:23:5c:03:7b:
6d:c3:3d:8b:d3:25:4e:ed:36:fb:07:1c:48:a8:0d:bc:68:77:
67:42:94:4c:09:df:61:9a:ad:bd:aa:66:eb:cb:85:b0:a6:6e:
82:70:6a:3f:05:43:47:cb:63:b6:98:d3:f1:c5:6a:28:53:3a:
be:b3:70:84:be:a8:fb:92:ea:09:ed:6d:74:99:da:12:43:25:
59:23:2a:59:b3:f9:87:d6:55:df:19:5c:a3:f8:5c:42:79:30:
60:4b:6b:34:8b:6f:56:15:78:e3:e0:f6:d1:ec:bd:83:dd:ef:
4f:a2:5b:15:7e:37:af:05:e9:01:ae:4e:75:cd:c4:f0:24:42:
c9:9c:88:5b:d7:ec:4b:97:38:2d:29:ba:be:0d:6e:0e:8e:7e:
c9:61:c5:93:3d:f6:46:2d:9a:73:6e:2d:2a:3d:46:71:f5:84:
b0:9c:f2:13:9f:aa:bf:98:45:91:36:56:e0:c7:d8:f8:c3:24:
a6:b5:18:23:07:81:60:7d:72:13:9b:b9:06:69:d1:ed:31:ba:
c8:59:15:e4:0f:30:20:82:75:87:1c:0f:de:40:69:6e:a1:e3:
79:57:a7:94:d8:b7:d2:21:09:02:f8:19:cd:54:31:c4:a9:95:
86:aa:3d:27:d6:02:8a:ac:75:f9:b2:39:f6:25:95:0e:60:b2:
2b:72:7c:55:90:de:f6:18:0b:19:54:a7:31:a6:1b:ca:ba:4d:
1b:d7:39:e3:0f:e5:b5:2d:00:f3:53:8b:cb:ce:62:cf:f3:ae:
2a:27:bd:2f:89:9c:21:07:68:f5:49:54:c8:ef:d1:eb:57:f1:
8f:7d:1b:7a:b8:5f:c2:03:8d:f7:44:7c:03:99:5b:2b:78:b4:
9c:fe:df:16:0f:61:2e:70:ab:a9:12:1c:c5:8e:4b:65:3a:d5:
e0:b3:9f:5e:eb:14:b8:9f:10:fe:c1:b5:5e:0a:af:0b:de:55:
a2:b0:b7:ef:bc:b5:f9:92:83:17:b3:9b:24:5b:3f:fa:f5:53:
8f:73:70:6d:05:33:0b:da:15:80:56:20:40:67:1c:42:76:19:
1c:80:ca:6d:d5:5d:94:05:b0:bd:bf:71:6c:e3:24:ee:c4:d3:
f9:38:4d:c4:41:7a:c2:3c
-----BEGIN CERTIFICATE-----
MIIEsDCCApigAwIBAgICEAUwDQYJKoZIhvcNAQELBQAwgZcxCzAJBgNVBAYTAlVT
MQ4wDAYDVQQIDAVUZXhhczEUMBIGA1UEBwwLU2FuIEFudG9uaW8xEDAOBgNVBAoM
B1JlZCBIYXQxEDAOBgNVBAsMB1NFQy1ERkcxHDAaBgNVBAMME3VuZGVyY2xvdWQu
b29vLnRlc3QxIDAeBgkqhkiG9w0BCQEWEWR3aWxkZUByZWRoYXQuY29tMB4XDTIw
MDQxNTE0MzkwOVoXDTMwMDQxMzE0MzkwOVowgYExCzAJBgNVBAYTAlVTMQ4wDAYD
VQQIDAVUZXhhczEQMA4GA1UECgwHUmVkIEhhdDEPMA0GA1UECwwGU0VDREZHMRUw
EwYDVQQDDAwxOTIuMTY4LjI0LjIxKDAmBgkqhkiG9w0BCQEWGW9yaWdpbmFsLWNl
cnRAZXhhbXBsZS5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCg
8z7DOKXZFsBfIW7Oqh0kZHdmgGuVkjHTkTv7JtfD21MCDhvpzsONIz89Sl15Woeo
YKxu4jvpmXkuo/tNBfDO7zziEOkcxT5TvSEhVD0BWBb1Tqh+bkZAi/PhfwTd74xn
E4Y1HtBcGo0nAoI39EpELcN5ZzxnFjrUYOk5GHDh2pjnvh0qttpOaIno148mjxLy
jMVaWKKf06iDWOwL/xn2PjUKEwtfXI0VxsMxAC4fBIgqw3z1jQ43AYOp/vnPd1s4
YxIgoZvj2rLwBfWH9c2KhYSSHRnc06U9LCmlTV7gxGO4wAuErHs0elvTZV0RA7TQ
wG/aCNiQQ5UN/Lv62nXNAgMBAAGjGjAYMAkGA1UdEwQCMAAwCwYDVR0PBAQDAgXg
MA0GCSqGSIb3DQEBCwUAA4ICAQBXeVYxNfU//sPLcBbC7Kt+bdgS2pZkaRrNrnbt
NWql2i0tCR9o+FOCqXH4g+gVmF44dV5BLbIk6YtvbsI00le3gKAwR09CeVfF33B4
h5s31wtL6LZJLSNcA3ttwz2L0yVO7Tb7BxxIqA28aHdnQpRMCd9hmq29qmbry4Ww
pm6CcGo/BUNHy2O2mNPxxWooUzq+s3CEvqj7kuoJ7W10mdoSQyVZIypZs/mH1lXf
GVyj+FxCeTBgS2s0i29WFXjj4PbR7L2D3e9PolsVfjevBekBrk51zcTwJELJnIhb
1+xLlzgtKbq+DW4Ojn7JYcWTPfZGLZpzbi0qPUZx9YSwnPITn6q/mEWRNlbgx9j4
wySmtRgjB4FgfXITm7kGadHtMbrIWRXkDzAggnWHHA/eQGluoeN5V6eU2LfSIQkC
+BnNVDHEqZWGqj0n1gKKrHX5sjn2JZUOYLIrcnxVkN72GAsZVKcxphvKuk0b1znj
D+W1LQDzU4vLzmLP864qJ70viZwhB2j1SVTI79HrV/GPfRt6uF/CA433RHwDmVsr
eLSc/t8WD2EucKupEhzFjktlOtXgs59e6xS4nxD+wbVeCq8L3lWisLfvvLX5koMX
s5skWz/69VOPc3BtBTML2hWAViBAZxxCdhkcgMpt1V2UBbC9v3Fs4yTuxNP5OE3E
QXrCPA==
-----END CERTIFICATE-----
```
## Investigation
The code path that is handling the reload of haproxy is the THT [haproxy-public-tls-certmonger.yaml](https://github.com/openstack/tripleo-heat-templates/blob/stable/train/deployment/haproxy/haproxy-public-tls-certmonger.yaml) which in turn uses the following puppet-tripleo manifest [certmonger/haproxy.pp](https://github.com/openstack/puppet-tripleo/blob/stable/train/manifests/certmonger/haproxy.pp). We can see that `/usr/bin/certmonger-haproxy-refresh.sh` is passed as the `postsave_cmd`.
### Look at `/usr/bin/certmonger-haproxy-refresh.sh`[^2]
```bash
#!/bin/bash
# This script is meant to reload HAProxy when certmonger triggers a certificate
# renewal. It'll concatenate the needed certificates for the PEM file that
# HAProxy reads.
die() { echo "$*" 1>&2 ; exit 1; }
[[ $# -eq 2 ]] || die "Invalid number of arguments"
[[ $1 == @(reload|restart) ]] || die "First argument must be one of 'reload' or 'restart'."
ACTION=$1
NETWORK=$2
certmonger_ca=$(hiera -c /etc/puppet/hiera.yaml certmonger_ca)
container_cli=$(hiera -c /etc/puppet/hiera.yaml container_cli docker)
service_certificate="$(hiera -c /etc/puppet/hiera.yaml tripleo::certmonger::haproxy_dirs::certificate_dir)/overcloud-haproxy-$NETWORK.crt"
service_key="$(hiera -c /etc/puppet/hiera.yaml tripleo::certmonger::haproxy_dirs::key_dir)/overcloud-haproxy-$NETWORK.key"
ca_path=""
if [ "$certmonger_ca" == "local" ]; then
ca_path="/etc/pki/ca-trust/source/anchors/cm-local-ca.pem"
elif [ "$certmonger_ca" == "IPA" ]; then
ca_path="/etc/ipa/ca.crt"
fi
if [ "$NETWORK" != "external" ]; then
service_pem="$(hiera -c /etc/puppet/hiera.yaml tripleo::certmonger::haproxy_dirs::certificate_dir)/overcloud-haproxy-$NETWORK.pem"
else
service_pem="$(hiera -c /etc/puppet/hiera.yaml tripleo::haproxy::service_certificate)"
fi
cat "$service_certificate" "$ca_path" "$service_key" > "$service_pem"
haproxy_container_name=$($container_cli ps --format="{{.Names}}" | grep haproxy)
if [ "$ACTION" == "reload" ]; then
# Copy the new cert from the mount-point to the real path
$container_cli exec "$haproxy_container_name" cp "/var/lib/kolla/config_files/src-tls$service_pem" "$service_pem"
# Set appropriate permissions
$container_cli exec "$haproxy_container_name" chown haproxy:haproxy "$service_pem"
# Trigger a reload for HAProxy to read the new certificates
$container_cli kill --signal HUP "$haproxy_container_name"
elif [ "$ACTION" == "restart" ]; then
# Copying the certificate and permissions will be handled by kolla's start
# script.
$container_cli restart "$haproxy_container_name"
fi
```
#### Conclusion
The issue is seen in the following line in the `"$ACTION" == "reload"` conditional statement:
```bash
$container_cli exec "$haproxy_container_name" cp "/var/lib/kolla/config_files/src-tls$service_pem" "$service_pem"
```
That command is making the false assumption that `/var/lib/kolla/config_files/src-tls` has an up to date copy of what is really on the hosts filesystem. A restart re-mounts the bind mount. Looking at the binds on the container:
```bash
[root@undercloud ~]# podman inspect $(podman ps | awk '/haproxy/ { print $1 }') | jq ".[].HostConfig.Binds"
[
"/etc/hosts:/etc/hosts:ro,rbind,rprivate",
"/etc/pki/tls/private/overcloud_endpoint.pem:/var/lib/kolla/config_files/src-tls//etc/pki/tls/private/overcloud_endpoint.pem:ro,shared,rbind",
"/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro,rbind,rprivate",
"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro,rbind,rprivate",
"/etc/puppet:/etc/puppet:ro,rbind,rprivate",
"/var/lib/kolla/config_files/haproxy.json:/var/lib/kolla/config_files/config.json:ro,rbind,rprivate",
"/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro,rbind,rprivate",
"/etc/localtime:/etc/localtime:ro,rbind,rprivate",
"/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro,rbind,rprivate",
"/dev/log:/dev/log:rbind,rw,rprivate,nosuid",
"/var/lib/haproxy:/var/lib/haproxy:rw,rbind,rprivate",
"/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro,rbind,rprivate",
"/var/lib/config-data/puppet-generated/haproxy:/var/lib/kolla/config_files/src:ro,rbind,rprivate",
"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro,rbind,rprivate"
]
```
We see that the actual `overcloud_endpoint.pem` certificate is bind mounted into the container at the path specified rather than the directory. A bind mount is essentially an inode link and we can see that indeed the two files share the same inode:
```
[root@undercloud ~]# stat /etc/pki/tls/private/overcloud_endpoint.pem
File: ‘/etc/pki/tls/private/overcloud_endpoint.pem’
Size: 7236 Blocks: 16 IO Block: 4096 regular file
Device: fd01h/64769d Inode: 46261016 Links: 1
Access: (0440/-r--r-----) Uid: ( 0/ root) Gid: ( 0/ root)
Context: system_u:object_r:cert_t:s0
Access: 2020-04-15 21:29:15.784456583 +0000
Modify: 2020-04-15 21:31:43.557632361 +0000
Change: 2020-04-15 21:31:43.557632361 +0000
Birth: -
[root@undercloud ~]# podman exec -it $(podman ps | awk '/haproxy/ { print $1 }') stat /var/lib/kolla/config_files/src-tls//etc/pki/tls/private/overcloud_endpoint.pem
File: ‘/var/lib/kolla/config_files/src-tls//etc/pki/tls/private/overcloud_endpoint.pem’
Size: 7236 Blocks: 16 IO Block: 4096 regular file
Device: fd01h/64769d Inode: 46261016 Links: 1
Access: (0440/-r--r-----) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2020-04-15 21:29:15.784456583 +0000
Modify: 2020-04-15 21:31:43.557632361 +0000
Change: 2020-04-15 21:31:43.557632361 +0000
Birth: -
```
After generating a new certificate and re-running the `openstack undercloud install` but not restarting the haproxy container we can see that the reference inside of the container is pointing at the previous inode:
```
(undercloud) [stack@undercloud ~]$ sudo stat /etc/pki/tls/private/overcloud_endpoint.pem
File: ‘/etc/pki/tls/private/overcloud_endpoint.pem’
Size: 3381 Blocks: 8 IO Block: 4096 regular file
Device: fd01h/64769d Inode: 71375612 Links: 1
Access: (0440/-r--r-----) Uid: ( 0/ root) Gid: ( 0/ root)
Context: system_u:object_r:cert_t:s0
Access: 2020-04-16 18:03:57.123790389 +0000
Modify: 2020-04-16 18:03:57.011785706 +0000
Change: 2020-04-16 18:03:57.329799001 +0000
Birth: -
(undercloud) [stack@undercloud ~]$ sudo podman exec -it $(sudo podman ps | awk '/haproxy/ { print $1}') stat /var/lib/kolla/config_files/src-tls//etc/pki/tls/private/overcloud_endpoint.pem
File: ‘/var/lib/kolla/config_files/src-tls//etc/pki/tls/private/overcloud_endpoint.pem’
Size: 7236 Blocks: 16 IO Block: 4096 regular file
Device: fd01h/64769d Inode: 46261016 Links: 0
Access: (0440/-r--r-----) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2020-04-16 18:03:54.787692730 +0000
Modify: 2020-04-15 21:31:43.557632361 +0000
Change: 2020-04-16 18:03:57.123790389 +0000
Birth: -
```
#### Remediation
I think that the issue lies ultimately with the bind mount inside of the haproxy container, the reason that an individual file is linked is probably that other sensitive files can live in `/etc/pki/tls/private` and haproxy should only have access to the certificates that it needs. I did notice that there is and `/etc/pki/tls/private/haproxy` directory that is created that could possibly hold all of the haproxy related certificates and keys.
The workaround is to issue the `restart` action to `certmonger-haproxy-refresh.sh` rather than the `reload` action, this will re-generate the bind mounted file with the updated inode. On master the container is restarted so this is not an issue.
[^1]: https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/13/html/director_installation_and_usage/appe-ssltls_certificate_configuration
[^2]: This is the actual file from my stable/train undercloud host. The source is found [here](https://github.com/openstack/puppet-tripleo/blob/stable/train/files/certmonger-haproxy-refresh.sh).
Google Drive
Gist
Clipboard
Sign in with Wallet
