RHEV: How to recreate RHEV-M CA and update RHEV setup with the new certificate (migration from 2.1 only)
 SOLUTION 확인됨 - 업데이트됨05시 38분 2014년 7월 25일 - English 환경
Red Hat Enterprise Virtualization (RHEV)
Migration Tool for RHEV 2.2->3.0 migration
RHEV-M 2.2, originally upgraded from 2.1
RHEL 6 hypervisors
문제
Use the information in this this article only if all of the following apply to your RHEV environment:

You used a migration tool to upgrade your RHEV2.2 environment to RHEV 3.0
and
You started using RHEV2.1 and upgraded your environment from 2.1 to 2.2
In this case you will have troubles upgrading your environment to full 3.0 compatibility mode cluster and won't be able to attach RHEL6 based host to your system. Check Diagnostic Steps section below for more details.

해결
This resolution contains 3 steps to cover all the involved entities: RHEV-M, Hosts, Client accessing Admin/User Portal via https:

I. On RHEV-M: - recreate RHEV-M CA; create and sign RHEV-M public-private key pair; update JBoss EAP with new certificates

II. Each host - re-register to RHEV-M

III. Import the new RHEV-M CA and public key on the clients used to access Admin / User Portal

Here are the detailed steps:
Note: type/copy the commands as is, just replace the IPs setup and DN values to match your setup.

I. On RHEV-M 3.0 perform the following steps:
0. Make a backup copy of the /etc/pki/rhevm directory.

Raw
# cp -pr /etc/pki/rhevm /var/tmp/

1. Reset the certificate database and serial number:

Raw
# cd /etc/pki/rhevm
# rm database.txt*
# rm serial.txt*
# touch database.txt
# echo 01 > serial.txt
# rm -rf certs/*
# rm -rf requests/*

2. Regenerate the CA certificate signing request preserving the existing private key:

Raw
# openssl req \
-new \
-key private/ca.pem \
-config cacert.conf \
-out requests/ca.csr

3. Sign again the CA certificate, adding the required X509v3 extensions:

Raw
# openssl ca \
-selfsign \
-out ca.pem \
-in requests/ca.csr \
-keyfile private/ca.pem \
-days 3650 \
-config openssl.conf \
-extfile cacert.conf \
-extensions v3_ca \
-startdate `date -d yesterday +%y%m%d%H%M%S+0000`

Verify visually that the generated ca.pem file contains the X509v3 extensions, something like this:

Raw
  X509v3 extensions:
    X509v3 Subject Key Identifier:
      45:31:B7:BF:3C:D3:D3:7C:03:2F:6B:A6:F7:03:B8:F4:C0:34:63:D9
    Authority Information Access:
      CA Issuers - URI:http://rhevm.example.com:8080/ca.crt

    X509v3 Authority Key Identifier:
      keyid:45:31:B7:BF:3C:D3:D3:7C:03:2F:6B:A6:F7:03:B8:F4:C0:34:63:D9
      DirName:/C=US/O=Example Inc./CN=RHEVM CA
      serial:00

    X509v3 Basic Constraints: critical
      CA:TRUE
      X509v3 Key Usage: critical
      Certificate Sign, CRL Sign

4. Generate a DER version of the CA certificate:

Raw
# openssl x509 \
-in ca.pem \
-out certs/ca.der \
-outform der

5. Regenerate the RHEV-M certificate signing request preserving the existing private key (this is very important in order to avoid having to decrypt/encrypt passwords stored in the database):

Important - replace RHEVM-FQDN and Company Name below
Raw
# openssl req \
-new \
-subj '/C=US/O=Example Inc./CN=RHEVM-FQDN' \
-key keys/rhevm_id_rsa \
-out requests/rhevm.req \
-passin pass:mypass

Note that the -subj option should be the DN of the RHEV-M certificate currently in use.

6. Sign again the RHEV-M certificate, to simplify this the
SignReq.sh script should be used:

Raw
# ./SignReq.sh \
rhevm.req \
rhevm.cer \
1800 \
/etc/pki/rhevm \
`date -d yesterday +%y%m%d%H%M%S+0000` \
mypass

Double check that the generated certificate is correct, visually and with the following command:

Raw
# openssl verify -CAfile ca.pem certs/rhevm.cer 

7. Generate also a DER encoded version of the RHEV-M certificate:

Raw
# openssl x509 \
-in certs/rhevm.cer \
-out certs/rhevm.der \
-outform DER

8. Export the RHEV-M private key and certificate to a PKCS12 file:

Raw
# openssl pkcs12 \
-export \
-name rhevm \
-inkey keys/rhevm_id_rsa \
-in certs/rhevm.cer \
-passin pass:mypass \
-out certs/rhevm.pfx \
-passout pass:mypass

9. Regenerate the keystore used by the engine:

Raw
# rm .keystore

# keytool \
-keystore .keystore \
-importkeystore \
-srckeystore certs/rhevm.pfx \
-srcalias rhevm \
-srcstoretype PKCS12 \
-srcstorepass mypass \
-destalias rhevm \
-deststorepass mypass

10. Add the CA certificate to the keystore used by the engine:

Raw
# keytool \
-keystore .keystore \
-storepass mypass \
-import \
-alias cacert \
-file certs/ca.der \
-noprompt

11. Regenerate the truststore used by the engine:

Raw
# rm .truststore

keytool \
-keystore .truststore \
-storepass mypass \
-import \
-alias cacert \
-file certs/ca.der \
-noprompt

12. Copy the new CA certificate to the root web application:

Raw
# cp /var/lib/jbossas/server/rhevm-slimmed/deploy/ROOT.war/ca.crt /var/tmp/
# cp ca.pem /var/lib/jbossas/server/rhevm-slimmed/deploy/ROOT.war/ca.crt

Note: this step is not required for 3.1 environments, since it is using the default location under /etc/pki.

13. Restart jboss EAP to refresh the certificate on it:

Raw
# service jbossas restart

14. Verify RHEV-M operates ok and Admin Portal is accessible.

back to top

II. Host side:
After replacing the certificates on RHEV-M side, it is time to update the hosts.
The best and easiest way to do it is to re-register the host to RHEV-M, so that the registration process will take care of all the certificates recreation.

It is recommended to wipe out the host and reinstall it from scratch before re-registering it to RHEV-M, so that not to have wrong certificates leftovers. Refer to 'How to prepare a RHEV Hypervisor system for re-registration' for details.

Here is the procedure to manually update the host with RHEV-M certificates, but you should not have to follow it and registration process should be enough.
Note: if it is RHEV-H host, you would have to unpersist /etc/pki before the procedure, and persist it back after the end.

1. Backup current certificates on the host. If RHEV-H, copy them to another machine, if RHEL, copy them to /var/tmp:

Raw
/etc/pki/vdsm/certs/cacert.pem
/etc/pki/vdsm/certs/vdsmcert.pem
/etc/pki/vdsm/libvirt-spice/ca-cert.pem
/etc/pki/vdsm/libvirt-spice/server-cert.pem

2. Create a config file named host_ip.conf (192.168.200.101.conf, for example) for openssl:

Raw
# cd /tmp
# touch 192.168.200.101.conf

with content similar to this:

Raw
RANDFILE = ~/.rnd
[ req ]
distinguished_name = req_distinguished_name
prompt = no
[ v3_ca ]
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid:always,issuer:always
basicConstraints = CA:true
[ req_distinguished_name ]
O = Example Inc.
CN = 192.168.200.101

3. Create a certificate signing request using the existing VDSM private key and the configuration file:

Raw
# openssl \
req \
-new \
-key \
/etc/pki/vdsm/keys/vdsmkey.pem \
-out /tmp/192.168.200.101.req \
-config 192.168.200.101.conf

4. Copy the resulting CSR file to RHEV-M, to the /etc/pki/rhevm/requests directory:

Raw
# scp /tmp/192.168.200.101.req root@rhevm-fqdn:/etc/pki/rhevm/requests/

5. In RHEV-M sign the CSR using the SingReq.sh script:

Raw
# cd /etc/pki/rhevm

# ./SignReq.sh \
192.168.200.101.req \
192.168.200.101.cer \
1800 \
/etc/pki/rhevm \
`date -d yesterday +%y%m%d%H%M%S+0000` \
mypass

Verify the generated certificate visually and with the following command:

Raw
# less certs/192.168.200.101.cer
# openssl verify -CAfile ca.pem certs/192.168.200.101.cer
certs/192.168.200.101.cer: OK

6. Replace the CA certificate and the VDSM certificates in the hypervisor with the new ones:

Raw
# cd /etc/pki/rhevm

# scp ca.pem root@192.168.200.101:/etc/pki/vdsm/certs/cacert.pem
# scp ca.pem root@192.168.200.101:/etc/pki/vdsm/libvirt-spice/ca-cert.pem

# scp certs/192.168.200.101.cer root@192.168.200.101:/etc/pki/vdsm/certs/vdsmcert.pem
# scp certs/192.168.200.101.cer root@192.168.200.101:/etc/pki/vdsm/libvirt-spice/server-cert.pem

Verify that the owner and group of the '/etc/pki/vdsm/certs/cacert.pem' and '/etc/pki/vdsm/certs/vdsmcert.pem' files is set to 'vdsm' and 'kvm' respectively, e.g.

Raw
# ls -al
total 10
drwxr-xr-x. 2 vdsm kvm   80 2012-10-11 14:10 .
drwxr-xr-x. 5 vdsm kvm  100 2012-09-27 19:11 ..
-r--r--r--. 1 vdsm kvm 3454 2012-10-11 14:10 cacert.pem
-r--r--r--. 1 vdsm kvm 3574 2012-10-11 14:10 vdsmcert.pem
#

If they are not, then this can be resolved by executing;

Raw
# chown vdsm:kvm /etc/pki/vdsm/certs/cacert.pem
# chown vdsm:kvm /etc/pki/vdsm/certs/vdsmcert.pem

7. On the hypervisor:

Raw
# initctl stop libvirtd
# initctl start libvirtd
# service vdsmd restart 

8. Activate the host on the Admin Portal. It should be green and responsive.

back to top

III. The procedure to update the client with new CA is described here:
RHEV: How to reinforce RHEV-M CA and ssl keys installation for https access on a client?

back to top

근본 원인
The RHEV migration tool, among other tasks, copies RHEV-M certificates from the old 2.2 environment to a new 3.0 environment in order to maintain connectivity with 2.2 hosts attached to the system. If the environment was originally migrated from 2.1, then it will preserve the old certificate that does not meet the format required by libvirt on RHEL6. Host registration process will start, libvirt will get vdsm.cert signed by RHEV-M CA, but will fail to start.

How do you know if your RHEV-M CA certificate in the old unsupported format?
On Windows machine go to RHEV Manager installation folder and navigate to:
\RhevManager\service\ca\
- Windows2003: certs\ca.pem
- Windows2008: ca.pem

Old format will not contain all the required 509v3 extensions:

Raw
        X509v3 extensions:
            Authority Information Access: 
                CA Issuers - URI:http://rhevm.gsslab.redhat.com/ca.crt

Compared to the required format:

Raw
       X509v3 extensions:
            X509v3 Subject Key Identifier:
                01:05:F1:EF:5C:F7:FB:D7:2A:F5:2E:51:AC:7B:9A:1A:B7:B3:E1:66
            Authority Information Access:
                CA Issuers - URI:http://rhevm.gsslab.redhat.com:8080/ca.crt

            X509v3 Authority Key Identifier:
                keyid:01:05:F1:EF:5C:F7:FB:D7:2A:F5:2E:51:AC:7B:9A:1A:B7:B3:E1:66
                DirName:/C=US/O=RedHat/CN=RHEVM CA
                serial:01

            X509v3 Basic Constraints: critical
                CA:TRUE
            X509v3 Key Usage: critical
                Certificate Sign, CRL Sign

Note: There is a private Bugzilla #827361 about RHEV-M 2.2 CA that does not contain all the X509v3 extensions required by RHEL 6.

진단 단계
RHEL6 based host registration will never end and Admin Portal will show the host in Installing stage forever.

Check /var/log/rhevm/rhevm.log:

Raw
2012-06-07 20:01:01,443 INFO  [org.ovirt.engine.core.bll.CBCInstaller] (pool-19-thread-49) Installation of 10.12.52.144. Executing oVirt ins
tallation stage. (Stage: Sign certificate request and generate certificate)
2012-06-07 20:01:02,462 ERROR [org.ovirt.engine.core.bll.ApproveVdsCommand] (pool-19-thread-49) Command org.ovirt.engine.core.bll.ApproveVds
Command throw exception java.lang.NullPointerException

Check /var/lib/libvird.log

Raw
2012-06-07 23:55:46.298+0000: 7354: info : libvirt version: 0.9.4, package: 23.el6_2.8 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2012-04-09-04:06:12, x86-003.build.bos.redhat.com)
2012-06-07 23:55:46.298+0000: 7354: error : virNetTLSContextCheckCertPair:483 : Our own certificate /etc/pki/vdsm/certs/vdsmcert.pem failed validation against /etc/pki/vdsm/certs/cacert.pem: The certificate is not trusted.

Verify vdsm certificate:

Raw
# cd /etc/pki/vdsm
# openssl verify -CAfile cacert.pem vdsmcert.pem 
vdsmcert.pem: O = RedHat, CN = 10.12.52.144
error 7 at 0 depth lookup:certificate signature failure
140513173755720:error:04067084:rsa routines:RSA_EAY_PUBLIC_DECRYPT:data too large for modulus:rsa_eay.c:742:
140513173755720:error:0D0C5006:asn1 encoding routines:ASN1_item_verify:EVP lib:a_verify.c:184:

It is required to clean the status of the host in the DB and remove it from the setup, before proceeding with the procedure.
To update host status on the DB, run the command on RHEV-M server:

Raw
# psql rhevm postgres -c "update vm_dynamic set status='0' where vm_host='Host_name'"