4.8. High availability

No HA out of the box

Central Management does not support high availability out-of-the-box. This example shows, however, how to set Central Management up for high availability using 3rd party tools.

Cold swap example

In this example we will use four servers installed with CentOS-7-x86_64-Minimal-1804.iso as OS and a managed instance to demonstrate a cold swap high availability setup. The network setup will be the following:

images/inline/be0d71148ce377f32e5ad6712e329f9fff411df5.png

Use nmtui to configure static IP addresses and static routes if needed.

Infrastructure

To build the infrastructure, perform the following steps:

  1. Install CentOS on all the four machines (shared storage, Central Management A (active), Central Management B (spare), network address translator)

  2. Enable networking on each

    1. # nmtui
    2. images/download/attachments/36838044/image2018-7-11_8-51-46.png
    3. images/download/attachments/36838044/image2018-7-11_8-52-28.png
    4. images/download/attachments/36838044/image2018-7-11_8-52-56.png

    5. Quit nmtui

    6. # service network restart

Shared storage

The shared storage will use Network File System (NFS) to share a directory where the data files will be stored for both the active and the spare Central Management servers.

Perform the following steps on the shared storage server:

  1. Install NFS utils

    # yum install nfs-utils

  2. Enable services

    # systemctl enable rpcbind
    # systemctl enable nfs-server
    # systemctl enable nfs-lock
    # systemctl enable nfs-idmap
    # systemctl start rpcbind
    # systemctl start nfs-server
    # systemctl start nfs-lock
    # systemctl start nfs-idmap

  3. Create the directory to be shared

    # mkdir /var/nfsshare
    # chmod -R 755 /var/nfsshare
    # chown nfsnobody:nfsnobody /var/nfsshare

  4. Share the directory

    # vi /etc/exports
     
    /var/nfsshare *(rw,sync,no_acl,no_root_squash,no_all_squash)

  5. Restart NFS server

    # systemctl restart nfs-server

  6. Enable NFS on the firewall

    # firewall-cmd --permanent --zone=public --add-service=nfs
    # firewall-cmd --permanent --zone=public --add-service=mountd
    # firewall-cmd --permanent --zone=public --add-service=rpc-bind
    # firewall-cmd --reload

Active Central Management node

The active Central Management server will use Network File System (NFS) to access the shared directory where the data files will be stored.

Perform the following steps on the active Central Management server:

  1. Install NFS utils

    # yum install nfs-utils

  2. Install Central Management

    # yum install mdcentralmgmt-5.2.0-1.x86_64.rpm

  3. Stop the service

    # service mdcentralmgmt stop

  4. Bind the data directory to the share

    1. Back up the contents of the installation data directory

      # cp -R /var/lib/mdcentralmgmt /root/backup

    2. Clear the contents of the installation data directory

      # rm -rf /var/lib/mdcentralmgmt/*

    3. Mount the shared directory on the data directory

      # vi /etc/fstab
       
      10.0.2.10:/var/nfsshare /var/lib/mdcentralmgmt nfs defaults 0 0
       
      # mount -a

    4. Restore the contents of the data directory

      cp -R /root/backup/mdcentralmgmt/* /var/lib/mdcentralmgmt

  5. Restart the service

    # service mdcentralmgmt start

  6. Record the UID and GID of the centralmgmt user and group (see Sync the UID and GID of the centralmgmt user and group with the active node)

    # id -u centralmgmt
    998
    #id -g centralmgmt
    996

Spare Central Management node

The spare Central Management server will use Network File System (NFS) to access the shared directory where the data files will be stored.

Perform the following steps on the active Central Management server:

  1. Install NFS utils

    # yum install nfs-utils

  2. Install Central Management

    # yum install mdcentralmgmt-5.2.0-1.x86_64.rpm

  3. Stop the service

    # service mdcentralmgmt stop

  4. Bind the data directory to the share

    1. Clear the contents of the installation data directory

      # rm -rf /var/lib/mdcentralmgmt/*

    2. Mount the shared directory on the data directory

      # vi /etc/fstab
       
      10.0.2.10:/var/nfsshare /var/lib/mdcentralmgmt nfs defaults 0 0
       
      # mount -a

  5. Sync the UID and GID of the centralmgmt user and group with the active node (see Record the UID and GID of the centralmgmt user and group)

    1. Record the current UID and GID values

      # id -u centralmgmt
      1001
      # id -g centralmgmt
      1002

    2. Set the UID and GID values to the same as on the active node

      # usermod -u 998 centralmgmt
      # groupmod -g 996 centralmgmt

    3. Set the new UID and GID values for all files and directories that must be owned by the centralmgmt user or group

      # find / -user 1001 -exec chown -h centralmgmt {} \;
      # find / -group 1002 -exec chgrp -h centralmgmt {} \;

  6. Remember to not (re)start the mdcentralmgmt service on the spare Central Management node as it may cause data inconsistency.

Network address translator

The network address translator server will provide the shared IP address for both the active and the spare Central Management nodes. This will ensure that the managed instances will always see the same IP address for the Central Management server either it is the active or the spare server that is doing the actual management.

Perform the following steps on the network address translator server:

  1. Enable packet forwarding

    # sysctl -w net.ipv4.ip_forward=1

    1. Persist packet forwarding

      # vi /etc/sysctl.conf
       
      net.ipv4.ip_forward = 1

  2. Assign the network interfaces to the firewall zones accordingly

    # firewall-cmd --zone=external --add-interface=eth0 --permanent
    # firewall-cmd --zone=internal --add-interface=eth1 --permanent

  3. Configure masquerading on the externally facing interface

    # firewall-cmd --zone=external --add-masquerade --permanent

  4. Add the NAT rule

    # firewall-cmd --permanent --direct --passthrough ipv4 -t nat -I POSTROUTING -o eth0 -j MASQUERADE -s 10.0.2.0/24

  5. Enable instance REST ports (e.g. 8008) so that the managed instances' REST interfaces can be accessible

    # firewall-cmd --permanent --zone=internal --add-port=8008/tcp

  6. Reload and verify the firewall configuration

    # firewall-cmd --complete-reload
    # firewall-cmd --list-all-zones

Switch-over

  1. Stop the active Central Management on the active Central Management server

    # service mdcentralmgmt stop

  2. On the spare Central Management server

    1. Start the spare Central Management service

      # service mdcentralmgmt start

    2. Activate the license and check if managed instances appear right.