How To Move Service From One Host To Another In Hdp

The upgrade process from HDP 2.half-dozen.5 to CDP private Cloud Base 7.one.6

Cloudera announced a new version of CDP that allows customers to upgrade from their legacy HDP2.6 clusters and drive even more value from their data by gaining access to new features and functionality to enhance their large data experience.

We, at Clairvoyant, get through efficient and strategic planning before starting the upgrade process. This helps us to get the upgrade completed smoothly and with reduced downtime. Beneath, we have listed a step-past-step procedure for the upgrade. For all your customized requests, achieve out to united states here.

This upgrade requires several pregnant steps, including upgrading to an acting version of Ambari. Subsequently the upgrade, our cluster will be managed by Cloudera Director, and the components will be upgraded to Cloudera Runtime vii.1.6.

Source Version: Ambari ii.6.2.ii and HDP two.6.5
Target Version: CM seven.three.1 and CDP 7.1.6

The high-level footstep of the upgrade procedure is to transition from the Ambari to Cloudera Director seven.1.six using the AM2CM tool. It includes the following steps:

HDP cluster upgrade from version 2.6.5 to seven.one.half-dozen.
HDP cluster pattern is exported from Ambari and converted to a Cloudera Manager deployment template using the AM2CM tool.
Deploy the template to Cloudera Managing director 7.one.6 and activate the Cloudera Runtime 7.1.vi parcels.
Start using the CDP Individual Deject Base cluster with Cloudera Director 7.1.half-dozen.
Uninstall Ambari and the HDP stack.

Compages diagram for the upgrade and the transition

Below are the steps to upgrade the Hadoop cluster from HDP 2.6.five to CDP 7.1.vi.

1. Environment requirement and support versions

Ane of the significant challenges while upgrading to a major release is to ensure that the environment has the supported Hardware, Operating System, Java version, and Database requirements. The steps in the link will help you to get compatibility details.

2. Pre-Upgrade steps

Ensure all services in the cluster are running. Run each Service Check (found under the Service Actions carte) and confirm they executed successfully.
Follow the link to take a backup of Ambari.
Nosotros must configure and review Kerberos as per the link.

Encrypt Database and LDAP Passwords

Remove the components that are unsupported in CDP Individual Deject Base. For more than information, see the supported component.
Hive prerequisites to handle missing table or partition locations and correct invalid SERDE definitions.
Cease SmartSense and Log Search service from the cluster.
Nosotros must non go on any services in "Maintenance mode" except for Ambari Metrics Organisation, SmartSense, and Log Search.
Plough off the "Auto Get-go Services."

Automobile Start service disabled

Download, Extract and Run the Ambari Metrics pre-upgrade script. Login to Metrics Collector host and run below command,

The output of the ams pre-upgrade script

3. Upgrading Ambari from 2.6.2.2 to 7.1.half-dozen

Stop the Ambari Server - On the host running Ambari Server.
Stop the Ambari Agents - On each host in your cluster running an Ambari Agent.
Download the Ambari Repo ambaridc.repo from https://annal.cloudera.com/p/ambaridc/7.x/7.1.6.0/ and copy it inside /etc/yum.repos.d/ambaridc.repo .
Open the ambaridc.repo file and update the username and password of baseurl and gpgkey.
Bank check the current and available Ambari Server version with the control

# yum info ambari-server

Installed and Available Ambari Server bundle

Once nosotros see the desired version in Available Packages, we tin upgrade Ambari Server.

# yum upgrade ambari-server

# yum info ambari-server

Ambari Server with 7.ane.6 version

Now copy the ambaridc.repo to all the Cluster hosts under /etc/yum.repos.d/and upgrade Ambari Agent.

# yum upgrade ambari-agent

# yum info ambari-agent

Ambari Amanuensis with 7.1.six version

Ambari Server database schema Upgrade - Login to Ambari server node and run beneath command.

# ambari-server upgrade

Ambari server database upgrade done successfully

Command to get the JDBC driver jar file in place,

# ambari-server setup — jdbc-db=mysql — jdbc-driver=/usr/share/java/mysql-connectorjava.jar

At present start the Ambari server and Ambari agents.
Log in to the Ambari UI with admin credentials.

Ambari interim UI

Ambari successfully upgraded to the 7.1.6 version

Till now, nosotros have successfully upgraded Ambari to version 7.1.6. Exercise not manually restart any HDP services unless future steps in the upgrade guide prompt usa to practice then. Manually restarting these services may significantly disrupt our upgrade. Ambari will restart each service automatically during the HDP upgrade.

4. Upgrading HDP from 2.half dozen.5 to 7.1.vi

Ensure all services in the cluster are running. Run each Service Check and confirm they execute successfully.
Follow the link to take a backup of the HDP cluster.
We can see in "Stack and Versions" only HDP 2.6.5.
Register software repositories for HDP seven.1.six by going to the "MANAGE VERSIONS" and then "REGISTER VERSION" with HDP 7.1.half-dozen.0

Annals new HDP version 7.i.six.0

Click on "INSTALL PACKAGES". This volition install all HDP vii.1.6 packages.

Install the HDP 7.1.six package

The "Upgrade" button replaces the "Install Packages" button when the installation completes.

Upgrade button nether HDP 7.1.6 is visible

Earlier going ahead with the Upgrade, there are some pre-upgrade steps to exist performed.
Follow the link for preparing Hive for an upgrade. This needs to be followed advisedly as nosotros are moving to Hive 3.
Follow the link for upgrading the Ambari Metrics system and SmartSense. Practise not start the Ambari Metrics System service. Information technology volition start automatically during the HDP upgrade process.
Let's complete the pre-upgrade cheque earlier going for an HDP seven upgrade. This will cheque if nosotros missed any requirements. If any, nosotros need to get it cleared.

Click Pre-Upgrade Check to review the requirement

At present click the "UPGRADE" push button under HDP 7.1.6 and become alee with the Express upgrade.

Click Express Upgrade

We tin see the upgrade process has started. Some of the stages require that we complete an action during normal operation. If any stage fails, the upgrade stops and prompts united states for action.
Once the upgrade is completed, we can Finalize the upgrade or to Finalize Later. Finalizing later gives you a take chances to perform more validation on the cluster. Click Finalize to complete the limited upgrade process.

Upgrade process finished

Now that the HDP Upgrade to the 7.1.half dozen version is finished, we need to follow the link for the mail service HDP upgrade task.
We have successfully completed Ambari and HDP upgrade to interim version seven.1.half dozen. Now permit us plan for transitioning to Cloudera Manager.

5. Setup of new Cloudera Manager 7.three.1

Follow the link to download the cloudera-manager.repo. Copy the repo to all the hosts.
Install Cloudera Manager and Agent on CM host.

# yum install cloudera-manager-daemons cloudera-manager-agent cloudera-director-server
Once Cloudera packages are installed, follow the link to Install and Configure MySQL for Cloudera Software.
We should configure the Cloudera Manager to point to the MySQL database with below control,

#sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm -v
At present let us start the Cloudera Manager server.

# service cloudera-scm-server restart
Login to the CM UI HTTP://<CM_HOST>:7180 and we can run into the below screen. Upload the License.

CM UI after the first login

From the UI, install Cloudera Agent and so add Cloudera Management Services.

Notation: Do not set up a cluster using the Wizard.
Fix Kerberos - If we have a Kerberos cluster, then we must add together the KDC details in the Administration>Security>Kerberos Credentials>Setup KDC for the Cloudera Director folio using Cloudera Manager.

vi. Transition from HDP acting version 7.1.half dozen to Cloudera Manager using the AM2CM tool

In the new tab, enter the URL and save the blueprint in JSON format:

                  http://<ambari_ip>:<port>/api/v1/clusters/<cluster-name>?
format=blueprint_with_hosts

Download the AM2CM tool and excerpt the tar.

# wget https://archive.cloudera.com/am2cm/i.x/am2cm-one.1.1.0-1.tar.gz

# tar -zxvf am2cm-1.1.1.0–1.tar.gz
Navigate to the am2cm-1.i.ane.0–1/conf/user-settings.ini file and update Parcels, Cluster name, passwords, and JDBC URL information.
Generate the Cloudera Manager Deployment template from the HDP blueprint.

# cd am2cm-1.i.ane.0–ane

# chmod +x ./am2cm.sh

# ./am2cm-one.ane.1.0–1/am2cm.sh -bp

HDP_blueprint_with_hosts_20210519142803.json
-dt cm_deployment_template.json

Output after successfully generating the CM deployment templete

Stop HDP services from Ambari UI

Stop All of the HDP services

Once all the services are stopped successfully, import the Cloudera Manager deployment template using the API through CLI.

# scroll - -user admin:admin -k -10 PUT -H "Content-Type: application/json" -d @cm_deployment_template.json 'http://<CM_HOST>:7180/api/v41/cm/deployment?deleteCurrentDeployment=simulated'
Nosotros can see a screen as below in CM UI. But many other errors and warnings demand to exist fixed.

CM UI later on importing the CM template

In the Cloudera Director Package screen, download Cloudera Runtime and distribute the parcel.

Download and distribute the CR seven.i.vi package to all hosts

After the parcels are deployed on Cloudera Manager, activate the Cloudera Runtime 7.1.x parcels.

Activate the CR vii.1.6 bundle

We demand to generate all the keytabs in Cloudera Managing director under the Security section ->Kerberos Credentials -> Generate Missing Credentials.

Generating Credentials for all the services

At that place are some major mail-transition steps that need to be followed which are mentioned in the link.
Afterwards the post-transition and troubleshooting of the errors, we get the services are in the green state.

Cluster with all services green

Time for running a smoke test on all services to verify all looks skillful.

Finally, we have successfully upgraded HDP to CDP private base of operations. To learn more well-nigh upgraading Cloudera CDH cluster to CDP Private Cloud Base, read our blog post.

For all your Cloud based services requirements, achieve out to us at Clairvoyant.