Installing R, RHadoop and RStudio over Cloudera Hadoop ecosystem (Revised)

The following is a step-by-step procedure of installing R, RStudio and RHadoop over a Cloudera hadoop system. OS is CentOS 6.5.

Step 1:

If you are behind proxy, as root user, set up environmental variable for proxy server, as:
# export http_proxy=http://192.168.1.254:6588
(http://proxy-server-ip:proxy-serverPort)
And in /etc/yum.conf, write one line as:
proxy=http://192.168.1.254:6588

Recheck yout CentOS version, if required
$  cat /etc/redhat-release
CentOS release 6.5 (Final)

Step 2:
Update your OS
# yum -y update
And, add EPEL repository to yum configuration
# rpm -ivh http://mirror.chpc.utah.edu/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm

(Note that rpmforge has an older version of R. Disable rpmforge, if active
by opening file /etc/yum.repos.d/rpmforge.repo and changing the line
enabled = 1
to
enabled = 0 )

Step 3

Install R & R-devel
# yum install R R-devel

Start R shell and issue commands within it to download packages  as below
(note that install.packages() line splits over many lines even though it is one
single line till the last closing brackets(s))

# R

(If your are behind proxy, set the proxyserver address and port in R-shell as follows before the next command)
> Sys.setenv(http_proxy=”http://192.168.1.254:6588″ )
(If you find strange characters here, the format is:   http://proxy-server:proxyserverPort enclosed within double inverted commas)
> install.packages(c(“rJava”, “Rcpp”, “RJSONIO”, “bitops”, “digest”, “functional”, “stringr”, “plyr”, “reshape2”))
> q()

Step 4
Download rhdfs and rmr2 packages to your local Download folder from here:

Step 5
Set up environmental variables. Write the following three lines in file  ~/.bashrc and /etc/profile

export HADOOP_HOME=/usr/lib/hadoop-0.20-mapreduce
export HADOOP_CMD=/usr/bin/hadoop
export HADOOP_STREAMING=/usr/lib/hadoop-0.20-mapreduce/contrib/streaming/hadoop-streaming-2.0.0-mr1-cdh4.5.0.jar

(Note: Depending upon the Cloudera version you have downloaded, check the version of jar file above and change it accordingly)

After writing these lines, export the variables as:
# source /etc/profile
$ source ~/.bashrc

Step 6
Install the downloaded rhdfs & rmr2 packages in R-shell by specifying its location in your machine. First, start R:
# R

Then, install packages:
> install.packages(“/home/ashokharnal/Downloads/rhdfs_1.0.8.tar.gz”, repos = NULL, type=”source”)
> install.packages(“/home/ashokharnal/Downloads/rmr2_2.3.0.tar.gz”, repos = NULL, type=”source”)

At this point R and RHadoop are installed

Step 7
Install RStudio as (my machine has 64 bit-OS. Check yours before install)
# wget http://download2.rstudio.org/rstudio-server-0.98.490-x86_64.rpm
# yum install –nogpgcheck rstudio-server-0.98.490-x86_64.rpm

Note: RStudio service can be started/stopped as:

# service rstudio-server stop
# service rstudio-server status
# service rstudio-server start

The RStudio server is accessible at: http://localhost:8787
Step 8

Restart your machine
Step 9

Log in as non-root (local) user and create a file ~/.Rprofile. Write in it the following seven lines and save the file:

Sys.setenv(HADOOP_HOME=”/usr/lib/hadoop-0.20-mapreduce”)
Sys.setenv(HADOOP_CMD=”/usr/bin/hadoop”)
Sys.setenv(HADOOP_STREAMING=”/usr/lib/hadoop-0.20-mapreduce/contrib/streaming/hadoop-streaming-2.0.0-mr1-cdh4.5.0.jar”)
library(rmr2)
library(rJava)
library(rhdfs)
hdfs.init()

The file ~/.Rprofile contains commands/environmental variables that need to be set/executed whenever
R starts. This file is picked up by R, as soon as it starts.

Step 10
Check hadoop-mapreduce capability in RHadoop as follows. Start R-shell as local user
and issue the following three commands in the shell, one after another.

> ints = to.dfs(1:100)
> calc = mapreduce(input = ints, map = function(k, v) cbind(v, 2*v))
> from.dfs(calc)

You will get a long series of output something as:

$key
NULL

$val
v
[1,]   1   2
[2,]   2   4
[3,]   3   6
[4,]   4   8
[5,]   5  10
[6,]   6  12
[7,]   7  14
[8,]   8  16
[9,]   9  18

Advertisements

Tags: R, RHadoop, RHadoop on CentOS 6.5