Install RockMongo For Mongodb On CentOS

Install the prerequisites for mongo-php-driver and some development tools

# yum install php-devel git httpd
# yum install gcc make

Compile the driver from the latest source code on Github

# git clone https://github.com/mongodb/mongo-php-driver.git
# cd mongo-php-driver/
# phpize
# ./configure
# make all
# make install

Add the following line to your php.ini file:

# vi /etc/php.ini

After opening the php.ini file,

extension=mongo.so

Get the latest rockmongo zip file and unzip it into root directory of the Apache web server, i.e., RockMongo v1.1.5 (2012/12/20); Remember to restart the Apache.

# wget http://rockmongo.com/downloads/go?id=12
# mv rockmongo-1.1.5.zip /var/www/html/
# cd /var/www/html/
# unzip rockmongo-1.1.5.zip
# /etc/init.d/httpd restart

Visit http://ipaddress-or-hostname/rockmongo and sign-in with the default username and password (admin/admin)
RockMongo

We can change the sing-in account in config.php file:

# vi /var/www/html/rockmongo/config.php

find the following snippet and change it

$MONGO["servers"][$i]["control_users"]["admin"] = "admin";

TROUBLESHOOTING – Open Apache httpd web server port 80

# vi /etc/sysconfig/iptables 

Add the following line to your iptables:

-A INPUT -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT

TROUBLESHOOTING – Unable to connect MongoDB

Unable to connect MongoDB, please check your configurations. MongoDB said:Failed to connect to: 127.0.0.1:27017: Permission denied.

# /usr/sbin/setsebool -P httpd_can_network_connect 1 

Then restart Apache.

REFERENCES

  1. PHP: Installation – Manual
  2. Install RockMongo For Mongodb On CentOS, RHEL, Debian, Ubuntu
Advertisements

Return a Swift object from Jersey REST service

201305141140

This scenario assumes that the clients want to grab some objects on OpenStack’s Swift via the RESTful request. The main concern is we don’t want to public our container or create Temporary URL every access. As shown in the figure above, the Tomcat instances can verify the access permission or wrap some information before return to the client.

In addition, our GET method didn’t indicate @Produces annotation to specify the MIME media types of representations a resource can produce and send back to the client. The following code snippet use the java-cloudfiles to communicate with the Swift service. First, we obtain the meta-data of the object to get the MIME type, then we store the object content into byte-array as return entity. In the end, we can return the content of a object with corresponding MIME type to the client.

@GET
public Response getSwiftObject() {
    FilesClient client = new FilesClient(username, password, authUrl);
    try {
        if (!client.login()) {
            throw new RuntimeException();
        }
        FilesObjectMetaData metaData = client.getObjectMetaData(
            container_name, object_name);
        byte[] objContent = client.getObject(container_name, object_name);

        return Response.ok(objContent, metaData.getMimeType()).build();
    } catch (Exception e) {
        // do something
    }
    
    return Response.noContent().build();
}

Visual Machine Network Failure on OpenStack

While we used Nova as computing nodes on OpenStack, the network will malfunction occasionally. The weird thing is the configuration of Ethernet (i.e., eth0) still correct. However, we yet figure out the true cause. The alternative solution is to check the functionality of Ethernet (e.g., ping somewhere) periodically. For example, I put the following command into /etc/crontab in CentOS 6.x and check the network per minute; If encounter the failure, system will restart the network.

1 * * * * ping 168.95.1.1 -c5 -w5 || /etc/init.d/network restart

Converting ISODate from MongoDB

I’m confused about the insertion operation with Date object via mongo-java-driver that always short of 8 hours where my place of residence is Taiwan (GMT+8). According to those posts [1, 2], we can observe that the incoming Date object will be set to ISO_8601_DATE_FORMAT as shown in the 1st code snippet. The 2nd example demonstrates that the given Date will plus 8 hours in the GMT+8 time zone [2].

if (o instanceof Date) {
    Date d = (Date) o;
    SimpleDateFormat format = new SimpleDateFormat(ISO_8601_DATE_FORMAT);
    serialize(new BasicDBObject("$date", format.format(d)), buf);
    return;
}
SimpleDateFormat format = 
    new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.SSS'Z'");
format.setCalendar(new GregorianCalendar(new SimpleTimeZone(0, "GMT")));
Date date = format.parse("2012-01-20T00:00:00.000Z");

References

  1. serializing and deserializing of java.util.Date can now handle timezone …
  2. mongo java driver日期转换问题
  3. A guide to Java SimpleDateFormat in examples

Jersey Test Framework with Maven

This memo records the issues while executing the unit-test with Jersey Test framework. We use the Jersey framework to implement the RESTful Web services and employ the Maven to manage the dependencies in project. First of all, we add the jersey-test-framework-grizzly2 dependency to enable the test framework in pom.xml; Second, we deploy the application using Jersey specific servlet in web.xml. Finally, we have the following java files within Maven Archetype – maven-archetype-webapp:

  • src/main/java – PersonResource.java (RESTful Resource)
  • src/main/java – Person.java (POJO class)
  • src/test/java – PersonResourceTest.java (Test Case)

The POJO class Person with two fields/properties/attributes, i.e., Name and Country with corresponding getter/setter functions. The PersonResource.java represents the RESTful Web service, here, we simply declare a Person instance and set the values. In test case PersonResourceTest.java, we’ll demonstrate how to retrieve/mapping the response.

Code Snippets


package net.jersey.rest;

import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.Produces;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.Response;
import net.jersey.pojo.Person;

@Path("person")
public class PersonResource {

    @GET
    @Produces(MediaType.APPLICATION_JSON)
    public Response fetchPerson(){      
        Person person = new Person();
        person.setName("MyName");
        person.setCountry("Taiwan, R.O.C.");
        
        return Response.status(200)
            .entity(person).build();
    }
}

package net.jersey.pojo;

import javax.xml.bind.annotation.XmlRootElement;
import org.codehaus.jackson.annotate.JsonProperty;

@XmlRootElement
public class Person {

    private String name;
    private String country;

    public String getName() {
        return name;
    }

    public void setName(String name) {
        this.name = name;
    }

    public String getCountry() {
        return country;
    }

    public void setCountry(String country) {
        this.country = country;
    }
}

Here we use GenericType<T> to restore the client response and map into the class we defined, i.e., Person.java; Moreover, if we want to evaluate the HTTP Response status, we can utilize the ClientResponse class.


package net.jersey.rest;

import static org.junit.Assert.assertEquals;
import javax.ws.rs.core.MediaType;
import net.jersey.pojo.Person;
import org.junit.Test;
import com.sun.jersey.api.client.GenericType;
import com.sun.jersey.api.client.WebResource;
import com.sun.jersey.test.framework.JerseyTest;

public class PersonResourceTest extends JerseyTest {

    public PersonResourceTest() throws Exception {
        super("net.jersey.rest");
    }

    @Test
    public void testFetchPerson() {     
        WebResource webResource = resource();       
        Person person = webResource.path("person")
            .accept(MediaType.APPLICATION_JSON)
            .get(new GenericType<Person>(){});
            
        assertEquals("MyName", person.getName());
    }
}

Console

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running net.jersey.rest.PersonResourceTest
Mar 05, 2013 5:52:09 PM com.sun.jersey.test.framework.spi.container.grizzly2.web.GrizzlyWebTestContainerFactory$GrizzlyWebTestContainer 
INFO: Creating Grizzly2 Web Container configured at the base URI http://localhost:9998/
Mar 05, 2013 5:52:09 PM org.glassfish.grizzly.http.server.NetworkListener start
INFO: Started listener bound to [localhost:9998]
Mar 05, 2013 5:52:09 PM org.glassfish.grizzly.http.server.HttpServer start
INFO: [HttpServer] Started.
Mar 05, 2013 5:52:09 PM org.glassfish.grizzly.servlet.WebappContext deploy
INFO: Starting application [TestContext] ...
Mar 05, 2013 5:52:09 PM org.glassfish.grizzly.servlet.WebappContext initServlets
INFO: [TestContext] Servlet [com.sun.jersey.spi.container.servlet.ServletContainer] registered for url pattern(s) [[/*]].
Mar 05, 2013 5:52:09 PM org.glassfish.grizzly.servlet.WebappContext deploy
INFO: Application [TestContext] is ready to service requests.  Root: [].
Mar 05, 2013 5:52:09 PM com.sun.jersey.test.framework.spi.container.grizzly2.web.GrizzlyWebTestContainerFactory$GrizzlyWebTestContainer start
INFO: Starting the Grizzly2 Web Container...
Mar 05, 2013 5:52:09 PM org.glassfish.grizzly.servlet.ServletHandler loadServlet
INFO: Loading Servlet: com.sun.jersey.spi.container.servlet.ServletContainer
Mar 05, 2013 5:52:09 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
  net.jersey.rest
Mar 05, 2013 5:52:09 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
  class net.jersey.rest.PersonResource
Mar 05, 2013 5:52:09 PM com.sun.jersey.api.core.ScanningResourceConfig init
INFO: No provider classes found.
Mar 05, 2013 5:52:09 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.17 01/17/2013 03:31 PM'
Mar 05, 2013 5:52:10 PM com.sun.jersey.test.framework.spi.container.grizzly2.web.GrizzlyWebTestContainerFactory$GrizzlyWebTestContainer stop
INFO: Stopping the Grizzly2 Web Container...
Mar 05, 2013 5:52:10 PM org.glassfish.grizzly.http.server.NetworkListener stop
INFO: Stopped listener bound to [localhost:9998]
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.707 sec

Results :

Tests run: 1, Failures: 0, Errors: 0, Skipped: 0

Troubleshooting

Mar 05, 2013 3:01:43 PM com.sun.jersey.spi.container.ContainerResponse write
SEVERE: A message body writer for Java class net.jersey.pojo.Person, and Java type class net.jersey.pojo.Person, and MIME media type application/json was not found

Maven developers, using JSON serialization support of JAXB beans when using the MIME media type application/json require a dependency on the jersey-json module (no explicit dependency on jaxb-impl is required) [1]

<dependency>
    <groupId>com.sun.jersey</groupId>
    <artifactId>jersey-json</artifactId>
    <version>1.17</version>
</dependency>
SEVERE: The registered message body writers compatible with the MIME media type are:
application/json ->
  com.sun.jersey.json.impl.provider.entity.JSONJAXBElementProvider$App
  com.sun.jersey.json.impl.provider.entity.JSONArrayProvider$App
  com.sun.jersey.json.impl.provider.entity.JSONObjectProvider$App
  com.sun.jersey.json.impl.provider.entity.JSONRootElementProvider$App
  com.sun.jersey.json.impl.provider.entity.JSONListElementProvider$App
*/* ->
...

Here we need to add a XmlRootElement annotation on that pojo-class this will enable jaxb to convert to and from json where necessary [2]. The @JsonProperty annotation is used to customise the string filed in JSON ouput which is not necessary.


package net.jersey.pojo;
 
import javax.xml.bind.annotation.XmlRootElement;
import org.codehaus.jackson.annotate.JsonProperty;
 
@XmlRootElement
public class Person {
    ...
}

References

  1. Stack Overflow – Convert JSON to POJO
  2. Jersey – Dependencies
  3. Jersey message body reader not found in maven-built JAR

Download Source Code

Download – jersey-test-framework.zip (15 KB)

A Jersey POJOMapping Example in Mapping Form Parameters

In Java Servlet circumstance, we usually harvest the form parameters by using request.getParameter(“FORM_FIELD_NAME”) syntax. Now we can do it more elegant while enabling Jsersey’s POJOMapping features. The following example demonstrates the account registration scenario. Here we have a Account class:

public class Account {
    private String email;
 
    @JsonProperty("email")
    public String getEmail() {
        return email;
    }
 
    public void setEmail(String email) {
        this.email = email;
    }
}

Make sure you have turned on the POJOMapping feature,

<servlet>
    <servlet-name>Jersey REST Service</servlet-name>
    <servlet-class>com.sun.jersey.spi.container.servlet.ServletContainer</servlet-class>
    <init-param>
        <param-name>com.sun.jersey.config.property.packages</param-name>
        <param-value>ROOT_RESOURCE_AND_PROVIDER_CLASSES_IN_THE_PACKAGES</param-value>
    </init-param>
    <init-param>
        <param-name>com.sun.jersey.api.json.POJOMappingFeature</param-name>
        <param-value>true</param-value>
    </init-param>
    <load-on-startup>1</load-on-startup>
</servlet>
<servlet-mapping>
    <servlet-name>Jersey REST Service</servlet-name>
    <url-pattern>/rest/*</url-pattern>
</servlet-mapping>

Back-End service,

@Path("/account")
public class AccountResource {
 
    @POST
    @Path("/register")
    @Consumes(MediaType.APPLICATION_JSON)
    public Response Register(InputStream is){
        Response response = null;
        try {
            Account acct = new ObjectMapper().readValue(is,
                Account.class);         
            System.out.println(acct.getEmail());
        } catch (IOException e) {
            response = Response.serverError().build();
        }   
                     
        if(response == null)
            response = Response.ok().build();
         
        return response;                
    }
}

As shown in above code, we can read value from input steam into custom class (i.e., Account class) then continue doing the following business logic. However, you may encounter the Unsupported Media Type status code (415) while using the static HTML form post method. This is because we identify the service-consume-type is JSON.

html-form-post

So we get correct response and mapping object while the corresponding way. Note that the differences in Content-Type and Request Payload parts.

jQuery-post

Running Hadoop on CentOS 6 (Multi-Node Cluster)

hadoop.apache.org

This demonstration has been tested with the following software versions:

  • Oracle VM VirtualBox 4.1.22 r80657
  • CentOS-6.3_x86_64 with Minimal Installation
  • Hadoop 1.0.3

Networking

hdp01 192.168.1.39 NameNode、JobTracker、DataNode
hdp02 192.168.1.40 DataNode、TaskTracker
hdp03 192.168.1.41 DataNode、TaskTracker

Candidate CentOS VM [1]

After installing the CentOS in VirtualBox, patch up system by applying all updates and install wget tool for future use. Note that the following instructions are executed on hdp01.

[root@hdp01 ~]# yum -y update
[root@hdp01 ~]# yum -y install wget

Here we disable the iptables firewall In Redhat/CentOS Linux for facilitating the demonstration.

[root@hdp01 ~]# service iptables stop && chkconfig iptables off

Install Oracle Java SDK and configure the system environment variables. Note that the Oracle has recently disallowed direct downloads of java from their servers, [2] provides an feasible solution:

wget -c --no-cookies --header "Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F" "<DOWNLOAD URL>" --output-document="<DOWNLOAD FILE NAME>"

For example, we can use following command to obtain the JDK 1.7.0_07 rpm:

[root@hdp01 ~]# wget -c --no-cookies --header "Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F" "http://download.oracle.com/otn-pub/java/jdk/7u7-b10/jdk-7u7-linux-x64.rpm" --output-document="jdk-7u7-linux-x64.rpm"
[root@hdp01 ~]# rpm -ivh jdk-7u7-linux-x64.rpm
[root@hdp01 ~]# vi /etc/profile

Add the following variables:

JAVA_HOME=/usr/java/jdk1.7.0_07
PATH=$PATH:$JAVA_HOME/bin
CLASSPATH=.:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/dt.jar
export PATH JAVA_HOME CLASSPATH

Finally, we take user hadoop to perform the Hadoop operations.

[root@hdp01 ~]# adduser hadoop
[root@hdp01 ~]# passwd hadoop
[root@hdp01 ~]# gpasswd -a hadoop root
[root@hdp01 ~]# grep "^root" /etc/group

OpenSSH Packages

There are some omissive packages in minimal installation of CentOS, e.g., scp command.

[root@hdp01 ~]# yum -y install openssh-server openssh-clients
[root@hdp01 ~]# chkconfig sshd on
[root@hdp01 ~]# service sshd start
[root@hdp01 ~]# yum -y install rsync

FQDN Mapping

[root@hdp01 ~]# vi /etc/hosts

Add the following variables (all machines) to make sure each particular host is reachable.

127.0.0.1 localhost
192.168.1.39 hdp01
192.168.1.40 hdp02
192.168.1.41 hdp03

*VBoxManage Duplication

VBoxManage clonehd <source_file> <output_file>

After clonehd, the VM raises the network issue, delete the eth0 line and modify the eth1 line to be eth0 [3]. In addition, modify the hostname value.

[root@hdp01 ~]# vi /etc/udev/rules.d/70-persistent-net.rules
[root@hdp01 ~]# vi /etc/sysconfig/network

SSH Access

[hadoop@hdp01 ~]$ ssh-keygen -t rsa -P ''
[hadoop@hdp01 ~]$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
[hadoop@hdp01 ~]$ chmod 600 ~/.ssh/authorized_keys
[hadoop@hdp01 ~]$ scp -r ~/.ssh hdp*:~/

Hadoop Configuration

[hadoop@hdp01 ~]$ wget http://ftp.tc.edu.tw/pub/Apache/hadoop/common/hadoop-1.0.3/hadoop-1.0.3.tar.gz
[hadoop@hdp01 ~]$ tar zxvf hadoop-1.0.3.tar.gz


Configure the following files.

[hadoop@hdp01 ~]$ vi hadoop-1.0.3/conf/core-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://hdp01:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/var/hadoop/hadoop-${user.name}</value>
</property>
</configuration>

[hadoop@hdp01 ~]$ vi hadoop-1.0.3/conf/hdfs-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>

[hadoop@hdp01 ~]$ vi hadoop-1.0.3/conf/mapred-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>mapred.job.tracker</name>
<value>hdp01:9001</value>
</property>
</configuration>

[hadoop@hdp01 ~]$ vi hadoop-1.0.3/conf/hadoop-env.sh

export JAVA_HOME=/usr/java/jdk1.7.0_07
export HADOOP_HOME=/opt/hadoop
export HADOOP_CONF_DIR=/opt/hadoop/conf
export HADOOP_HOME_WARN_SUPPRESS=1

Deployment

[hadoop@hdp01 ~]$ scp -r hadoop-1.0.3 hdp01:~/
[hadoop@hdp02 ~]$ scp -r hadoop-1.0.3 hdp02:~/
 
// directories setting for each machines (hdp*)
[root@hdp01 ~]# cp hadoop-1.0.3 /opt/hadoop
[root@hdp01 ~]# cd /opt
[root@hdp01 ~]# mkdir /var/hadoop
[root@hdp01 ~]# chown -R hadoop.hadoop hadoop
[root@hdp01 ~]# chown -R hadoop.hadoop /var/hadoop

conf/masters (master only)

[hadoop@hdp01 ~]$ vi hadoop-1.0.3/conf/masters
 
hdp01

[hadoop@hdp01 ~]$ vi hadoop-1.0.3/conf/slaves
 
hdp01
hdp02
hdp03

Formatting the HDFS filesystem via the NameNode (master only)

[hadoop@hdp01 hadoop]$ bin/hadoop namenode -format

Now, we can start the multi-node cluster via bin/start-all.sh, and stop the service via bin/stop-all.sh command. Finally, we can put some materials into HDFS and conduct the wordcount on master machine to verify the whole process.

References

  1. 详细 完整分布模式安装hadoop VirtualBox 3虚拟机 简单复制
  2. wget for Oracle JDK is broken
  3. VirtualBox Clone Root HD / Ubuntu / Network issue