ODI - Hive and NoSQL

The Hive external table let's us do lots of cool stuff including processing data from NoSQL. We have seen how custom SerDes are used, Hive storage handlers also provide some cool capabilities. Using the Hive storage handler defined here, an external table can be defined to project data from a NoSQL key-value store. The external table can then be used as a source in ODI, very simple.

The illustration on github has the following data stored in a Oracle NoSQL Database (the key is the lastname/firstname, the value is name-value pair):

Using the Hive external table and the custom storage handler for a key value store, we define a mask to project the data through the external table. 

1. ADD JAR /home/oracle/kv/HiveKVStorageHandler.jar;
2. CREATE EXTERNAL TABLE MY_KV_TABLE (lastname string, firstname string, birthdate string, phonenumber string, userid string)
3.       STORED BY 'org.vilcek.hive.kv.KVHiveStorageHandler'
4.       WITH SERDEPROPERTIES ("kv.major.keys.mapping" = "lastname,firstname", "kv.minor.keys.mapping" = "birthdate,phonenumber,userID")
5.       TBLPROPERTIES ("kv.host.port" = "localhost:5000", "kv.name" = "kvstore");

There are a few interesting properties here;

• we specify the keyvalue store using TBLPROPERTIES, identify the host/port and the keystore name (kvstore).
• the SerDe properties contains the mapping of the keys to column names, you will get a row for each value of birthdate, phonenumber, userID

Fairly straightforward. We can then reverse engineer this into ODI, using the same mechanism as I described in previous posts here setting the ODI_HIVE_SESSION_JARS and so forth. The data projected looks like this;

1. hive> SELECT * FROM MY_KV_TABLE;
2. OK
3. Smith     Patricia     10/25/1967     NULLNULL
4. SmithPatriciaNULL2222-2222     NULL
5. SmithPatriciaNULLNULL2
6. SmithBob05/02/1975NULLNULL
7. SmithBobNULL1111-1111NULL
8. SmithBobNULLNULL1
9. WongBill03/10/1982NULLNULL
10. WongBillNULL3333-3333NULL
11. WongBillNULLNULL3

In ODI by defining the Hive collect_set function as an aggregation function, we can then aggregate the data and pivot the data to get it as a row;

1. SELECT lastname, firstname, collect_set(birthdate)[0], collect_set(phonenumber)[0], collect_set(userid)[0]
2.       FROM MY_KV_TABLE
3.       GROUP BY lastname, firstname;

So another interesting illustration of external tables in Hive and what they can provide.