Category Archives: Bigdata

Real time use cases of Hadoop

As data continues to grow, businesses now have access to (or generate) more data than ever before–much of which goes unused. How can you turn this data into a competitive advantage? In this article, we explore different ways businesses are capitalizing on data.

We keep hearing statistics about the growth of data. For instance:

• Data volume in the enterprise is going to grow 50x year-over-year between now and 2020.
• The volume of business data worldwide, across all companies, doubles every 1.2 years.
• Back in 2010, Eric Schmidt famously stated that every 2 days, we create as much information as we did from the dawn of civilization up until 2003.

The big questions: Where is this data? How can you use it to your advantage?

If you want to capitalize on this data, you must first begin storing it somewhere. But, how can you store and process massive data sets without spending a fortune on storage? “That’s where Hadoop comes into play”

Hadoop is an open-source software framework for storing and processing large data sets. It stores data in a distributed fashion on clusters of commodity hardware, and is designed to scale up easily as needed. Hadoop helps businesses store and process massive amounts of data without purchasing expensive hardware.

The great advantage of Hadoop: It lets you collect data now and ask questions later. You don’t need to know every question you want answered before you start using Hadoop.

Once you begin storing data in Hadoop, the possibilities are endless. Companies across the globe are using this data to solve big problems, answer pressing questions, improve revenue, and more. How? Here are some real-life examples of ways other companies are using Hadoop to their advantage.

1. Analyze life-threatening risks: Suppose you’re a doctor in a busy hospital. How can you quickly identify patients with the biggest risks? How can you ensure that you’re treating those with life-threatening issues, before spending your time on minor problems? Here’s a great example of one hospital using big data to determine risk–and make sure they’re treating the right patients.

“Patients in a New York hospital with suspicion of heart attack were submitted to series of tests, and the results were analyzed with use of big data – history of previous patients,” says Agnieszka Idzik, Senior Product Manager at SALESmanago. “Whether a patient was admitted or sent home depended on the algorithm, which was more efficient than human doctors.”

2. Identify warning signs of security breaches: What if you could stop security breaches before they happened? What if you could identify suspicious employee activity before they took action? The solution lies in data.

As explained below, security breaches usually come with early warning signs. Storing and analyzing data in Hadoop is a great way to identify these problems before they happen.

“Data breaches like we saw with Target, Sony, and Anthem never just happen; there are typically early warning signs – unusual server pings, even suspicious emails, IMs or other forms of communication that could suggest internal collusion,” according to Kon Leong, CEO, ZL Technologies. “Fortunately, with the ability to now mine and correlate people, business, and machine-generated data all in one seamless analytics environment, we can get a far more complete picture of who is doing what and when, including the detection of collusion, bribery, or an Ed Snowden in progress even before he has left the building.”

3. Prevent hardware failure:Machines generate a wealth of information–much of which goes unused. Once you start collecting that data with Hadoop, you’ll learn just how useful this data can be.

For instance, this recent webinar on “Practical Uses of Hadoop,” explores one great example. Capturing data from HVAC systems helps a business identify potential problems with products and locations.

Here’s another great example: One power company combined sensor data from the smart grid with a map of the network to predict which generators in the grid were likely to fail, and how that failure would affect the network as a whole. Using this information, they could react to problems before they happened.

4. Understand what people think about your company: Do you ever wonder what customers and prospects say about your company? Is it good or bad? Just imagine how useful that data could be if you captured it.

With Hadoop, you can mine social media conversations and figure out what people think of you and your competition. You can then analyze this data and make real-time decisions to improve user perception.

For instance, this article explains how one company used Hadoop to track user sentiment online. It gave their marketing teams the ability to assess external perception of the company (positive, neutral, or negative), and make adjustments based on that data.

5. Understand when to sell certain products:

“Done well, data can help companies uncover, quantitatively, both pain points and areas of opportunity,” says Mark Schwarz, VP of Data Science, at Square Root. “For example, tracking auto sales across dealerships may highlight that red cars are selling and blue cars or not. Knowing this, the company could adjust inventory to avoid the cost of blue cars sitting on the lot and increase revenue from having more red cars. It’s a data-driven way to understand what’s working and what’s not in a business and helps eliminate “gut reaction” decision making.”

Of course, this can go far beyond determining which product is selling best. Using Hadoop, you can analyze sales data against any number of factors.

For instance, if you analyzed sales data against weather data, you could determine which products sell best on hot days, cold days, or rainy days.

Or, what if you analyzed sales data by time and day. Do certain products sell better on specific weeks/days/hours?

Those are just a couple of examples, but I’m sure you get the point. If you know when products are likely to sell, you can better promote those products.

6. Find your ideal prospects: Chances are, you know what makes a good customer. But, do you know exactly where they are? What if you could use freely available data to identify and target your best prospects?

There’s a great example in this article. It explains how one company compared their customer data with freely available census data. They identified the location of their best prospects, and ran targeted ads at them. The results: Increased conversions and sales.

7. Gain insight from your log files: Just like your hardware, your software generates lots of useful data. One of the most common examples: Server log files. Server logs are computer-generated log files that capture network and server operations data. How can this data help? Here are a couple examples:

Security: What happens if you suspect a security breach? The server log data can help you identify and repair the vulnerability.

Usage statistics: As demonstrated in this webinar, server log data provides valuable insight into usage statistics. You can instantly see which applications are most popular, and which users are most active.

8. Threat Analysis:  How can companies detect threats and fraudulent activity?
Businesses have struggled with theft, fraud, and abuse since long before computers existed. Computers and on-line systems create new opportunities for criminals to act swiftly, efficiently, and anonymously. On-line businesses use Hadoop to monitor and combat criminal behavior.
Challenge: Online criminals write viruses and malware to take over individual computers and steal valuable data. They buy and sell using fraudulent identities and use scams to steal money or goods. They lure victims into scams by sending email or other spam over networks. In “pay-per-click” systems like online advertising, they use networks of compromised computers
to automate fraudulent activity, bilking money from advertisers or ad networks. Online businesses must capture, store, and analyze both the content and the pattern of messages that flow through the network to tell the difference between a legitimate
transaction and fraudulent activity by criminals.

Solution: 
One of the largest users of Hadoop, and in particular of HBase, is a global developer of software and services to protect against computer viruses. Many detection systems compute a “signature” for a virus or other malware, and use that signature to spot instances of the virus in the wild. Over the decades, the company has built up an enormous library of malware indexed by signatures. HBase provides an inexpensive and high-performance storage system for this data. The vendor uses MapReduce to compare instances of malware to one another, and to build higher-level models of the threats that the different pieces of malware pose. The ability to examine all the data comprehensively allows the company to build much more robust tools for detecting known and emerging threats. A large online email provider has a Hadoop cluster that provides a similar service. Instead of detecting viruses, though, the system recognizes spam messages. Email flowing through the system is examined automatically. New spam messages are properly flagged, and the system detects and reacts to new attacks as criminals create them. Sites that sell goods and services over the internet are particularly vulnerable to fraud and theft. Many use web logs to monitor user behavior on the site. By tracking that activity, tracking IP addresses and using knowledge of the location of individual visitors, these sites are able to recognize and prevent fraudulent activity. The same techniques work for online advertisers battling click fraud. Recognizing patterns of activity by individuals permits the ad networks to detect and reject fraudulent activity. Hadoop is a powerful platform for dealing with fraudulent and criminal activity like this. It is
flexible enough to store all of the data—message content, relationships among people and computers, patterns of activity—that matters. It is powerful enough to run sophisticated detection and prevention algorithms and to create complex models from historical data to monitor real-time activity

9. Ad Targeting:  How can companies increase campaign efficiency?

Two leading advertising networks use Hadoop to choose the best ad to show to any given user.

Challenge:  Advertisement targeting is a special kind of recommendation engine. It selects ads best suited to a particular visitor. There is, though, an additional twist: each advertiser is willing to pay a certain amount to have its ad seen. Advertising networks auction ad space, and advertisers want their ads shown to the people most likely to buy their products. This creates a complex optimization challenge. Ad targeting systems must understand user preferences and behavior, estimate how interested a given user will be in the different ads available for display, and choose the one that maximizes revenue to both the advertiser and the advertising network. The data managed by these systems is simple and structured. The ad exchanges, however, provide services to a large number of advertisers, deliver advertisements on a wide variety of Web properties and must scale to millions of end users browsing the web and loading pages that must include advertising. The data volume is enormous. Optimization requires examining both the relevance of a given advertisement to a particular user, and the collection of bids by different advertisers who want to reach that visitor. The analytics required to make the correct choice are complex, and running them on the large dataset requires a large-scale, parallel system.

Solution: One advertising exchange uses Hadoop to collect the stream of user activity coming off of its servers. The system captures that data on the cluster, and runs analyses continually to determine how successful the system has been at displaying ads that appealed to users. Business analysts at the exchange are able to see reports on the performance of individual ads, and to adjust the system to improve relevance and increase revenues immediately. A second exchange builds sophisticated models of user behavior in order to choose the right ad for a given visitor in real time. The model uses large amounts of historical data on user behavior to cluster ads and users, and to deduce preferences. Hadoop delivers much better-targeted advertisements by steadily refining those models and delivering better ads.

Article credit: Joe Stangarone and Cloudera

How to change knox heap size

Some time due to heavy load you may a requirement to increase your knox jvm size to deal more reques and to give response in a time.

So in that case you can change your knox jvm size in following ways.

1. go to /usr/hdp/current/knox-server/bin/gateway.sh and seach for APP_MEM_OPTS string.
2. Once you get it then you can change it accordingly:

             APP_MEM_OPTS=“-Xms2g -Xmx2g”

You need to restart your knox gateway services to take effect.

Import & Export in Hive

When we work on Hive, there would be lots of scenarios that we need to move data(i.e tables from one cluster to another cluster) from one cluster to another.

For example, sometimes we need to copy some production table from one cluster to another cluster. Now we have got very good functionality in hive which give us two easy commands to do it.

Version 0.8 onwards, Hive supports EXPORT and IMPORT features that allows us to export the metadata as well as the data for the corresponding table to a directory in HDFS, which can then be imported back to another database or Hive instance.

Now with the help of following example I have tried to copy cars_beeline from cluster A to Cluster B :

Cluster A: 

hive>show databases;

OK

admintestdb

default

kmsdatabase

kmstest

samplebeelinetest

samplehivetest

sampletest

sampletest1

Time taken: 2.911 seconds, Fetched: 14 row(s)

hive> use samplebeelinetest;

OK

Time taken: 0.287 seconds

hive> show tables;

OK

cars

cars_beeline

cars_internal

i0014_itm_typ

test

Time taken: 0.295 seconds, Fetched: 5 row(s)

hive> select * from cars_beeline limit 1;

OK

Name NULL NULL NULL NULL NULL NULL NULL O

Time taken: 1.118 seconds, Fetched: 1 row(s)

hive> select * from cars_beeline limit 10;

OK

Name NULL NULL NULL NULL NULL NULL NULL O

“chevrolet chevelle malibu” 18 8 307 130 3504 12 1970-01-01 A

“buick skylark 320” 15 8 350 165 3693 12 1970-01-01 A

“plymouth satellite” 18 8 318 150 3436 11 1970-01-01 A

“amc rebel sst” 16 8 304 150 3433 12 1970-01-01 A

“ford torino” 17 8 302 140 3449 11 1970-01-01 A

Time taken: 0.127 seconds, Fetched: 6 row(s)

hive> export table cars_beeline to ‘/tmp/cars_beeline’;

Copying data from file:/tmp/s0998dnz/0bd6949f-c28c-4113-a9ab-eeaea4dcd434/hive_2016-07-11_05-41-39_786_4427147069708259788-1/-local-10000/_metadata

Copying file: file:/tmp/s0998dnz/0bd6949f-c28c-4113-a9ab-eeaea4dcd434/hive_2016-07-11_05-41-39_786_4427147069708259788-1/-local-10000/_metadata

Copying data from hdfs://HDPCLUSTERAHA/zone_encr2/data

Copying file: hdfs://HDPCLUSTERAHA/zone_encr2/data/cars.csv

OK

Time taken: 0.52 seconds

hive> dfs -ls /tmp/cars_beeline;

Found 2 items

-rwxrwxrwx   3 s0998dnz hdfs       1701 2016-07-11 05:41 /tmp/cars_beeline/_metadata

drwxrwxrwx   – s0998dnz hdfs          0 2016-07-11 05:41 /tmp/cars_beeline/data

Now use distcp to copy that dir from Cluster A to Cluster B: 

[root@server1 ~]$ hadoop distcp hdfs://HDPCLUSTERAHA/tmp/cars_beeline hdfs://HDPCLUSTERBHA/tmp/cars_beeline

16/07/11 05:43:09 INFO tools.DistCp: Input Options: DistCpOptions{atomicCommit=false, syncFolder=false, deleteMissing=false, ignoreFailures=false, maxMaps=20, sslConfigurationFile=’null’, copyStrategy=’uniformsize’, sourceFileListing=null, sourcePaths=[hdfs://HDPCLUSTERAHA/tmp/cars_beeline], targetPath=hdfs://HDPCLUSTERAHA/tmp/cars_beeline, targetPathExists=false, preserveRawXattrs=false}

16/07/11 05:43:09 INFO impl.TimelineClientImpl: Timeline service address: http://server2:8188/ws/v1/timeline/

16/07/11 05:43:11 INFO impl.TimelineClientImpl: Timeline service address: http://server2:8188/ws/v1/timeline/

16/07/11 05:43:11 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2

16/07/11 05:43:11 INFO mapreduce.JobSubmitter: number of splits:4

16/07/11 05:43:11 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1468229364042_0002

16/07/11 05:43:11 INFO impl.YarnClientImpl: Submitted application application_1468229364042_0002

16/07/11 05:43:11 INFO mapreduce.Job: The url to track the job: http://server1:8088/proxy/application_1468229364042_0002/

16/07/11 05:43:11 INFO tools.DistCp: DistCp job-id: job_1468229364042_0002

16/07/11 05:43:11 INFO mapreduce.Job: Running job: job_1468229364042_0002

16/07/11 05:43:25 INFO mapreduce.Job: Job job_1468229364042_0002 running in uber mode : false

16/07/11 05:43:25 INFO mapreduce.Job:  map 0% reduce 0%

16/07/11 05:43:31 INFO mapreduce.Job:  map 75% reduce 0%

16/07/11 05:43:35 INFO mapreduce.Job:  map 100% reduce 0%

16/07/11 05:43:36 INFO mapreduce.Job: Job job_1468229364042_0002 completed successfully

Run following steps on target cluster B:

[root@server1ofclusterb ~]$ hadoop fs -ls /tmp/cars_beeline;

Found 2 items

-rw-r–r–   3 s0998dnz hdfs       1701 2016-07-11 05:43 /tmp/cars_beeline/_metadata

drwxr-xr-x   – s0998dnz hdfs          0 2016-07-11 05:43 /tmp/cars_beeline/data

[root@server1ofclusterb ~]$ hive

WARNING: Use “yarn jar” to launch YARN applications.

16/07/11 05:45:22 WARN conf.HiveConf: HiveConf of name hive.server2.enable.impersonation does not exist

16/07/11 05:45:22 WARN conf.HiveConf: HiveConf of name hive.server2.http.port does not exist

Logging initialized using configuration in file:/etc/hive/2.3.4.0-3485/0/hive-log4j.properties

hive> create database testing;

OK

Time taken: 0.342 seconds

hive> use testing;

OK

Time taken: 0.411 seconds

hive> IMPORT TABLE cars_beeline from ‘/tmp/cars_beeline’;

Copying data from hdfs://HDPCLUSTERBHA/tmp/cars_beeline/data

Copying file: hdfs://HDPCLUSTERBHA/tmp/cars_beeline/data/cars.csv

Loading data to table testing.cars_beeline

OK

Time taken: 1.196 seconds

hive> show tables;

OK

cars_beeline

Time taken: 0.246 seconds, Fetched: 1 row(s)

hive> select * from cars_beeline limit 10;

OK

Name NULL NULL NULL NULL NULL NULL NULL O

“chevrolet chevelle malibu” 18 8 307 130 3504 12 1970-01-01 A

“buick skylark 320” 15 8 350 165 3693 12 1970-01-01 A

“plymouth satellite” 18 8 318 150 3436 11 1970-01-01 A

“amc rebel sst” 16 8 304 150 3433 12 1970-01-01 A

“ford torino” 17 8 302 140 3449 11 1970-01-01 A

Time taken: 0.866 seconds, Fetched: 6 row(s)

hive>

I hope it will help you to move table from one cluster to another cluster. Please fell free to give your suggestion to improve this article.

How to read compressed data from hdfs through hadoop command

Sometime we have a requirement where we need to read compressed data from hdfs through hdfs command. And we have many compressed algorithms like(.gz, .snappy, .lzo and .bz2 etc).

I have tried to explain how we can achieve this requirement with the help of following ways :

Step 1: Copy any compressed file to your hdfs dir: 

[s0998dnz@hdpm1 ~]$ hadoop fs -put logs.tar.gz /tmp/

Step 2: Now you can use in-build hdfs text command to read this .gz file. This command-line will automatically find the right decompressor for any simple text file and print the uncompressed data to standard output:

[user1@hdpm1 ~]$ hadoop fs -text /tmp/logs.tar.gzvar/log/hadoop/hdfs/gc.log-2016052306240000644002174336645170000001172412720563430016153 0ustar   hdfshadoop2016-05-23T06:24:03.539-0400: 2.104: [GC2016-05-23T06:24:03.540-0400: 2.104: [ParNew: 163840K->14901K(184320K), 0.0758510 secs] 163840K->14901K(33533952K), 0.0762040 secs] [Times: user=0.51 sys=0.01, real=0.08 secs]2016-05-23T06:24:04.613-0400: 3.178: [GC2016-05-23T06:24:04.613-0400: 3.178: [ParNew: 178741K->16370K(184320K), 0.1591140 secs] 965173K->882043K(33533952K), 0.1592230 secs] [Times: user=1.21 sys=0.03, real=0.16 secs]2016-05-23T06:24:06.121-0400: 4.686: [GC2016-05-23T06:24:06.121-0400: 4.686: [ParNew: 180210K->11741K(184320K), 0.0811950 secs] 1045883K->887215K(33533952K), 0.0813160 secs] [Times: user=0.63 sys=0.00, real=0.09 secs]2016-05-23T06:24:12.313-0400: 10.878: [GC2016-05-23T06:24:12.313-0400: 10.878: [ParNew: 175581K->9827K(184320K), 0.0751580 secs] 1051055K->892704K(33533952K), 0.0752800 secs] [Times: user=0.56 sys=0.01, real=0.07 secs]2016-05-23T06:24:13.881-0400: 12.445: [GC2016-05-23T06:24:13.881-0400: 12.445: [ParNew: 173667K->20480K(184320K), 0.0810330 secs] 1056544K->920485K(33533952K), 0.0812040 secs] [Times: user=0.58 sys=0.01, real=0.08 secs]2016-05-23T06:24:16.515-0400: 15.080: [GC2016-05-23T06:24:16.515-0400: 15.080: [ParNew: 184320K->13324K(184320K), 0.0867770 secs] 1084325K->931076K(33533952K), 0.0870140 secs] [Times: user=0.63 sys=0.01, real=0.08 secs]2016-05-23T06:24:17.268-0400: 15.833: [GC2016-05-23T06:24:17.268-0400: 15.833: [ParNew: 177164K->11503K(184320K), 0.0713880 secs] 1094916K->929256K(33533952K), 0.0715820 secs] [Times: user=0.55 sys=0.00, real=0.07 secs]2016-05-23T06:25:14.412-0400: 72.977: [GC2016-05-23T06:25:14.412-0400: 72.977: [ParNew: 175343K->18080K(184320K), 0.0779040 secs] 1093096K->935833K(33533952K), 0.0781710 secs] [Times: user=0.59 sys=0.01, real=0.07 secs]2016-05-23T06:26:49.597-0400: 168.161: [GC2016-05-23T06:26:49.597-0400: 168.162: [ParNew: 181920K->13756K(184320K), 0.0839120 secs] 1099673K->941811K(33533952K), 0.0841350 secs] [Times: user=0.62 sys=0.01, real=0.08 secs]2016-05-23T06:26:50.126-0400: 168.691: [GC2016-05-23T06:26:50.127-0400: 168.691: [ParNew: 177596K->9208K(184320K), 0.0641380 secs] 1105651K->937264K(33533952K), 0.0644310 secs] [Times: user=0.50 sys=0.00, real=0.07 secs]2016-05-23T06:27:19.282-0400: 197.846: [GC2016-05-23T06:27:19.282-0400: 197.847: [ParNew: 173048K->10010K(184320K), 0.0687210 secs] 1101104K->938065K(33533952K), 0.0689210 secs] [Times: user=0.54 sys=0.00, real=0.07 secs]2016-05-23T06:30:45.428-0400: 403.992: [GC2016-05-23T06:30:45.428-0400: 403.992: [ParNew: 173850K->9606K(184320K), 0.0723210 secs] 1101905K->937661K(33533952K), 0.0726160 secs] [Times: user=0.56 sys=0.00, real=0.07 secs]2016-05-23T06:37:15.629-0400: 794.193: [GC2016-05-23T06:37:15.629-0400: 794.193: [ParNew: 173446K->9503K(184320K), 0.0723460 secs] 1101501K->937558K(33533952K), 0.0726260 secs] [Times: user=0.57 sys=0.0

In the above example I have tried to read .gz files. It probably works for .snappy, .lzo and .bz2 files.

This is an important feature because Hadoop uses a custom file format for Snappy files. This is the only direct way to uncompress a Hadoop-created Snappy file.

Note: hadoop fs -text is single-threaded and runs the decompression on the machine where you run the command.