[Link].

COM HADOOP DEVELOPER INTERVIEW GUIDE

Sample Questions
Hadoop Developer Interview Guide

OM
1. How do you debug a performance issue or a long running job?

.C
This is an open ended question and the interviewer is trying to see the level of hands-on experience you have in
solving production issues. Use your day to day work experience to answer this question. Here are some of the

S
scenarios and responses to help you construct your answer. On a very high level you will follow the below steps.

ON
 Understand the symptom

TI
 Analyze the situation

ES
 Identify the problem areas
 Propose solution
W s
IE tion
QU
Scenario 1 - Job with 100 mappers and 1 reducer takes a long time for the reducer to start after all the mappers are
RV ues

complete. One of the reasons could be that reduce is spending a lot of time copying the map outputs. So in this case
we can try couple of things.
TE Q

1. If possible add a combiner to reduce the amount of output from the mapper to be sent to the reducer
IN ple

2. Enable map output compression - this will further reduce the size of the outputs to be transferred to the
reducer.
TA am

Scenario 2 - A particular task is using a lot of memory which is causing the slowness or failure, I will look for ways
DA S

to reduce the memory usage.

1. Make sure the joins are made in an optimal way with memory usage in mind. For e.g. in Pig joins, the
LEFT hand side tables are sent to the reducer first and held in memory and the RIGHT most table in
streamed to the reducer. So make sure the RIGHT most table is largest of the datasets in the join.
2. We can also increase the memory requirements needed by the map and reduce tasks by setting -
IG

[Link] and [Link]

Scenario 3 - Understanding the data helps a lot in optimizing the way we use the datasets in PIG and HIVE scripts.
W
W

1. If you have smaller tables in join, they can be sent to distributed cache and loaded in memory on the Map
side and the entire join can be done on the Map side thereby avoiding the shuffle and reduce phase
W

altogether. This will tremendously improve performance. Look up USING REPLICATED in Pig and
MAPJOIN or [Link] in Hive
2. If the data is already sorted you can use USING MERGE which will do a Map Only join
3. If the data is bucketted in hive, you may use [Link] or
[Link] depending on the characteristics of the data

Scenario 4 - The Shuffle process is the heart of a MapReduce program and it can be tweaked for performance
improvement.

1. If you see lots of records are being spilled to the disk (check for Spilled Records in the counters in your
MapReduce output) you can increase the memory available for Map to perform the Shuffle by increasing
[Link] HADOOP DEVELOPER INTERVIEW GUIDE

the value in [Link]. This will reduce the amount of Map Outputs written to the disk so the sorting of the
keys can be performed in memory.
2. On the reduce side the merge operation (merging the output from several mappers) can be done in disk by
setting the [Link] to 0

2. Assume you have Research, Marketing and Finance teams funding 60%, 30% and 10%
respectively of your Hadoop Cluster. How will you assign only 60% of cluster resources
to Research, 30% to Marketing and 10% to Finance during peak load?

OM
Capacity scheduler in Hadoop is designed to support this use case. Capacity scheduler supports hierarchical queues
and capacity can be defined for each queue.

.C
For this use case, you would have to define 3 queues under the root queue and give appropriate capacity in % for

S
each queue.

ON
Illustration

TI
Below properties will be defined in [Link]

ES
W s
<property>
IE tion
<name>[Link]</name>
<value>research,marketing,finance</value>
QU
RV ues

</property>
TE Q

<property>
<name>[Link]</name>
IN ple

<value>60</value>
</property>
TA am
DA S

<property>
<name>[Link]</name>
<value>30</value>
</property>

<property>
IG

<name>[Link]</name>
<value>10</value>
.B

</property>
W
W

3. How do you benchmark your Hadoop cluster with tools that come with Hadoop?
W

 TestDFSIO

TestDFSIO gives you an understanding of the I/O performance of your cluster. It is a read and write test for HDFS
and helpful in identifying performance bottlenecks in your network, hardware and set up of your NameNode and
DataNodes.

 NNBench

NNBench simulate requests for creating, reading, renaming and deleting files on HDFS and is useful for load testing
NameNode hardware configuration
[Link] HADOOP DEVELOPER INTERVIEW GUIDE

 MRBench

MRBench is a test for the MapReduce layer. It loops a small MapReduce job for a specific number of times and
checks the responsiveness and efficiency of the cluster.

Illustration

TestDFSIO write test with 100 files and file size of 100 MB each.

$ hadoop jar /dirlocation/[Link] TestDFSIO -write -nrFiles 100 -fileSize 100

OM
TestDFSIO read test with 100 files and file size of 100 MB each.

.C
$ hadoop jar /dirlocation/[Link] TestDFSIO -read -nrFiles 100 -fileSize 100

S
ON
MRBench test to run a lob of 50 small test jobs

$ hadoop jar /dirlocation/[Link] mrbench -numRuns 50

TI
ES
NNBench test that creates 1000 files using 12 maps and 6 reducers.
W s
IE tion
QU
$ hadoop jar /dirlocation/[Link] nnbench -operation create_write \
-maps 12 -reduces 6 -blockSize 1 -bytesToWrite 0 -numberOfFiles 1000 \
RV ues

-replicationFactorPerFile 3
TE Q

4. Assume you are doing a join and you notice that all but one reducer is running for a
IN ple

long time how do you address the problem in Pig?

Pig collects all of the records for a given key together on a single reducer. In many data sets, there are a few keys
DA S

that have three or more orders of magnitude more records than other keys. This results in one or two reducers that
will take much longer than the rest. To deal with this, Pig provides skew join.

 In the first MapReduce job pig scans the second input and identifies keys that have so many records.
 In the second MapReduce job, it does the actual join.

IG

For all except the records with the key(s) identified from the first job, pig would do a standard join.
 For the records with keys identified by the second job, bases on how many records were seen for a given
.B

key, those records will be split across appropriate number of reducers.

to each reducer that contains that key

Illustration

jnd = join cinfo by city, users by city using 'skewed';

5. What is the difference between SORT BY and ORDER BY in Hive?

ORDER BY performs a total ordering of the query result set. This means that all the data is passed through a single
reducer, which may take an unacceptably long time to execute for larger data sets.
[Link] HADOOP DEVELOPER INTERVIEW GUIDE

SORT BY orders the data only within each reducer, thereby performing a local ordering, where each reducer’s output
will be sorted. You will not achieve a total ordering on the dataset. Better performance is traded for total ordering.

6. Assume you have a sales table in a company and it has sales entries from salesman
around the globe. How do you rank each salesperson by country based on their sales
volume in Hive?

Hive support several analytic functions and one of the functions is RANK() and it is designed to do this operation.

OM
Lookup details on other window and analytic functions -
[Link]

.C
Illustration

S
ON
Hive>SELECT
rep_name, rep_country, sales_volume,

TI
rank() over (PARTITION BY rep_country ORDER BY sales_volume DESC) as rank
FROM

ES
salesrep;
W s
IE tion
QU
RV ues
TE Q
IN ple
TA am
DA S
IG
.B
W
W
W