Saturday, 11 November 2017

Program of Maximum/Minimum Temperature in Weather Dataset in Hadoop Mapreduce Programme

Program of Maximum/Minimum Temperature in weather Dataset

Download or create sample weather Dataset.

We have to create 3 java file called
MapReduce is based on set of key value pairs. So first we have to decide on the types for the key/value pairs for the input.
Map Phase: The input for Map segment is set of weather dataset. The types of input key value pairs are LongWritable and Text and the types of output key value pairs are Text and IntWritable. Each Map task extracts the temperature data from the given year file. The output of the map segment is set of key value pairs. Set of keys are the years. Values are the temperature of each year.

Reduce Phase: Reduce phase obtains all the values linked with a particular key. That is all the temperature values belong to a particular year is fed to a same reducer. Then each reducer finds the highest recorded temperature for each year. The types of output key value pairs in Map phase is same for the types of input key value pairs in reduce phase (Text and IntWritable). The types of output key value pairs in reduce phase is also Text and IntWritable.


maxTemperature.java,
maxTemperatureMapper.java,
maxTemperatureReducer.java

maxTemperature.java


import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;
import org.apache.hadoop.conf.Configuration;


public class MaxTemperature
{
 public static void main (String args[]) throws Exception
 {
  if (args.length!=2)
  {
  System.err.println("Use : MaxTemperature -inputpath- -outputpath-");
  System.exit(-1);
  }
  Configuration conf = new Configuration ();
  Job job = new Job(conf,"MaxTemperature");
  job.setJarByClass(MaxTemperature.class);
  job.setJar("MaxTemperature.jar");
  
  //Job job = new Job(conf, "Temperature");
  //job.setJarByClass(MaxTemperature.class);
  job.setJobName("Max Temperature");
  job.setJobName("Max Temperature");
  
  FileInputFormat.addInputPath(job, new Path (args[0]));
  FileOutputFormat.setOutputPath(job, new Path(args[1]));
 
  job.setMapperClass(MaxTempMapper.class);
  job.setReducerClass(MaxTempReducer.class);


  job.setInputFormatClass(TextInputFormat.class);
  job.setOutputFormatClass(TextOutputFormat.class);
  job.setMapOutputKeyClass(Text.class);
  job.setMapOutputValueClass(IntWritable.class);
  
  job.setOutputKeyClass(Text.class);
  job.setOutputValueClass(IntWritable.class);
  

  System.exit(job.waitForCompletion(true) ? 0 : 1);
 }
}


maxTemperatureMapper.java


import java.io.IOException;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;
public class MaxTempMapper extends Mapper
{
 
  public void map(LongWritable key,Text value,Context context)throws IOException,InterruptedException
  {
   String line=value.toString();
   String year=line.substring(0,4);
   int airtemp;
   if(line.charAt(47)=='+')
    airtemp=Integer.parseInt(line.substring(48,50));
   else
    airtemp=Integer.parseInt(line.substring(47,49));
    
   context.write(new Text(year),new  IntWritable(airtemp));


   
 
  }

}


maxTemperatureReducer.java


import java.io.IOException;

import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Reducer;
//import org.apache.hadoop.mapreduce.lib.input.*;

public class MaxTempReducer extends Reducer  
{
 public void reduce(Text key, Iterable values, Context context)
  throws IOException, InterruptedException
 {
  int maxValue = Integer.MIN_VALUE;
  for ( IntWritable value : values)
  {
   maxValue = Math.max (maxValue, value.get());
  }
  context.write (key, new IntWritable(maxValue));
 }
}



When You compile programme map ane reduce job will be complate 100% and successfull complition,
Output  :

hadoop jar MaxTemp.jar MaxTemp /Sample.txt /Out3

 hadoop fs -cat /Out3/part-r-00000
16/08/05 14:24:24 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2010   0
2011   6
2012   5
2013   6
2014   11
2015   23
2016   24


0 comments:

Post a Comment