Saturday, 28 October 2017

Read & Write Data to HDFS using Java Api Programs

Reading data from and writing data to Hadoop Distributed File System (HDFS) can be done in a many ways. Now let us start by using the FileSystem API to create and write to a file in HDFS, followed by an application to read a file from HDFS and write it back to the local file system.


Program 1 : Display the files from hadoop system using UrlStreamHandlerfactory...




import java.io.IOException;
import java.io.InputStream;
import java.net.URL;

import org.apache.hadoop.fs.FsUrlStreamHandlerFactory;
import org.apache.hadoop.io.IOUtils;

public class URLCat {
 static { 
  URL.setURLStreamHandlerFactory(new FsUrlStreamHandlerFactory());
  }
 public static void main(String[] args) throws Exception
 {
  InputStream in = null;
  try
  {
   in = new URL(args[0]).openStream();
   IOUtils.copyBytes(in,System.out,4096,false);
  }
  finally
  {
   
   IOUtils.closeStream(in);
  }
 }
}



OUTPUT :
administrator@dev143:/usr/local/hadoop/sbin$ hadoop URlCat hdfs://localhost:9000/employee/output.txt
17/08/17 11:38:20 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

“This is hadoop tutorial
Display file using using UrlStreamHandlerfactory java api from hdfs”







Program 2 : Display files from a hadoop filesystem on standard output by using filesystem...





import java.io.IOException;
import java.io.InputStream;
import java.net.URI;

import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.io.IOUtils;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.conf.Configuration;



public class FileSystemCat 
{
 
 public static void main(String[] args) throws Exception
 {
  String uri = args[0];
  Configuration conf = new Configuration();
  FileSystem fs = FileSystem.get(URI.create(uri), conf); 
  InputStream in = null;
  try
  {
   in = fs.open(new Path(uri));
   IOUtils.copyBytes(in,System.out,4096,false);
  }
  finally
  {
   
   IOUtils.closeStream(in);
  }
 }
}



OUTPUT :
administrator@dev143:/usr/local/hadoop/sbin$ hadoop FileSystemCat hdfs://localhost:9000/employee/output.txt17/08/17 11:43:38 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

“This is hadoop tutorial
standard output by using filesystem using java api from hdfs”




.

Program 3 : Displaying files from hadoop filesystem on standard output using read().




import java.io.IOException;
import java.io.InputStream;
import java.net.URI;

import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.io.IOUtils;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.conf.Configuration;

public class FileSystemReadCat
{
 public static void main(String args[]) throws Exception
 {
  
  String uri = args[0];
  byte[] buffer=new byte[256];
  Configuration conf = new Configuration();
  FileSystem fs = FileSystem.get(URI.create(uri), conf); 
  FSDataInputStream in = null;
  try
  {
   in = fs.open(new Path(uri));
   in.read(buffer); 
   String st=new String(buffer);
   System.out.println(st); 
      
  }
  finally
  {
   
   IOUtils.closeStream(in);
  }
 }
}




OUTPUT :
administrator@dev143:/usr/local/hadoop/sbin$ hadoop FileSystemReadCat hdfs://localhost:9000/employee/output.txt17/08/17 11:47:29 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

“This is hadoop tutorial
standard output using read()using java api from hdfs”



Program 4 : How To get the current position of the curser…




import java.io.IOException;
import java.io.InputStream;
import java.net.URI;

import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.io.IOUtils;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.conf.Configuration;

public class FileGetPos 
{
 
 public static void main(String[] args) throws Exception
 {
  String uri = args[0];
  Configuration conf = new Configuration();
  FileSystem fs = FileSystem.get(URI.create(uri), conf); 
  FSDataInputStream in = null;
  long tmp=0l;
  try
  {
   in = fs.open(new Path(uri));
   IOUtils.copyBytes(in,System.out,4096,false);
   
   tmp=in.getPos();//get the current position...
   System.out.println(tmp);   
  }
  finally
  {
   
   IOUtils.closeStream(in);
  }
 }
}


OUTPUT :
hadoop FileGetPos hdfs://localhost:9000/Employee/student.txt17/08/11 11:29:54 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
NAME : Dev Patel

ROLL NO : 48811113

STD : MSc SEM-1

59


Program 5 : To print the output for 2 times...




import java.io.IOException;
import java.io.InputStream;
import java.net.URI;

import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.io.IOUtils;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.conf.Configuration;

public class FileSeek 
{
 
 public static void main(String[] args) throws Exception
 {
  String uri = args[0];
  Configuration conf = new Configuration();
  FileSystem fs = FileSystem.get(URI.create(uri), conf); 
  FSDataInputStream in = null;
  try
  {
   in = fs.open(new Path(uri));
   IOUtils.copyBytes(in,System.out,4096,false);
  
   in.seek(0); //go back to the start of the file...
   IOUtils.copyBytes(in,System.out,4096,false);
   
  }
  finally
  {
   
   IOUtils.closeStream(in);
  }
 }
}

OUTPUT :


hadoop FileSeek hdfs://localhost:9000/Employee/student.txt17/08/11 11:31:52 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
NAME : Dev Patel

ROLL NO : 48811113

STD : MSC SEM-1



NAME : Dev Patel

ROLL NO : 48811113

STD : MSC SEM-1

0 comments:

Post a Comment