Friday, 29 September 2017

Hadoop File System Shell Commands

The File System (FS) shell includes different shell-like commands that directly interrelate with the Hadoop Distributed File System (HDFS) as well as other file systems that Hadoop supports, such as Local FS, HFTP FS, S3 FS, and others

Hadoop file system (fs) shell commands are used to perform various file operations like copying file, changing permissions, viewing the contents of the file, changing ownership of files, creating directories etc. 

ls

The hadoop ls command is used to show the directories and files. An example is shown below

c:\hadoop-2.3.0\bin>hadoop fs -ls /
Found 5 items
drwxr-xr-x   - TNC supergroup          0 2017-10-11 19:14 /mir
drwx------   - TNC supergroup          0 2017-10-11 19:58 /tmp
drwxr-xr-x   - TNC supergroup          0 2017-10-11 19:46 /wcop
drwxr-xr-x   - TNC supergroup          0 2017-10-11 20:00 /wcop1
drwxr-xr-x   - TNC supergroup          0 2017-10-11 23:58 /wcop2




ls –R

The hadoop lsr command recursively shows the directories, sub directories and files in the specified directory. The usage example is shown below: 

c:\hadoop-2.3.0\bin>hadoop fs -ls -R /wcop1
-rw-r--r--   1 TNC supergroup          0 2017-10-11 20:00 /wcop1/_SUCCESS
-rw-r--r--   1 TNC supergroup         20 2017-10-11 19:59 /wcop1/part-r-00000




Cat

Hadoop cat command is used to print the contents of the file on the terminal (stdout). The usage example of hadoop cat command is shown below: 

c:\hadoop-2.3.0\bin>hadoop fs -ls -R /wcop1
-rw-r--r--   1 TNC supergroup          0 2017-10-11 20:00 /wcop1/_SUCCESS
-rw-r--r--   1 TNC supergroup         20 2017-10-11 19:59 /wcop1/part-r-00000



Chmod

The hadoop chmod command is used to change the permissions of files. The -R option can be used to recursively change the permissions of a directory.

c:\hadoop-2.3.0\bin>hadoop fs -chmod 777 /HdoopTutorial





Mkdir

The hadoop mkdir command is for creating directories in the hdfs. You can use the -p option for creating parent directories. This is like to the unix mkdir command. The usage example is shown below: 

c:\hadoop-2.3.0\bin>hadoop fs -mkdir /HadoopTutorial

c:\hadoop-2.3.0\bin>hadoop fs -ls /
Found 6 items
drwxr-xr-x   - TNC supergroup          0 2017-10-14 20:37 /HadoopTutorial
drwxr-xr-x   - TNC supergroup          0 2017-10-11 19:14 /mir
drwx------   - TNC supergroup          0 2017-10-11 19:58 /tmp
drwxr-xr-x   - TNC supergroup          0 2017-10-11 19:46 /wcop
drwxr-xr-x   - TNC supergroup          0 2017-10-11 20:00 /wcop1
drwxr-xr-x   - TNC supergroup          0 2017-10-11 23:58 /wcop2





copyFromLocal

The hadoop copyFromLocal command is used to copy a file from the local file system to the hadoop hdfs. The syntax and usage example are shown below: 

c:\hadoop-2.3.0\bin>Hadoop fs -copyFromLocal C:\pract\hadoopTutorialbyMir.txt /H
doopTutorial/

c:\hadoop-2.3.0\bin>hadoop fs -ls -R /Hdoop*
-rw-r--r--   1 TNC supergroup         51 2017-10-14 20:40 /HdoopTutorial/hadoopT
utorial.txt
-rw-r--r--   1 TNC supergroup         51 2017-10-14 20:47 /HdoopTutorial/hadoopT
utorialbyMir.txt



copyToLocal

The hadoop copyToLocal command is used to copy a file from the hdfs to the local file system. The syntax and usage example is shown below: 

c:\hadoop-2.3.0\bin>Hadoop fs -copyToLocal /HdoopTutorial/hadoopTutorial.txt C:\
pract\


Cp

The hadoop cp command is for copying the source into the destination. The cp command can also be used to copy multiple files into the destination. In this case the destination should be a directory

c:\hadoop-2.3.0\bin>hadoop fs -cp /HdoopTutorial/ /wcop/

c:\hadoop-2.3.0\bin>hadoop fs -ls -R /wcop
drwxr-xr-x   - TNC supergroup          0 2017-10-14 20:53 /wcop/HdoopTutorial
-rw-r--r--   1 TNC supergroup         51 2017-10-14 20:53 /wcop/HdoopTutorial/ha
doopTutorial.txt
-rw-r--r--   1 TNC supergroup         51 2017-10-14 20:53 /wcop/HdoopTutorial/ha
doopTutorialbyMir.txt




Rm

Delete files specified as args.
If trash is enabled, file system instead moves the deleted file to a trash directory

hadoop fs -rm /exercise2/hamlet.txt

Deletion interval = 0 minutes, Emptier interval = 0 minutes.
Deleted /exercise2/hamlet.txt

hadoop fs -rm -r /exercise1

Emptier interval = 0 minutes.
Deleted /exercise1



AppendToFile

Append single src, or multiple srcs from local file system to the destination file system. Also reads input from stdin and appends to target file system.

hadoop fs -appendToFile /exercise1/hamlet.txt /home/administrator/$PLAY_AREA/exercises/filesystem/hamlet.txt



Put

Hadoop put command is used to copy multiple sources to the destination system.

c:\hadoop-2.3.0\bin>Hadoop fs -put C:/pract/sample.txt /Tutorial



Get

get

Hadoop get command copies the files from hdfs to the local file system. The syntax of the get command is shown below

hadoop fs -get /user/hadoop/file localfile




Getmerge

hadoop getmerge command concatenates the files in the source directory into the destination file.

hadoop fs -getmerge -nl /exercise2/stu*.txt /home/administrator/Desktop/mergeAll.txt /home/administrator/stu.txt



Du

The hadoop dus command prints the summary of file lengths .

c:\hadoop-2.3.0\bin>hadoop fs -du -s /Tutorial
51  /Tutorial



Setrep

Hadoop setrep is used to change the replication factor of a file. Use the -R option for recursively changing the replication factor.

c:\hadoop-2.3.0\bin>hadoop fs -setrep -w 4 -R /Tutorial
Replication 4 set: /Tutorial/hadoopTutorial.txt
Waiting for /Tutorial/hadoopTutorial.txt .....



Stat

Hadoop stat returns the stats information on a path. The syntax of stat is shown below: 

c:\hadoop-2.3.0\bin>Hadoop fs -stat /Tutorial
2017-10-14 15:31:29



Tail

Hadoop tail command prints the last lines of the file. The -f option can be used same as in unix

c:\hadoop-2.3.0\bin>Hadoop fs -tail /Tutorial/sample.txt
This is hadoop tutorial By Mir
-Tech News n Career





Touchz

The hadoop touchz command creates a zero byte file. This is similar to the touch command in unix. The syntax is shown below: 

c:\hadoop-2.3.0\bin>Hadoop fs -touchz /Tutorial/sample1.txt

c:\hadoop-2.3.0\bin>hadoop fs -ls /Tu*
Found 3 items
-rw-r--r--   4 TNC supergroup         51 2017-10-14 20:40 /Tutorial/hadoopTutori
al.txt
-rw-r--r--   1 TNC supergroup         51 2017-10-14 21:12 /Tutorial/sample.txt
-rw-r--r--   1 TNC supergroup          0 2017-10-14 21:14 /Tutorial/sample1.txt




Mv

Moves files from source to destination. This command allows multiple sources as well in which case the destination needs to be a directory. Moving files across file systems is not permitted.

hadoop fs -mv /exercise1/hamlet.txt /exercise2

drwxr-xr-x   - administrator supergroup          0 2017-07-15 17:12 /exercise1
-rw-r--r--   1 administrator supergroup       2276 2017-07-15 16:44 /exercise1/hamlet_hdfsCopy.txt
drwxr-xr-x   - administrator supergroup          0 2017-07-15 17:12 /exercise2
-rw-r--r--   3 administrator supergroup       2276 2017-07-15 16:27 /exercise2/hamlet.txt

0 comments:

Post a Comment