Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Using -text command and creating multiple files

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Using -text command and creating multiple files
# 1  
Old 06-02-2016
Using -text command and creating multiple files
Currently using the below script to being all compressed files .gz files from source folder and appending to the target txt file uncompressed.
Teh target txt file is getting too large in size, right now the size of the target txt file is almost 350GB


Code:
hadoop fs -text /user/hive/warehouse/stage.db/CLINICAL_EVENT/CLINICAL_EVENT* | hadoop fs -put - /user/hive/warehouse/stage.db/Clinical_event/final/clinical_event.txt



Is there a way to create multiple files at the time of executing the above -text script.

while appending just want to maintain each file of size max 5GB?

as long as the files all are in folder Final, then the hadoop will automatically read.

is there way to create the files like:

Code:
clinical_event_1.txt
clinical_event_2.txt
clinical_event_3.txt

so on so forth.


Thanks a lot for the helpful info.

Thank you.
Last edited by RudiC; 06-02-2016 at 01:22 PM.. Reason: changed icode to code tags.
# 2  
Old 06-02-2016
have a look at the split command.
# 3  
Old 06-03-2016
please try with split -b 5g YOURFILE clinical_event_1txt
# 4  
Old 06-03-2016
...and put the split at the end of your pipe, so you do not need to write an intermediate 350 GB file(Will be a lot of faster because it saves you to write+read 350 GB).

Code:
hadoop bla ... |  hadoop fs -put - - | split ... outputfile

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Creating multiple empty files with touch

Hello, sorry to bother anyone reading this I have an assignment with a question that reads: Your current directory is stenton. Create empty files called f1, f2, and f12 (in that order), within stenton So my first thought was to enter: touch f1 f2 f12 but that does not work, does anyone... (1 Reply)
Discussion started by: eleuin
1 Replies

2. Shell Programming and Scripting

Creating Multiple data files with spaces in the filename

Hi, I have a list of filenames in the format with Spaces in the filename. As an example : Sample File 1.txt Sample File 2.txt Sample File 3.txt.....I have about 100 files like this. I am trying to create a block of code or use an available command to a) Create a file b) Put in some... (2 Replies)
Discussion started by: ban3rj33
2 Replies

3. Shell Programming and Scripting

Split a text file into multiple text files?

I have a text file with entries like 1186 5556 90844 7873 7722 12 7890.6 78.52 6679 3455 9867 1127 5642 ..N so many records like this. I want to split this file into multiple files like cluster1.txt, cluster2.txt, cluster3.txt, ..... clusterN.txt. (4 Replies)
Discussion started by: sammy777
4 Replies

4. Shell Programming and Scripting

Creating single pattern for matching multiple files.

Hi friends, I have a some files in a directory. for example 856-abc 856-def 851-abc 945-def 956-abc 852-abc i want to display only those files whose name starts with 856* 945* and 851* using a single pattern. i.e 856-abc 856-def 851-abc 945-def the rest of the two files... (2 Replies)
Discussion started by: Little
2 Replies

5. UNIX for Dummies Questions & Answers

Changing text in multiple files, but with different text for each file

Hello, I have a situation where I want to change a line of text in multiple files, but the problem is that I want to change the text to something unique for each file. For example, let's say I have five files named bob.txt, joe.txt, john.txt, tom.txt, and zach.txt. Each of these files has a... (5 Replies)
Discussion started by: Scatterbrain26
5 Replies

6. UNIX for Dummies Questions & Answers

Grep multiple strings in multiple files using single command

Hi, I will use below command for grep single string ("osuser" is search string) ex: find . -type f | xarg grep -il osuser but i have one more string "v$session" here i want to grep in which file these two strings are present. any help is appreciated, Thanks in advance. Gagan (2 Replies)
Discussion started by: gagan4599
2 Replies

7. Shell Programming and Scripting

creating a multiple zip files

Hi, Please help me, i need a single line command for te below steps 1: listing a directory 2: searching a particular pattern in the names and creating the zip files of each eg : ls -lrt | grep jil | awk '{print $9}' output is kap.jil pranabjil pranabjil1 pranabjil2... (1 Reply)
Discussion started by: pranabrana
1 Replies

8. Shell Programming and Scripting

Replacing text from multiple files at multiple location

Hi, I have many files scattered in all different folders. I want to replace the text within all the files using a single command ( awk, sed...) Is it possible? example find all the files in which there is text "memory" and replace it with "branded_memories". the files can be at the... (2 Replies)
Discussion started by: rudoraj
2 Replies

9. Shell Programming and Scripting

Dynamically creating text files using shell script

Hi All, I want to create a shell script which dynamically create text files. i am using the following script $i=1 while do cat > test_$i.txt done but while running the script it was stopping(the cursor not going to next step, i have to enter ctrl+c to make it stop). it is creating only... (2 Replies)
Discussion started by: KiranKumarKarre
2 Replies

10. UNIX for Dummies Questions & Answers

grep multiple text files in folder into 1 text file?

How do I use the grep command to take mutiple text files in a folder and make one huge text file out of them. I'm using Mac OS X and can not find a text tool that does it so I figured I'd resort to the BSD Unix CLI for a solution... there are 5,300 files that I want to write to one huge file so... (7 Replies)
Discussion started by: coppertone
7 Replies
Login or Register to Ask a Question