Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Using -text command and creating multiple files
Top Forums Shell Programming and Scripting Using -text command and creating multiple files Post 302974700 by cplusplus1 on Thursday 2nd of June 2016 11:39:17 AM
Old 06-02-2016
Using -text command and creating multiple files
Currently using the below script to being all compressed files .gz files from source folder and appending to the target txt file uncompressed.
Teh target txt file is getting too large in size, right now the size of the target txt file is almost 350GB


Code:
hadoop fs -text /user/hive/warehouse/stage.db/CLINICAL_EVENT/CLINICAL_EVENT* | hadoop fs -put - /user/hive/warehouse/stage.db/Clinical_event/final/clinical_event.txt



Is there a way to create multiple files at the time of executing the above -text script.

while appending just want to maintain each file of size max 5GB?

as long as the files all are in folder Final, then the hadoop will automatically read.

is there way to create the files like:

Code:
clinical_event_1.txt
clinical_event_2.txt
clinical_event_3.txt

so on so forth.


Thanks a lot for the helpful info.

Thank you.
Last edited by RudiC; 06-02-2016 at 01:22 PM.. Reason: changed icode to code tags.
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

grep multiple text files in folder into 1 text file?

How do I use the grep command to take mutiple text files in a folder and make one huge text file out of them. I'm using Mac OS X and can not find a text tool that does it so I figured I'd resort to the BSD Unix CLI for a solution... there are 5,300 files that I want to write to one huge file so... (7 Replies)
Discussion started by: coppertone
7 Replies

2. Shell Programming and Scripting

Dynamically creating text files using shell script

Hi All, I want to create a shell script which dynamically create text files. i am using the following script $i=1 while do cat > test_$i.txt done but while running the script it was stopping(the cursor not going to next step, i have to enter ctrl+c to make it stop). it is creating only... (2 Replies)
Discussion started by: KiranKumarKarre
2 Replies

3. Shell Programming and Scripting

Replacing text from multiple files at multiple location

Hi, I have many files scattered in all different folders. I want to replace the text within all the files using a single command ( awk, sed...) Is it possible? example find all the files in which there is text "memory" and replace it with "branded_memories". the files can be at the... (2 Replies)
Discussion started by: rudoraj
2 Replies

4. Shell Programming and Scripting

creating a multiple zip files

Hi, Please help me, i need a single line command for te below steps 1: listing a directory 2: searching a particular pattern in the names and creating the zip files of each eg : ls -lrt | grep jil | awk '{print $9}' output is kap.jil pranabjil pranabjil1 pranabjil2... (1 Reply)
Discussion started by: pranabrana
1 Replies

5. UNIX for Dummies Questions & Answers

Grep multiple strings in multiple files using single command

Hi, I will use below command for grep single string ("osuser" is search string) ex: find . -type f | xarg grep -il osuser but i have one more string "v$session" here i want to grep in which file these two strings are present. any help is appreciated, Thanks in advance. Gagan (2 Replies)
Discussion started by: gagan4599
2 Replies

6. UNIX for Dummies Questions & Answers

Changing text in multiple files, but with different text for each file

Hello, I have a situation where I want to change a line of text in multiple files, but the problem is that I want to change the text to something unique for each file. For example, let's say I have five files named bob.txt, joe.txt, john.txt, tom.txt, and zach.txt. Each of these files has a... (5 Replies)
Discussion started by: Scatterbrain26
5 Replies

7. Shell Programming and Scripting

Creating single pattern for matching multiple files.

Hi friends, I have a some files in a directory. for example 856-abc 856-def 851-abc 945-def 956-abc 852-abc i want to display only those files whose name starts with 856* 945* and 851* using a single pattern. i.e 856-abc 856-def 851-abc 945-def the rest of the two files... (2 Replies)
Discussion started by: Little
2 Replies

8. Shell Programming and Scripting

Split a text file into multiple text files?

I have a text file with entries like 1186 5556 90844 7873 7722 12 7890.6 78.52 6679 3455 9867 1127 5642 ..N so many records like this. I want to split this file into multiple files like cluster1.txt, cluster2.txt, cluster3.txt, ..... clusterN.txt. (4 Replies)
Discussion started by: sammy777
4 Replies

9. Shell Programming and Scripting

Creating Multiple data files with spaces in the filename

Hi, I have a list of filenames in the format with Spaces in the filename. As an example : Sample File 1.txt Sample File 2.txt Sample File 3.txt.....I have about 100 files like this. I am trying to create a block of code or use an available command to a) Create a file b) Put in some... (2 Replies)
Discussion started by: ban3rj33
2 Replies

10. UNIX for Beginners Questions & Answers

Creating multiple empty files with touch

Hello, sorry to bother anyone reading this I have an assignment with a question that reads: Your current directory is stenton. Create empty files called f1, f2, and f12 (in that order), within stenton So my first thought was to enter: touch f1 f2 f12 but that does not work, does anyone... (1 Reply)
Discussion started by: eleuin
1 Replies

LEARN ABOUT DEBIAN

trcs

trcs(1) 							   User Manuals 							   trcs(1)

NAME

       trcs - convert text files from one codeset to other

SYNOPSIS

       trcs -C | -l | -h

       trcs [-s] [-f input-codeset ] [-t output-codeset ] [ file ]...

DESCRIPTION

       trcs  is a program for translation of text files from one codeset to other.  When exact translation is impossible it tries to make approxi-
       mations.  For example pseudographic characters are approximated by ASCII characters in codesets that have no pseudographic  characters  and
       characters with diacritical marks are approximated by existing characters.

       If the option -s or --gen-script is given trcs will generate simple script that invokes tr(1) This is useful if you want to include this in
       your script or for faster conversion of many files.

       If none of -C, -s, -l, -h and their equivalents is given each file will be read assuming it is coded with codeset input-codeset On standard
       output  it will be recoded so to use codeset output-codeset If there is no file given trcs will act as filter.  Both input-codeset and out-
       put-codeset are case insensitive and default to `cp1251'.

OPTIONS

       -C, --copyright
	      Display copying conditions and warranty information.

       -s, --gen-script
	      Generates conversion script for given input and output encodings instead to convert text files.

       -l, --list
	      List all known codesets with their aliases.

       -h, --help
	      Display this help and exit.

       -f input-codeset, --from[=]input-codeset
	      Codeset of the source.

       -f output-codeset, --to[=]output-codeset
	      Codeset of the output.

EXAMPLES

       trcs -fcyrillic -t koi8-r letter.txt
	      Convert letter.txt from ISO-8859-5 (cyrillic) to KOI8-R and output the result on standard output.

       trcs --from cp437 --to=mik --gen-script >ibm2mik
	      Generates script ibm2mik for converting from CP437 to MIK.

       sh ibm2mik screen.txt
	      Convert screen.txt from CP437 to MIK.  Output the result on standard output.  Without parameters ibm2mik acts as filter.

AUTHOR

       Anton Zinoviev <anton@lml.bas.bg>, <zinoviev@fmi.uni-sofia.bg>

SEE ALSO

       tr(1)

Linux								     JUN 2000								   trcs(1)
All times are GMT -4. The time now is 11:21 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy