Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Splitting large file into multiple files in unix based on pattern

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Splitting large file into multiple files in unix based on pattern
# 8  
Old 07-05-2011
Try:
Code:
awk -v ll=`wc -l < file` 'NR>1 && NR<ll{f=substr($1,length($1)-2)".txt";print > f;close(f)}' file > newfile

# 9  
Old 07-05-2011
try with sed Smilie
Code:
# for((i=1;i<$(sed -n '$=' file);i++));do sed -n "$i p" file > $(sed -n "$i s/^.\{10\}\(...\).*/\1/p" file).txt;done

# 10  
Old 07-05-2011
Use nawk on Solaris.

Franklin's code needs just a little modification:

Code:
nawk -v ll=$(wc -l < test) 'NR>1 && NR<ll{f=substr($1,length($1)-2)".txt"; print > f}' test

# 11  
Old 07-06-2011
---------- Post updated at 11:38 PM ---------- Previous update was at 10:19 PM ----------

Hi
None of the commands are working all having same error error: `(' unexpected
Below is requirement:
Code:
txtytg09dfgdfg
qwerty0101TWE 12345 01022005 01022005 datainala alanfernanded 26
qwerty0101mXZ 12349 01022005 06022008 datainalb johngalilo 28 
qwerty0101TWE 12342 01022005 07022009 datainalc hitalbert 43 
qwerty0101CFG 12345 01022005 01022005 datainala alanfernanded 26
qwerty0101mXZ 12349 01022005 06022008 datainalb johngalilo 28 
qwerty0101CFG 12342 01022005 07022009 datainalc hitalbert 43
byetekr 09

In the above file I have to skip the first&last line i.e not to the process these lines.
Based on the last three characters of the first field of the second record
qwerty0101TWE i.e. TWE I want to put this record in file TWE.txt
then next record mxz in MXZ.txt.
With the above queries it is creating files for first and last line also
like dgf.txt anf kr0.txt> I don't to create this files.
Also need to keep the input file as it is.
Plzz help its urgent!!!!

---------- Post updated at 11:38 PM ---------- Previous update was at 11:38 PM ----------
Last edited by Franklin52; 07-06-2011 at 03:56 AM.. Reason: Please use code tags for code and data samples, thank you
# 12  
Old 07-06-2011
Code:
 
$nawk '{f=substr($1,length($1)-2)".txt";print $2,$3,$4,$5,$6 >> f;close(f)}' test
$
$ cat TWE.txt
12345 01022005 01022005 datainala alanfernanded
12342 01022005 07022009 datainalc hitalbert
$ cat mXZ.txt
12349 01022005 06022008 datainalb johngalilo
12349 01022005 06022008 datainalb johngalilo
$ cat CFG.txt
12345 01022005 01022005 datainala alanfernanded
12342 01022005 07022009 datainalc hitalbert

# 13  
Old 07-06-2011
Sample file
Code:
$ cat test
nala alanfernanded 26
qwerty0101mXZ 12349 01022005 06022008 datainalb johngalilo 28
qwerty0101TWE 12342 01022005 07022009 datainalc hitalbert 43
qwerty0101CFG 12345 01022005 01022005 datainala alanfernanded 26
qwerty0101mXZ 12349 01022005 06022008 datainalb johngalilo 28
qwerty0101CFG 12342 01022005 07022009 datainalc hitalbert 43
byetekr 09

Command:
Code:
nawk '{f=substr($1,length($1)-2)".txt";print $2,$3,$4,$5,$6 >> f;close(f)}' test

its creating files as

ala.txt -- created for first file
ekr.txt -- created for last file n other files too
Actually don't want to create files for 1st n last line
also there are n no. of records n not just 6 I don't want to hard code anything

Code:
$ cat TWE.txt
12342 01022005 07022009 datainalc hitalbert

I do want the qwerty0101mXZ field i.e first field in the output file
this command works fine

Code:
awk '{f=substr($1,length($1)-2)".txt";print > f;close(f)}' test

but want to modify cmd to skip the first and line for processing
Last edited by Franklin52; 07-06-2011 at 03:56 AM.. Reason: Please use code tags for code and data samples, thank you
# 14  
Old 07-06-2011
Code:
 
$ lineno=`wc -l < test`; nawk -v lineno="$lineno" '{ if (NR>1 && NR < lineno){f=substr($1,length($1)-2)".txt";print >> f;close(f)}}' test

$ cat TWE.txt
qwerty0101TWE 12342 01022005 07022009 datainalc hitalbert 43

$ cat mXZ.txt
qwerty0101mXZ 12349 01022005 06022008 datainalb johngalilo 28
qwerty0101mXZ 12349 01022005 06022008 datainalb johngalilo 28

$ cat CFG.txt
qwerty0101CFG 12345 01022005 01022005 datainala alanfernanded 26
qwerty0101CFG 12342 01022005 07022009 datainalc hitalbert 43

$ cat test
nala alanfernanded 26
qwerty0101mXZ 12349 01022005 06022008 datainalb johngalilo 28
qwerty0101TWE 12342 01022005 07022009 datainalc hitalbert 43
qwerty0101CFG 12345 01022005 01022005 datainala alanfernanded 26
qwerty0101mXZ 12349 01022005 06022008 datainalb johngalilo 28
qwerty0101CFG 12342 01022005 07022009 datainalc hitalbert 43
byetekr 09
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

UNIX script to append multiple text files into one file based on pattern present in filaname

Hi All-I am new to Unix , I need to write a script. Can someone help me with a requirement where I have list of files in a directory, I want to Merge the files if a pattern of string matches in filenames? AAAL_555A_ORANGE1_F190404.TXT AAAL_555A_ORANGE2_F190404.TXT AAAL_555A_ORANGE3_F190404.TXT... (6 Replies)
Discussion started by: Shankar455
6 Replies

2. UNIX for Advanced & Expert Users

Concatenation of multiple files based on file pattern

Hi, I have the following reports that get generated every 1 hour and this is my requirement: 1. 5 reports get generated every hour with the names "Report.Dddmmyy.Thhmiss.CTLR" "Report.Dddmmyy.Thhmiss.ACCD" "Report.Dddmmyy.Thhmiss.BCCD" "Report.Dddmmyy.Thhmiss.CCCD"... (1 Reply)
Discussion started by: Jesshelle David
1 Replies

3. Shell Programming and Scripting

Help with Splitting a Large XML file based on size AND tags

Hi All, This is my first post here. Hoping to share and gain knowledge from this great forum !!!! I've scanned this forum before posting my problem here, but I'm afraid I couldn't find any thread that addresses this exact problem. I'm trying to split a large XML file (with multiple tag... (7 Replies)
Discussion started by: Aviktheory11
7 Replies

4. Shell Programming and Scripting

Splitting textfile based on pattern and name new file after pattern

Hi there, I am pretty new to those things, so I couldn't figure out how to solve this, and if it is actually that easy. just found that awk could help:(. so i have a textfile with strings and numbers (originally copy pasted from word, therefore some empty cells) in the following structure: SC... (9 Replies)
Discussion started by: luja
9 Replies

5. Shell Programming and Scripting

Sed: Splitting A large File into smaller files based on recursive Regular Expression match

I will simplify the explaination a bit, I need to parse through a 87m file - I have a single text file in the form of : <NAME>house........ SOMETEXT SOMETEXT SOMETEXT . . . . </script> MORETEXT MORETEXT . . . (6 Replies)
Discussion started by: sumguy
6 Replies

6. Shell Programming and Scripting

Help needed - Split large file into smaller files based on pattern match

Help needed urgently please. I have a large file - a few hundred thousand lines. Sample CP START ACCOUNT 1234556 name 1 CP END ACCOUNT CP START ACCOUNT 2224444 name 1 CP END ACCOUNT CP START ACCOUNT 333344444 name 1 CP END ACCOUNT I need to split this file each time "CP START... (7 Replies)
Discussion started by: frustrated1
7 Replies

7. Shell Programming and Scripting

Help me pls : splitting single file in unix into different files based on data

I have a file in unix with sample data as follows : -------------------------------------------------------------- -------------------------------------------------------------- {30001002|XXparameter|Layout|$ I want this file to be splitted into different files and corresponding to the sample... (54 Replies)
Discussion started by: Ravindra Swan
54 Replies

8. Shell Programming and Scripting

Splitting large file and renaming based on field

I am trying to update an older program on a small cluster. It uses individual files to send jobs to each node. However the newer database comes as one large file, containing over 10,000 records. I therefore need to split this file. It looks like this: HMMER3/b NAME 1-cysPrx_C ACC ... (2 Replies)
Discussion started by: fozrun
2 Replies

9. Shell Programming and Scripting

Problem with splitting large file based on pattern

Hi Experts, I have to split huge file based on the pattern to create smaller files. The pattern which is expected in the file is: Master..... First... second.... second... third.. third... Master... First.. second... third... Master... First... second.. second.. second..... (2 Replies)
Discussion started by: saisanthi
2 Replies

10. Shell Programming and Scripting

awk - splitting 1 large file into multiple based on same key records

Hello gurus, I am new to "awk" and trying to break a large file having 4 million records into several output files each having half million but at the same time I want to keep the similar key records in the same output file, not to exist accross the files. e.g. my data is like: Row_Num,... (6 Replies)
Discussion started by: kam66
6 Replies
Login or Register to Ask a Question