Login or Register to Ask a Question and Join Our Community


UNIX for Beginners Questions & Answers

Conditional Split

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums UNIX for Beginners Questions & Answers Conditional Split
# 1  
Old 04-28-2018
Conditional Split
Greetings,

I need help in splitting the files in an efficient way while accommodating the below requirements . I am on AIX.

Split condition

Split the file based on the record type and the position of the data pattern that appears on the on the record type.

Both record type and and the position of the data pattern should be parameter to the script

The header and trailer records on the splitted files may or may not be needed and and I want to control them by passing a parameter to the script.

Split File Name should be same as Input File Name_split value_count of total splited records


Example

Input File Name : File123

Code:
H00000000000000000000000000000000123567890
D00000000000000000000000000000ABC123567890
D00000000000000000000000000000ABC123567890
D00000000000000000000000000000XYZ123567890
D00000000000000000000000000000XYZ123567890
D00000000000000000000000000000XYZ123567890
D00000000000000000000000000000PQR123567890
D00000000000000000000000000000PQR123567890
D00000000000000000000000000000PQR123567890
D00000000000000000000000000000PQR123567890
T00000000000000000000000000000000123567890

File123_ABC_2

Code:
H00000000000000000000000000000000123567890
D00000000000000000000000000000ABC123567890
D00000000000000000000000000000ABC123567890
T00000000000000000000000000000000123567890

File123_XYZ_3

Code:
H00000000000000000000000000000000123567890
D00000000000000000000000000000XYZ123567890
D00000000000000000000000000000XYZ123567890
D00000000000000000000000000000XYZ123567890
T00000000000000000000000000000000123567890

File123_PQR_4

Code:
H00000000000000000000000000000000123567890
D00000000000000000000000000000PQR123567890
D00000000000000000000000000000PQR123567890
D00000000000000000000000000000PQR123567890
D00000000000000000000000000000PQR123567890
T00000000000000000000000000000000123567890

# 2  
Old 04-28-2018
What be the parameters that you mentioned?
# 3  
Old 04-28-2018
RudiC

Script should have the below parameters/variables

Input File Name
Record Type
Position
Header
Trailer
# 4  
Old 04-28-2018
OK - I can't see how you call the script. Try
Code:
awk '
NR == 1         {HD = $0
                 next
                }
                {SFX = substr ($0, 31, 3)
                 FN  = FILENAME "_" SFX
                 if (SFX == "000") next
                }

!(FN in CNT)    {print HD > FN
                }
                {print > FN
                 CNT[FN]++
                }
END             {for (FN in CNT)        {print > FN
                                         close (FN)
                                         print "mv " FN " " FN "_" CNT[FN]
                                        }
                }
' file | sh

and adapt to your needs.
# 5  
Old 04-28-2018
RudiC - Thanks

Can we make the position as a varaiable as well ? substr ($0, 31, 3) ?

Also greatly appreciate if can you explain how the script works.

Thanks in Advance
# 6  
Old 04-29-2018
Had you given the resp. info before, when requested to do so, the script had already had the necessary constructs. Try
Code:
awk  -vPOS=31 -vLEN=3 '                                 # run awk, pass parameters for suffix position and length
NR == 1         {HD = $0                                # save header to be printed in all output files
                 next                                   # no further action needed on this line
                }

                {SFX = substr ($0, POS, LEN)            # extract suffix from POSition, LENgth characters
                 if (SFX == "000") next                 # no further action on trailer line
                 FN  = FILENAME "_" SFX                 # compose output file name
                }

!(FN in CNT)    {print HD > FN                          # print header for new files (FN not yet registered)
                }

                {print > FN                             # print to resp. output file
                 CNT[FN]++                              # count hits for file name AND register it  
                }

END             {for (FN in CNT)        {print > FN     # print trailer record to every single out file
                                         close (FN)     # close file (could be dropped)
                                         print "mv " FN " " FN "_" CNT[FN]
                                                        # output rename commands to stdout for execution by sh
                                        }
        }
' file | sh                                             # pipe stdout to shell for rename operations

This User Gave Thanks to RudiC For This Post:
techedipro
# 7  
Old 04-29-2018
Thannks RudiC
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Split and Rename Split Files

Hello, I need to split a file by number of records and rename each split file with actual filename pre-pended with 3 digit split number. What I have tried is the below command with 2 digit numeric value split -l 3 -d abc.txt F (# Will Produce split Files as F00 F01 F02) How to produce... (19 Replies)
Discussion started by: techedipro
19 Replies

2. Shell Programming and Scripting

awk to format file with conditional split

In the awk below I am splitting $7 on the : (colon) then - (hyphen) as array a. The word chr is printed at the start of every $1 line. Next, $4 is split on the > (greater then) as array b. I am not sure how to account for the two other possibilities in $4 so the correct output is printed. Every... (3 Replies)
Discussion started by: cmccabe
3 Replies

3. Shell Programming and Scripting

awk to split one field and print the last two fields within the split part.

Hello; I have a file consists of 4 columns separated by tab. The problem is the third fields. Some of the them are very long but can be split by the vertical bar "|". Also some of them do not contain the string "UniProt", but I could ignore it at this moment, and sort the file afterwards. Here is... (5 Replies)
Discussion started by: yifangt
5 Replies

4. Shell Programming and Scripting

Conditional variables

if {-a file} if {-s file} what it will do if {-z string} what it will do please help on this (1 Reply)
Discussion started by: thelakbe
1 Replies

5. Shell Programming and Scripting

If conditional

Hi, I am new to unix and shell scripting.In my script,there is a line using the "if" conditional - if && ; then do something Here "x" is a variable holding string value.If it is not equal to a comma or a string,only then I want to enter the "if" loop. But I am getting error while... (12 Replies)
Discussion started by: abhinavsinha
12 Replies

6. UNIX for Dummies Questions & Answers

If conditional

Hi, I am new to unix and shell scripting.In my script,there is a line using the "if" conditional - if && ; then do something Here "x" is a variable holding string value.If it is not equal to a comma or a string,only then I want to enter the "if" loop. But I am getting error while... (1 Reply)
Discussion started by: abhinavsinha
1 Replies

7. UNIX for Dummies Questions & Answers

conditional

conditional is not wworking can any one figure out what goes wrong xx1=`$ORACLE_HOME/bin/sqlplus -s apps/ostgapps1 2>/dev/null << EOF WHENEVER SQLERROR EXIT 1 set head off feedback off ; WHENEVER SQLERROR EXIT SQL.SQLCODE; select count(*) from CMS_INVOICE_ALL... (2 Replies)
Discussion started by: u263066
2 Replies

8. Shell Programming and Scripting

Conditional FTP

Hi Have a Nice Day i m stuck with this quite easy task assume that i m in certain directory and i open a ftp connection from there to some other directory and in second directory there are like so many files from july 02 till today what would be the simple unix command to ftp all files from this... (2 Replies)
Discussion started by: Dastard
2 Replies

9. UNIX for Dummies Questions & Answers

Split a file with no pattern -- Split, Csplit, Awk

I have gone through all the threads in the forum and tested out different things. I am trying to split a 3GB file into multiple files. Some files are even larger than this. For example: split -l 3000000 filename.txt This is very slow and it splits the file with 3 million records in each... (10 Replies)
Discussion started by: madhunk
10 Replies

10. Shell Programming and Scripting

conditional split

Hi, Can someone let me know how I can split a record when it contains a vairable length of fields. Line1 field1,field101,field102,field 103,field104 Line 2 field1,field101,field102,field 103,field104,field201,field202,field 203,field204 Line 3 field1,field101,field102,field... (5 Replies)
Discussion started by: braindrain
5 Replies
Login or Register to Ask a Question