File splitting script help


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers File splitting script help
# 1  
Old 05-06-2015
Question File splitting script help

Hi All,

I have file in my system with below data

Code:
PP1234512345671234567CABC
PP1234512345671234567BABC
PP1234512345671234567BABC
PP1234512345671234567CABC
PP1234512345671234567DABC
PP1234512345671234567EABC
PP1234512345671234567DABC
PP1234512345671234567EABC
PP1234512345671234567DABC

Above is sample file.

Inside that file at postion 21 and length 1 of each line we have record type indicator like C,D,B,E

I'm trying to create on unix program which will check the record type indicator and store in different file

So here all records with record type indicator as 'C' will should be store in file C.txt and similar to other record type.
Code:
PP1234512345671234567CABC
PP1234512345671234567CABC

Script should supply position and length as arguments
Code:
Position=$1
Length=$2

I tried below code and seems its not good and not working too Smilie
Code:
position=$1 
Length=$2
for line in $(cat file.txt) ; do  echo ${line:$position:$Length} >> recordType.dat; done
sort -u recordType.dat > RecordType.txt
for line in $(cat file.txt) ;
do
for Rec in $(cat RecordType.txt) ;
do
awk -F '' '"$position=$Rec"' $line > FILE_$REC.txt
done
done

Please help

Regards
Ibrar Ahmad
Moderator's Comments:
Mod Comment Please use CODE tags (not ICODE tags) for multi-line sample input, output, and code segments.

Last edited by Don Cragun; 05-06-2015 at 04:31 AM.. Reason: Change ICODE tags to CODE tags.
# 2  
Old 05-06-2015
How many different output files (maximum) will need to be created from a single input file? (If the answer is no more than nine, the solution is much simpler and faster on many systems.)

What operating system are you using? And, what version(s) of awk are available?
# 3  
Old 05-06-2015
Moderator's Comments:
Mod Comment This thread has been moved into a technical forum for discussion (from an administrative forum).
# 4  
Old 05-06-2015
Your approach is a bit overcomplicated and suboptimal. Why use all those for loops to extract and sort when finally all data end up in some few text files? Why use shell builtins and awk in parallel (a new awk process is created rectype x lines times! Each overwriting FILE_$REC)? Decide what to use, and the result will be a simple one liner.
# 5  
Old 05-06-2015
Hi Rudic,

I tried below code and its working fine as per my expectation in awk

Code:
awk ' { print $1 } ' file.txt  | cut -c 22-22 | uniq | awk ' { print "grep \"[0-9]"$0"\" file.txt > "$0".txt" } ' | sh

It split file based on requirement but Is there any way where I can use variable on in second awk command
Code:
Filename=file.txt
Position=22
NextPostion=22
TargetfilePrefix=PPlus_GIC
 
awk ' { print $1 } ' $Filename | cut -c $Position-$NextPostion | uniq | awk ' { print "grep \"[0-9]"$0"\" $Filename > $TargetfilePrefix_"$0".txt" } ' | sh
 
Above awk is not working 
I believe below command is not accepting variable 
awk ' { print "grep \"[0-9]"$0"\" $Filename > $TargetfilePrefix_"$0".txt" } '

can you please suggest a way forward

Regards
Ibrar Ahmad

---------- Post updated at 07:14 AM ---------- Previous update was at 07:11 AM ----------

Hi Don,

I'm not sure about number of file or record type coming is file, but Lets assume its less then 9 or more then 9 (both case)

I'm using cygwin do perform this activity and using windows machine.


Regards
Ibrar Ahmad

Last edited by Corona688; 05-06-2015 at 12:04 PM..
# 6  
Old 05-06-2015
You can't use shell variables in an awk script; although, awk has mechanisms to pass values to internal variables. See man awk.
Your new awk is way better than what you did in post#1, but still has some potential.
Try
Code:
while read line; do echo $line >> ${TargetfilePrefix}_${line:$Position-1:$NextPostion-$Position+1}.txt; done < $Filename

or
Code:
awk -v POS="$Position" -v NXT="$NextPostion" -v PRF="$TargetfilePrefix" '{print >  PRF "_" substr($0, POS, NXT-POS+1) ".txt"}'  $Filename

If chances are there will be > 9 (or any largest open file count) output files, you'll need to close them after each print and use >> for an append operation in awk.

Last edited by RudiC; 05-06-2015 at 12:56 PM..
This User Gave Thanks to RudiC For This Post:
# 7  
Old 05-06-2015
Please use code instead of icode tags, that is the Image button.
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Splitting a text file into smaller files with awk, how to create a different name for each new file

Hello, I have some large text files that look like, putrescine Mrv1583 01041713302D 6 5 0 0 0 0 999 V2000 2.0928 -0.2063 0.0000 N 0 0 0 0 0 0 0 0 0 0 0 0 5.6650 0.2063 0.0000 N 0 0 0 0 0 0 0 0 0 0 0 0 3.5217 ... (3 Replies)
Discussion started by: LMHmedchem
3 Replies

2. Shell Programming and Scripting

Script for splitting file of records into multiple files

Hello I have a file of following format HDR 1234 abc qwerty abc def ghi jkl HDR 4567 xyz qwerty abc def ghi jkl HDR 890 mno qwerty abc def ghi jkl HDR 1234 abc qwerty abc def ghi jkl HDR 1234 abc qwerty abc def ghi jkl -Need to split this into multiple files based on tag... (8 Replies)
Discussion started by: wincrazy
8 Replies

3. Shell Programming and Scripting

Execution of loop :Splitting a single file into multiple .dat file

hdr=$(cut -c1 $path$file|head -1)#extract header”H” trl=$(cut -c|path$file|tail -1)#extract trailer “T” SplitFile=$(cut -c 50-250 $path 1$newfile |sed'$/ *$//' head -1')# to trim white space and extract table name If; then # start loop if it is a header While read I #read file Do... (4 Replies)
Discussion started by: SwagatikaP1
4 Replies

4. Shell Programming and Scripting

Splitting XML file on basis of line number into multiple file

Hi All, I have more than half million lines of XML file , wanted to split in four files in a such a way that top 7 lines should be present in each file on top and bottom line of should be present in each file at bottom. from the 8th line actual record starts and each record contains 15 lines... (14 Replies)
Discussion started by: ajju
14 Replies

5. Shell Programming and Scripting

Splitting a file and creating new files using Perl script

Hi All, I am new to Scripting language. I want to split a file and create several subfiles using Perl script. Example : File format : Sourcename ID Date Nbr SU IMYFDJ 9/17/2012 5552159976555 SU BWZMIG 9/14/2012 1952257857887 AR PEHQDF 11/26/2012 ... (13 Replies)
Discussion started by: Deepak9870
13 Replies

6. Shell Programming and Scripting

Splitting a file in to multiple files and passing each individual file to a command

I have an input file with contents like: MainFile.dat: 12247689|7896|77698080 16768900|hh78|78959390 12247689|7896|77698080 16768900|hh78|78959390 12247689|7896|77698080 16768900|hh78|78959390 12247689|7896|77698080 16768900|hh78|78959390 12247689|7896|77698080 16768900|hh78|78959390 ... (4 Replies)
Discussion started by: rkrish
4 Replies

7. UNIX for Dummies Questions & Answers

Is there any way of splitting the script (Noob Here).

I m writing a script to check Server Hardening. The problem is whenever i add new point it grows and it become very tedious to edit the script file. Is there any way of making them separate and call them from one base script? Is it possible to define global variable that can be accessed via... (5 Replies)
Discussion started by: pinga123
5 Replies

8. Shell Programming and Scripting

script for splitting file

Can anyone help me in giving a script for the below scenario I have file from the source increamenting in size... I require to write a script witch will move the data to the new file once the file reaches 50MB of size . This needs If the first file reaches 50MB then my script has to generate... (3 Replies)
Discussion started by: Sudhakishore.P
3 Replies

9. Shell Programming and Scripting

File splitting and grouping using unix script

Hello All, I have a small problem with file group/splitting and I am trying to get the best way to perform this in unix. I am trying with awk but need some suggestion what would be the best and fastest way to-do it. Here is the problem. I have a fixed length file with filled with product... (4 Replies)
Discussion started by: nandhan11891
4 Replies

10. Shell Programming and Scripting

Help with shell script - splitting

Hi, I need to split the file lines in below format as. Input file : Sample.txt <Rule expression="DeliverToCompID IS NULL" invert="true"> <Rule field="PossDupFlag" value="Y" > <Rule expression="OrdStatus = '2' AND OrigClOrdID IS NULL"> Output... (5 Replies)
Discussion started by: manosubsulo
5 Replies
Login or Register to Ask a Question