Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

divide a single file with different Weboffercodes into different files with each of o

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting divide a single file with different Weboffercodes into different files with each of o
# 1  
Old 03-31-2008
Question divide a single file with different Weboffercodes into different files with each of o
Here is the format of my file; I do not have the delimiter in the file for the data to be separated.

Each line in the file is in the following format. File contains the data of different WebOfferCodes

Item Code | Account Number | Card Number | Source code | WebOfferCode

12digits | 10 Digits | 15 digits | 10 letters | 20 letters
Format of file:-

3412222240171M41006422373273564021006SDFGHJRSGJBBBBBBBBBBBBBBBBBBBB
0009546640171M41006400373273564531345TTTTPAPAPBBBBBBBBBBBBBBBBBBBB
0009546640171M41006433373273564021006YYYYPAPAPHHHHHHHHHHHHHHHHHH
3412222240171M41006455373273564021116PAPAPAPAPBBBBBBBBBBBBBBBBBBBB
0009546640171M41006400373273564531345UUUUPAPAPHHHHHHHHHHHHHHHHHH
0009546640171M41006477373273564021006SDFGHJRSGJBBBBBBBBBBBBBBBBBBBBB

Item Code: 12digits

Account Number: 10 Digits

Card Number: 15 digits

Source_code:10 letters

WebOfferCode: 20 letters

I need the optimal ways to sort this file of million records.So i want to divide the single file with multiple WebOffercode into distinct file containing data of 1 Weboffercode each

Validation to be made:-

Card Member fields should not be null
Web Offer code fields should not be null
# 2  
Old 03-31-2008
A possible solution :
Code:
$ cat web.awk
#!/usr/bin/awk -f

BEGIN {
   ErrorCode = "invalid";
}

{
   #Item Code | Account Number | Card Number | Source code | WebOfferCode
   #12digits | 10 Digits | 15 digits | 10 letters | 20 letters

   ItemCode      = substr($0,  1, 12);
   AccountNumber = substr($0, 13, 10);
   CardNumber    = substr($0, 23, 15);
   SourceCode    = substr($0, 38, 10);
   WebOfferCode  = substr($0, 48, 20);


   cn  = CardNumber  ; gsub(/[0 ]/, "", cn  );
   woc = WebOfferCode; gsub(/[0 ]/, "", woc );
   if (cn == "" || woc == "")
      OutputFile = "woc_" ErrorCode ".dat"
   else
      OutputFile = "woc_" WebOfferCode ".dat"
   print $0 >> OutputFile;

   printf("%-28s : %s|%s|%s|%s|%s\n", OutputFile, ItemCode, AccountNumber, CardNumber, SourceCode, WebOfferCode);
}

$ cat web.dat
3412222240171M41006422373273564021006SDFGHJRSGJBBBBBBBBBBBBBBBBBBBB
0009546640171M41006400373273564531345TTTTPAPAP
0009546640171M41006400373273564531345TTTTPAPAP00000000000000000000
0009546640171M41006400               TTTTPAPAPBBBBBBBBBBBBBBBBBBBB
0009546640171M41006400000000000000000TTTTPAPAPBBBBBBBBBBBBBBBBBBBB
0009546640171M41006400373273564531345TTTTPAPAPBBBBBBBBBBBBBBBBBBBB
0009546640171M41006433373273564021006YYYYPAPAPHHHHHHHHHHHHHHHHHH
3412222240171M41006455373273564021116PAPAPAPAPBBBBBBBBBBBBBBBBBBBB
0009546640171M41006400373273564531345UUUUPAPAPHHHHHHHHHHHHHHHHHH
0009546640171M41006477373273564021006SDFGHJRSGJBBBBBBBBBBBBBBBBBBBBB
$ web.awk web.dat
woc_BBBBBBBBBBBBBBBBBBBB.dat : 341222224017|1M41006422|373273564021006|SDFGHJRSGJ|BBBBBBBBBBBBBBBBBBBB
woc_invalid.dat              : 000954664017|1M41006400|373273564531345|TTTTPAPAP |
woc_invalid.dat              : 000954664017|1M41006400|373273564531345|TTTTPAPAP0|0000000000000000000
woc_invalid.dat              : 000954664017|1M41006400|               |TTTTPAPAPB|BBBBBBBBBBBBBBBBBBB
woc_invalid.dat              : 000954664017|1M41006400|000000000000000|TTTTPAPAPB|BBBBBBBBBBBBBBBBBBB
woc_BBBBBBBBBBBBBBBBBBB.dat  : 000954664017|1M41006400|373273564531345|TTTTPAPAPB|BBBBBBBBBBBBBBBBBBB
woc_HHHHHHHHHHHHHHHHH.dat    : 000954664017|1M41006433|373273564021006|YYYYPAPAPH|HHHHHHHHHHHHHHHHH
woc_BBBBBBBBBBBBBBBBBBB.dat  : 341222224017|1M41006455|373273564021116|PAPAPAPAPB|BBBBBBBBBBBBBBBBBBB
woc_HHHHHHHHHHHHHHHHH.dat    : 000954664017|1M41006400|373273564531345|UUUUPAPAPH|HHHHHHHHHHHHHHHHH
woc_BBBBBBBBBBBBBBBBBBBB.dat : 000954664017|1M41006477|373273564021006|SDFGHJRSGJ|BBBBBBBBBBBBBBBBBBBB
$ ls -1 woc_*.dat
woc_BBBBBBBBBBBBBBBBBBB.dat
woc_BBBBBBBBBBBBBBBBBBBB.dat
woc_HHHHHHHHHHHHHHHHH.dat
woc_invalid.dat
$

With gawk, you can also do :
Code:
$ cat web2.awk
#!/usr/bin/gawk -f

BEGIN {
   FIELDWIDTHS = "12 10 15 10 20";
   ErrorCode = "invalid";
}

{
   #Item Code | Account Number | Card Number | Source code | WebOfferCode
   #12digits | 10 Digits | 15 digits | 10 letters | 20 letters

   ItemCode      = $1;
   AccountNumber = $2;
   CardNumber    = $3;
   SourceCode    = $4;
   WebOfferCode  = $5;


   cn  = CardNumber  ; gsub(/[0 ]/, "", cn  );
   woc = WebOfferCode; gsub(/[0 ]/, "", woc );
   if (cn == "" || woc == "")
      OutputFile = "woc_" ErrorCode ".dat"
   else
      OutputFile = "woc_" WebOfferCode ".dat"
   print $0 >> OutputFile;

   # DebuG
   printf("%-28s : %s|%s|%s|%s|%s\n", OutputFile, ItemCode, AccountNumber, CardNumber, SourceCode, WebOfferCode);
}
$

Jean-Pierre.
# 3  
Old 04-01-2008
Hey!!!
Thanks a lot your reply is of great help to me...
# 4  
Old 04-02-2008
Code:
awk '{
t=substr($0,48)
if(file[t]=="")
        file[t]=$0
else
        file[t]=sprintf("%s\n%s",file[t],$0)
}
END{
for (i in file)
{
        print file[i] > i
        close(i)
}
}' filename

# 5  
Old 04-02-2008
Join the data in two files
hello friendz...

I have multiple around 6 files in the folder I have to load the each file to the different tables in DB2 database.
List of tables is present in a file called table_list.txt file
MR_Detail_1

MR_Detail_4

MR_Detail_5

Note:-

LET.MRTEST.BBBB.txt present in xyz folder should be loaded into MR_Detail_1 table

LET.MRTEST.CCCC.txt present in xyz folder should be loaded into MR_detail_4 table

LET.MRTEST.DDDD.txt present in xyz folder should be loaded into MR_detail_5 table


There are around 6 columns in the database and the data comes from the file.

Format for the file

3412222240171M41006422373273564021006SDFGHJRSGJBBBBBBBBBBBBBBBBBBBB

0009546640171M41006400373273564531345TTTTPAPAPABBBBBBBBBBBBBBBBBBBB

Each file contains this type of data .How will I write my shell script so that my code picks the file automatically from xyz folder and keep loading on the Table_list.txt.There should me mapping between the Files and tables

What do you suggest shall I keep this mapping between file name and table in 1 file and read this file..how can do this

like creating this format file

LET.MRTEST.BBBB.txt MR_DETAIL_1
LET.MRTEST.BBBB.txt MR_DETAIL_4
LET.MRTEST.BBBB.txt MR_DETAIL_5

AND READING these elements and substituting in my loadutility command how can I do this...please help me
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Output file name and file contents of multiple files to a single file

I am trying to consolidate multiple information files (<hostname>.Linux.nfslist) into one file so that I can import it into Excel. I can get the file contents with cat *Linux.nfslist >> nfslist.txt. I need each line prefaced with the hostname. I am unsure how to do this. --- Post updated at... (5 Replies)
Discussion started by: Kentlee65
5 Replies

2. Shell Programming and Scripting

Paste 2 single column files to a single file

Hi, I have 2 csv/txt files with single columns. I am trying to merge them using paste, but its not working.. output3.csv: flowerbomb everlon-jewelry sofft steve-madden dolce-gabbana-watchoutput2.csv: http://www1.abc.com/cms/slp/2/Flowerbomb http://www1.abc.com/cms/slp/2/Everlon-Jewelry... (5 Replies)
Discussion started by: ajayakunuri
5 Replies

3. Shell Programming and Scripting

Divide an EBCDIC files into multiple files based on value at 45-46 bytes

Hi All, I do have an EBCDIC file sent from the z/os , this file has records with different record types in it, the type of record is identified by bytes 45-46 like value 12 has employee record value 14 has salaray record and etc.... we do now want to split the big ebcdic file into multiple... (3 Replies)
Discussion started by: okkadu
3 Replies

4. Shell Programming and Scripting

Divide data with specific column values into separate files

hello! i need a little help from you :) ... i need to split a file into separate files depending on two conditions using scripting. The file has no delimiters. The conditions are col 17 = "P" and col 81 = "*", this will go to one output file; col 17 = "R" and col 81 = " ". Here is an example. ... (3 Replies)
Discussion started by: chanclitas
3 Replies

5. Shell Programming and Scripting

Divide data into separate files

frnds: i want to divide data on the behalf of dotted line and redirectd into new files ) ------------------------- M-GET CONFIRMATION ( ------------------------- M-GET CONFIRMATION ( INVOKE IDENTIFIER final data shuld be into 3 files ...... (6 Replies)
Discussion started by: dodasajan
6 Replies

6. UNIX for Dummies Questions & Answers

divide the file into multiple files based on the city name

Hi, I have a file abc.dat. It contains the fileds of empid, empname, empcity. each city contains 10 records. i want to create the city file and pass the same city records into the file. I don't know the city names. In unix using awk command how can we do? abc.dat: 1 john delhi 2... (2 Replies)
Discussion started by: raghukreddy.ab
2 Replies

7. Shell Programming and Scripting

Match two files and divide fields

I have two files that have the date field in common. I request your help with some script that divide each field value from file1 by the correspond field value of the file2 only when the field date is equal in both files. Thanks in advance ! This is a sample of the files file 1 12/16/2010,... (2 Replies)
Discussion started by: csierra
2 Replies

8. Shell Programming and Scripting

Divide large data files into smaller files

Hello everyone! I have 2 types of files in the following format: 1) *.fa >1234 ...some text... >2345 ...some text... >3456 ...some text... . . . . 2) *.info >1234 (7 Replies)
Discussion started by: ad23
7 Replies

9. Shell Programming and Scripting

Match two files and divide a field !

Hello, I have two files that have the date field as a common. I request your help with some script that divide the value of the file1 by the value in the file2 only when the field date are the same between both files and create a new text file. This is a sample of the files file1... (1 Reply)
Discussion started by: csierra
1 Replies

10. Shell Programming and Scripting

how to divide single large log file into multiple files.

Can you please help me with writing script for following purpose. I have to divide single large web access log file into multiple log files based on dates inside the log file. For example: if data is logged in the access file for jan-10-08 , jan-11-08 , Jan-12-08 then make small log file... (1 Reply)
Discussion started by: kamleshm
1 Replies
Login or Register to Ask a Question