Merging data in a file


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Merging data in a file
# 1  
Old 10-27-2011
Merging data in a file

Hello,

Firstly I just wanted to say that I'm not a programmer at all and appreciate any help you can give. I am trying to create a shellscript that reformats the file and adding up colums 5 and 6 for those sections that are continuation of the previous line(s) (signified by beginning with '*') and displaying the final output below.

This is my input file -
Code:
$ cat test.csv
*,None Pack,1003036,Curr Emp Ref Request Quote No 2,2,2
 ,None Pack,1003036,Curr Emp Ref Request Quote No 2,8,2
 ,None Pack,1003036,Curr Emp Ref Request Quote No 2,99,6
*,None Pack,1004345,KFI Chase Quote No 1,2,2
*,None Pack,1006148,KFI Chase Quote No 1,1,1
*,None Pack,1004979,Curr Emp Ref Request Quote No 1,2,2
*,None Pack,1003031,Curr Emp Ref Request Quote No 3,2,2
*,None Pack,1003030,Curr Emp Ref Request Quote No 2,4,4
*,None Pack,1003037,Curr Acc Ref Request Quote No 2,3,3
*,None Pack,1004820,Curr Acc Ref Request Quote No 1,3,3
*,None Pack,1003037,Curr Emp Ref Request Quote No 2,7,7
*,None Pack,1003030,Curr Emp Ref Request Quote No 2,4,4
$

The lines that do not contain an asterisk need to have colums 5 and 6 added to the previous line(s) etc.

This is how I want the output to be:
Code:
$ cat test.csv
None Pack,1003036,Curr Emp Ref Request Quote No 2,109,10
None Pack,1004345,KFI Chase Quote No 1,2,2
None Pack,1006148,KFI Chase Quote No 1,1,1
None Pack,1004979,Curr Emp Ref Request Quote No 1,2,2
None Pack,1003031,Curr Emp Ref Request Quote No 3,2,2
None Pack,1003030,Curr Emp Ref Request Quote No 2,4,4
None Pack,1003037,Curr Acc Ref Request Quote No 2,3,3
None Pack,1004820,Curr Acc Ref Request Quote No 1,3,3
None Pack,1003037,Curr Emp Ref Request Quote No 2,7,7
None Pack,1003030,Curr Emp Ref Request Quote No 2,4,4
$

I have created the following script but are struggling trying to merge the data.

Can you help?

Code:
 
#!/bin/ksh
cat test.csv | while read LINE; do
col1=$(echo $LINE | awk -F, '{print $1}')
stream_id=$(echo $LINE|awk -F, '{print $2}')
doc_type=$(echo $LINE|awk -F, '{print $4}')
no_of_sheets=$(echo $LINE|awk -F, '{print $5}')
no_of_images=$(echo $LINE|awk -F, '{print $6}')
if [ "$col1" = "*" ] ; then
    total_no_sheets=$no_of_sheets
    total_no_images=$no_of_images
    echo ${stream_id},${doc_type},${total_no_sheets},${total_no_images} >> file1
else
    total_no_sheets=`expr ${total_no_sheets} + ${no_of_sheets}`
    total_no_images=`expr ${total_no_images} + ${no_of_images}`
    echo ${stream_id},${doc_type},${total_no_sheets},${total_no_images} >> file2
fi
    echo "${stream_id},${doc_type},${total_no_sheets},${total_no_images}" >> file3
done

Thanks,
Neil.

Moderator's Comments:
Mod Comment Put data/logs etc. in code tags too please, thanks^^

Last edited by zaxxon; 10-27-2011 at 09:56 AM.. Reason: code tags
# 2  
Old 10-27-2011
You could do something like this:
Code:
# cat xx.awk
function dumpline ()
{
   for (i=2;i<=linesize;i++) {
      if (i > 2) {
         printf ("%s",FS);
      }
      printf ("%s",currline[i]);
   }
   printf ("\n");
}
$1 ~ /*/ {
   if (NR != 1) {
      dumpline();
   }
   linesize=split ($0, currline);
}
$1 !~ /*/ {
   currline[5]=currline[5]+$5
   currline[6]=currline[6]+$6
}
END {
   dumpline();
}
# awk -F, -f xx.awk test.csv
None Pack,1003036,Curr Emp Ref Request Quote No 2,109,10
None Pack,1004345,KFI Chase Quote No 1,2,2
None Pack,1006148,KFI Chase Quote No 1,1,1
None Pack,1004979,Curr Emp Ref Request Quote No 1,2,2
None Pack,1003031,Curr Emp Ref Request Quote No 3,2,2
None Pack,1003030,Curr Emp Ref Request Quote No 2,4,4
None Pack,1003037,Curr Acc Ref Request Quote No 2,3,3
None Pack,1004820,Curr Acc Ref Request Quote No 1,3,3
None Pack,1003037,Curr Emp Ref Request Quote No 2,7,7
None Pack,1003030,Curr Emp Ref Request Quote No 2,4,4

# 3  
Old 10-27-2011
Hello,

Thanks for your quick response. I am not very good with awk scripts and have tried what you suggested but get the following error:
Code:
$ awk -F, -f test.awk test.csv
awk: 0602-521 There is a regular expression error.
        *?+ not preceded by valid expression
 The source line number is 12.
 The error context is
                $1 ~ /*/ >>>  { <<<
$

Am I missing something? I am using AIX 5.2 if that helps.

Many Thanks,
Neil.


Moderator's Comments:
Mod Comment Video tutorial on how to use code tags in The UNIX and Linux Forums.

Last edited by Franklin52; 10-27-2011 at 10:34 AM.. Reason: Please use code tags, thank you
# 4  
Old 10-27-2011
Escaping the asterisks (/\*/) should work. Or try with gawk instead of awk.
# 5  
Old 10-27-2011
Hello Carlo,

That's worked. Thanks for all your help. Smilie
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Merging data horizontally with newlines in files

Hi Everyone, I have two files file1 and file2 with these contents cat file1 AAAAA 01/03/2014 04:01:23 BBBB 01/03/2014 03:03:34 CCCcc 01/03/2014 03:03:34 cat file2 1 RED 1 HHHH 1 TTTT 1 BBBBB I tried the below... (2 Replies)
Discussion started by: Aditya_001
2 Replies

2. Shell Programming and Scripting

Help with merging data into single line.

Hi, My input is <message> looking for a big <message>Does fit my G74 laptop. Makes the 10 pound. <message> <message>This bag is the only one I could find to fit my awesome ASUS G74S. <message> <message> Great bag my only wish is that they had put a pocket in which to store and... (6 Replies)
Discussion started by: pamu
6 Replies

3. Shell Programming and Scripting

Merging data from one file into another

Hello, I have a master database of a dictionary with the following structure: a=b (b is a Unicode string) a is the English part and b is the equivalent in a foreign language I have also another file which has a database where the /b/ part of the string has been corrected by an expert. let us... (5 Replies)
Discussion started by: gimley
5 Replies

4. Shell Programming and Scripting

Merging data from 2 files of different lengths?

Hi all, Sorry if someone has answered something like this already, but I have a problem. I am not brilliant with "awk" but think it should be the command to use to get what I am after. I have 2 files: job-file (several hundred lines like): 1018003,LONG MU WAN,1113S 1018004,LONG MU... (4 Replies)
Discussion started by: sgb2301
4 Replies

5. Shell Programming and Scripting

formatting and merging 2 data files

Hi, I have 2 files that I got as an output from another program. They are : File 1 ((((((CtBJa:197.0,CtBTz:197.0):85.0,CtAHr:197.0):116.0,CtDUw:197.0):176.0,CtSwe:197.0):110.0, (CtL2b:197.0,Ct4Bu:197.0):196.0):197.0,CmuNg:197.0);... (5 Replies)
Discussion started by: Lucky Ali
5 Replies

6. Shell Programming and Scripting

Removing Carriage return and merging data

Hi, I am trying to remove the carriage return on the record which starts with ADD, MODIFY, or DELETE keyword as the first value in the record. If the records does not start with anyone of these keywords then combine the records with the previous record (line). Input File name xyz.txt... (6 Replies)
Discussion started by: naveed
6 Replies

7. Shell Programming and Scripting

Merging last and syslog data on time

This is on a HP-UX system. I need to merge the 2 reports, for each line in syslog I need to lookup who was logged in to the pts/# based on the time from the last.txt report. Here is what I get from sulog.log cat syslog | grep "su:" | grep "Jun 14" Jul 14 08:02:48 server1 su: - 2 ... (8 Replies)
Discussion started by: Ikon
8 Replies

8. Shell Programming and Scripting

merging CSV data using a one liner from shell?

I'm trying to merge multiple CSV (comma separated value) files into one large master file. All files have a field that is unique to act as the key for entry/merging into the master file & and all files have the same number of fields that are in the master file. I'll give an example here: ... (2 Replies)
Discussion started by: jjinca
2 Replies

9. Shell Programming and Scripting

Need help for 2 data file merging

Hello Please help me to write Shell script. I want to merge 2 data files . The data files have common columns The data file A have 3 columns Host Version Numberof Failuers The data file B have also 3 coulmns Host Version NumberofFailuers . I want to merge A and B file... (2 Replies)
Discussion started by: getdpg
2 Replies

10. Shell Programming and Scripting

Merging data

Hi, I have the following problem: Input: "num1","num2","num3",num4,num5,"num6" required output: "num1num2","num3",num4,num5,"num6" I need to join field 1 and field 2 together but I always end up getting: "num1""num2","num3",num4,num5,"num6" Note that not all fields have " at both... (8 Replies)
Discussion started by: ReV
8 Replies
Login or Register to Ask a Question