Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Modification of Summation Script

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Modification of Summation Script
# 15  
Old 08-30-2012
Try:

Code:
FILES="$1"
DELIMITER="$2"
COL_LIST=$3
EXC4_FILE=${4:-/dev/null}
EXC2_FILE=${5:-/dev/null}
 
for FILE in $FILES
do
    gzip -t ${FILE} 2>/dev/null
    if [ $? -eq 1 ];
    then
        comm=cat
    else
        comm=gzcat
    fi
    $comm $FILE | awk -v col_list=$COL_LIST -v sourcefile=$FILE -F "$DELIMITER" '
        BEGIN {
            cols=split(col_list, col, ",");
            split("sales_date,cust_id,UPC", titles, ",")
        }
        FILENAME != "-" {
            if(FNR>1) {
               gsub(/,/,FS)
               if(file==1) EXC_4[$2]
               else EXC_2[$1]
            } else file++
            next
        }
        !($4 in EXC_4) && !($2 in EXC_2) {
            if(FNR!=1) {
                if($col[3]!="" && $col[1]!="" && $col[2]!="") {
                   val=$col[3]
                   for(m=4;m<=cols;m++) val=val "|" $col[m]
                   v[val] += $col[1]
                   d[val] += $col[2]
                }
            }
        }
        END{
            for(i=2;i<cols;i++) printf "%s|", titles[i-1];
            printf("sum(POS_QTY)|sum(POS_AMT)|<source_file>\n")
            for (i in v)
                printf("%s|%d|%10.4f|%s\n",i,v[i],d[i],sourcefile)
        }' $EXC4_FILE $EXC2_FILE -
done

For multi-input files you are probably best merging files into 1 first eg:
Code:
( cat file1.csv ; sed 1d file2.csv ; sed 1d file3.csv ) > merged.csv

# 16  
Old 08-31-2012
hi Chubler_XL, thanks again for your reply.
you have no idea, how much i appreciate it.

i apologize if i still have concerns, even after such a long conversation.

it's just that for this 2nd and 3rd file, it is possible that there is no 2nd file and only the third file... i forgot to mention that in the "usage" in my previous post.
it was very irresponsible of me. im sorry.

so these are the second and third file:
--second file--
Code:
$ cat flagfile_2.csv
ID,UPC,name
1,PL_2003007476012,pl1
2,PL_2003000322606,pl2
3,PL_2003005081201,pl3
4,PL_2003010201151,pl4

--third file--
Code:
$ cat flagfile.csv
UPC,name
PL_000000000034011,pl1
PL_000000000034012,pl2
PL_000000000034013,pl3
PL_000000000034010,pl4

anyway, so i tried executing, the modified code, with both second and third file present, and as expected of your codes, it resulted perfectly.

Code:
$ sh qa_del_9.sh ft-GNCT-3398-CD-2012-07-07-140113.txt "        " 22,23,1,2,4 flagfile_2.csv flagfile.csv
sales_date|cust_id|UPC|sum(POS_QTY)|sum(POS_AMT)|<source_file>
2012-07-05|PL_000000000034014|PL_2003006845246|11|   11.0800|ft-GNCT-3398-CD-2012-07-07-140113.txt
2012-07-05|PL_000000000034014|PL_2003008732988|27|   27.6300|ft-GNCT-3398-CD-2012-07-07-140113.txt
2012-07-05|PL_000000000034014|PL_2003008035225|63|   63.2700|ft-GNCT-3398-CD-2012-07-07-140113.txt

although, i completely forgot to mention the possibility of the third file.

but then, i tried to execute it, with only the third file.

it seems that it treated it as the second file, since it was placed as $4... right?

could you suggest on what is the usage, if there is only a third file?
so far, it passes perfectly in these scenarios:

-second file only
-second and third file

is it possible to add/modify the code to be able to accept only the third file, but not treat it as second file?

Thanks again, in advance, you never fail to amaze me with your skills and thanks for sharing it with me, and to all those people you've helped here. thanks and i mean it.

PS i'm still trying to test for the merging that you said on your last note... i'll get back maybe post later, if i understood it correctly. thanks man.
# 17  
Old 08-31-2012
Just use /dev/null as the 2nd file.


Edit: No that dosn't work, without any changes to the code you can only A) use a dummy file with 1 record in it as the 2nd file or B) pass 3nd file in twice.

---------- Post updated at 02:24 PM ---------- Previous update was at 02:07 PM ----------

Here's a really kludgy solution (pass "none" in as your 2nd file):

Code:
EXC4_FILE=${4:-/dev/null}
EXC2_FILE=${5:-/dev/null}
 
if [ $EXC4_FILE = "none" ]
then
   trap 'rm /tmp/$$.empty' 0 1 3 15
   echo "dummy" > /tmp/$$.empty
   EXC4_FILE="/tmp/$$.empty"
fi

Last edited by Chubler_XL; 08-31-2012 at 01:18 AM..
# 18  
Old 08-31-2012
wow, that's a really clever thing you did there. really brilliant idea.

Though, does it really have to be "none" with quotation marks(" ")?

it seems that when i put only none, it doesn't work.
Code:
$ sh qa_del_11.sh ft-GNCT-3398-CD-2012-07-07-140113.txt "       " 22,23,1,2,4 "none" flagfile.csv
sales_date|cust_id|UPC|sum(POS_QTY)|sum(POS_AMT)|<source_file>
2012-07-05|PL_000000000034014|PL_2003006845246|11|   11.0800|ft-GNCT-3398-CD-2012-07-07-140113.txt
2012-07-05|PL_000000000034014|PL_2003007476012|75|   75.9200|ft-GNCT-3398-CD-2012-07-07-140113.txt
2012-07-05|PL_000000000034014|PL_2003008732988|27|   27.6300|ft-GNCT-3398-CD-2012-07-07-140113.txt
2012-07-05|PL_000000000034014|PL_2003008035225|63|   63.2700|ft-GNCT-3398-CD-2012-07-07-140113.txt

Though i think, it's pretty good now, but i still have to check for multiple files.

so, i need to merge the files manually first? and use the final merge file as first input file?

Thanks a lot man, though i would really like to know what exactly this code does? there seems to be unusual numbers (0 1 3 15) and directories there..?
is it going to create file or something?

Code:
trap 'rm /tmp/$$.empty' 0 1 3 15
   echo "dummy" > /tmp/$$.empty
   EXC4_FILE="/tmp/$$.empty"

Thanks again buddy, you are really a vital part of this site, thanks for all your help and patience this past few days. i really learned a lot from you, i hope i can become like you someday. thanks man
# 19  
Old 09-02-2012
Quote:
Originally Posted by ramneim
Thanks a lot man, though i would really like to know what exactly this code does? there seems to be unusual numbers (0 1 3 15) and directories there..?
is it going to create file or something?
Yes a dummy files is being created with just the header line of "dummy" the trap command is designed to ensure these dummy files are cleaned up, even when you ctrl-c or kill the script.

The numbers are the signal codes to trap, and thinking about the values used you are probably best with (1 2 3 15) instead of the list I originally had 1 is HUP (is terminal logs off, eg lost connection); 2 is Interupt (Ctrl-C), 3 is QUIT (normal termination) and 15 is TERM (killed).

Not sure why you need quotes around none, it works fine without them on my system.
This User Gave Thanks to Chubler_XL For This Post:
ramneim
# 20  
Old 09-02-2012
Quote:
Originally Posted by Chubler_XL
Yes a dummy files is being created with just the header line of "dummy" the trap command is designed to ensure these dummy files are cleaned up, even when you ctrl-c or kill the script.

The numbers are the signal codes to trap, and thinking about the values used you are probably best with (1 2 3 15) instead of the list I originally had 1 is HUP (is terminal logs off, eg lost connection); 2 is Interupt (Ctrl-C), 3 is QUIT (normal termination) and 15 is TERM (killed).

Not sure why you need quotes around none, it works fine without them on my system.
You may want to trap on conditions 0 1 2 3 and 15. 3 does mean that the process was terminated by a SIGQUIT signal, but that is not normal termination. 0 is normal termination (i.e., terminated by the process exiting).

Given the complexity of the command line processing here, I would suggest using getopts, with a -x excl_file option to specify the exclude file, -a add_file option for the newest operand that was added, and putting the file operands at the end of the command line as separate operands instead of as a quoted, space separated list in the 1st operand (thus making it much simpler to specify a group of file operands as something likefile*.csv).
This User Gave Thanks to Don Cragun For This Post:
ramneim
# 21  
Old 09-03-2012
Thanks you guys, i think the first one you posted works just fine, i guess.

Thanks a lot for all your help! Smilie
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Need small modification in script

Hi All, In the below script, I am calling one sql file test.sql If this file returns any data then I have to generate this file test_$RUN_DATE.FCNA If the sql files returns no data then I dont want to generate this file test_$RUN_DATE.FCNA. I tried one approach like: check the size of FCNA files... (1 Reply)
Discussion started by: praveenk768
1 Replies

2. Shell Programming and Scripting

awk script modification

can someone help me identify what i'm doing wrong here: awk -F'|' 'BEGIN{c=0} /./ && /./ { if ($3 < 2) { print ; c++ } END { print c":OK" } else if (($3 >= 2) && ($3 < 4)) { print ; c++ } END { print c":WARNING" } else if ($3 >= 4) { print ; c++ } END { print c":CRITICAL" } }'... (4 Replies)
Discussion started by: SkySmart
4 Replies

3. Shell Programming and Scripting

Modification in script

Hi, I have below script, i want to monitor that that ntp server listed in setting is under sync or not. I wrote below script but it is not working properly. Here are problems, first it should server under sync if "*" shows and rest if shows "+" it means it is next server in waiting list.... (4 Replies)
Discussion started by: learnbash
4 Replies

4. Shell Programming and Scripting

ksh script modification

Hi I have some list of files in a .dat i need to read them line by line and assing them to variables. For ex: list of files are some,some1 i need two variables g1 as some and g2 as some1. and then need to perform some operations on g1 and g2 for which i can get some o/p, i need to capture... (2 Replies)
Discussion started by: Ravindra Swan
2 Replies

5. Shell Programming and Scripting

awk script modification

I want the below script to omit every chunk of data that contains a specific hostname. here's the scenario. i have a configuration file that contains the configuration of several hosts. a sample of this configuration file is this: define host { address ... (12 Replies)
Discussion started by: SkySmart
12 Replies

6. Shell Programming and Scripting

Modification in shell script

Hello Team, I have prepared script which will check for listening message for ports 1199,1200 and 1201. I need modifcation in script in such a way that if port 1200 is not listening then it should message rmi port 1200 is not listening. Smap for port 1199 and 1201. kindly guide me to acheive... (4 Replies)
Discussion started by: coolguyamy
4 Replies

7. Shell Programming and Scripting

Help with Shell Script Modification

Hi all Iam very new to Shell Scripting, I have to modify a shell script looking at an existing one except that it will query against some table X in A database. Befor Spooling check if there are any reload files if there archive the files. The above scipt executes some abc.sql which will b a new... (2 Replies)
Discussion started by: Varunkv
2 Replies

8. Shell Programming and Scripting

time modification in script

Hi All.. I have a file with a number of non-unique entries as below: 1243 01:42:29,567 --> 01:42:32,108 blah blah .... blah blah .. 1244 01:42:32,709 --> 01:42:34,921 blah blah .... 1245 01:42:35,214 --> 01:42:36,533 blah blah .... blah blah .. blah blah .... blah blah .. (4 Replies)
Discussion started by: UniRock
4 Replies

9. Shell Programming and Scripting

Need a modification on this script

Hi All I have files contains rows which look like this: 2 20090721_16:58:47.173 JSUD2 JD1M1 20 IAM 966591835270 249918113182 b 3610 ACM b 3614 ACM b 3713 CPG b 3717 CPG f 5799 REL b 5815 RLC b 5817 RLC :COMMA: NCI=00,FCI=6101,CPC=0A,TMR=00,OFI=00,USI: :COMMB: BCI=1234: :RELCAUSE:10: ... (1 Reply)
Discussion started by: zanetti321
1 Replies

10. Shell Programming and Scripting

help in script modification

i have the following perl script.but it searches for a given filename. i want to run the same script in my directoy which has subdirectories too and it has to display the file if sreach satisfies along with directory name. can anyone help me: perl script: my $FILE = $ARGV; for zf in... (4 Replies)
Discussion started by: a.suryakumar
4 Replies
Login or Register to Ask a Question