Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Need to combine counts from diff files

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Need to combine counts from diff files
# 1  
Old 04-07-2011
Need to combine counts from diff files
Ok i have a script that goes into the last 3 files, grabs a value from that file which is a date, counts the number of results of that date.

So this is the output:
FILED001.20110407.175709.086912
49995 04-07T12
4 04-07T07
1 04-07T17

FILED001.20110407.180111.086913
50000 04-07T12

FILED006.20110407.175229.081672
48939 04-07T12
626 04-06T21
427 04-06T20
6 04-07T11
2 04-07T07


FILED006.20110407.175316.081673
32721 04-06T21
8895 04-06T20
8383 04-07T12
1 04-06T16

But I want the output to combine the two files that have the same extension in the front so:
FILED001
FILED006

And add the numbers for each date.

so instead it would look like this:
FILED006
33347 04-06T21
9322 04-06T20
57322 04-07T12
6 04-07T11
2 04-07T07
1 04-06T16
# 2  
Old 04-07-2011
See if this works for you:
Code:
#!/usr/bin/ksh
typeset -i mTot=0
rm -f F_form
while read mFld1 mFld2
do
  if [[ "${mFld2}" = "" ]]; then
    mPref=$(echo ${mFld1} | cut -d'.' -f1)
  else
    echo ${mPref} ${mFld2} ${mFld1} >> F_form
  fi
done < F_inp_file
mPPref="First"
mPTag="First"
sort F_form |
while read mPref mTag mCount
do
  if [[ "${mPref}" != "${mPPref}" || "${mTag}" != "${mPTag}" ]]; then
    if [[ "${mPPref}" != "First" ]]; then
      echo ${mPPref} ${mPTag} ${mTot}
    fi
    mTot=0
  fi
  mTot=${mTot}+${mCount}
  mPPref=${mPref}
  mPTag=${mTag}
done
if [[ "${mPPref}" != "First" ]]; then
  echo ${mPPref} ${mPTag} ${mTot}
fi
exit

# 3  
Old 04-07-2011
Something like this? (output unsorted)
Code:
[MIRO@stroke Desktop]$ cat tst
FILED001.20110407.175709.086912
49995 04-07T12
4 04-07T07
1 04-07T17

FILED001.20110407.180111.086913
50000 04-07T12

FILED006.20110407.175229.081672
48939 04-07T12
626 04-06T21
427 04-06T20
6 04-07T11
2 04-07T07


FILED006.20110407.175316.081673
32721 04-06T21
8895 04-06T20
8383 04-07T12
1 04-06T16
[MIRO@stroke Desktop]$ awk '/^FILE/{hdr=gensub(/\..*$/,"",$1); next}{a[hdr" "$2]+=$1}
END{n = asorti(a,b);for(i=1;i<=n;i++){
  if(a[b[i]]==0) {print b[i]} 
  else {print a[b[i]]" "gensub(/FILE.* /,"","",b[i]) }}
}' tst
FILED001 
4 04-07T07
99995 04-07T12
1 04-07T17
FILED006 
1 04-06T16
9322 04-06T20
33347 04-06T21
2 04-07T07
6 04-07T11
57322 04-07T12

Last edited by mirni; 04-07-2011 at 06:17 PM..
# 4  
Old 04-07-2011
Ok it does not exactly output the way you want but maybe it could be a start :
Code:
nawk '/^FILE/{match($0,"^[^\.]*");x=substr($0,RSTART,RLENGTH)}NF=2{B[x":"$2]+=$1}END{for(i in B) print B[i],i}' output | nawk '$1{sub(":"," ",$0);print $2,$3,$1}' | sort -k 1 -k 2n

---------- Post updated at 10:29 PM ---------- Previous update was at 10:26 PM ----------

Code:
# cat tst
FILED001.20110407.175709.086912
49995 04-07T12
4 04-07T07
1 04-07T17

FILED001.20110407.180111.086913
50000 04-07T12

FILED006.20110407.175229.081672
48939 04-07T12
626 04-06T21
427 04-06T20
6 04-07T11
2 04-07T07


FILED006.20110407.175316.081673
32721 04-06T21
8895 04-06T20
8383 04-07T12
1 04-06T16

Code:
# nawk '/^FILE/{match($0,"^[^\.]*");x=substr($0,RSTART,RLENGTH)}NF=2{B[x":"$2]+=$1}END{for(i in B) print B[i],i}' tst | nawk '$1{sub(":"," ",$0);print $2,$3,$1}' | sort -k 1 -k 2n
FILED001 04-07T07 4
FILED001 04-07T12 99995
FILED001 04-07T17 1
FILED006 04-06T16 1
FILED006 04-06T20 9322
FILED006 04-06T21 33347
FILED006 04-07T07 2
FILED006 04-07T11 6
FILED006 04-07T12 57322

# 5  
Old 04-07-2011
Perl

Code:
#!/usr/bin/perl

while(<DATA>) {
chomp;
next if (/^$/);
if(/^\D+/) {@file=split(/\./);next;}
@fld=split(/\s+/);
$a{$file[0]." ".$fld[1]}+=$fld[0];
}
foreach (sort(keys(%a))) {
$cur_fname=(split(/\s+/,$_))[0];
if ( $cur_fname ne $fname ) {
print "\n",$cur_fname,"\n";
}
print $a{$_}," ",(split(/\s+/,$_))[1],"\n";
$fname=(split(/\s+/,$_))[0];
}

__DATA__
FILED001.20110407.175709.086912
49995 04-07T12
4 04-07T07
1 04-07T17

FILED001.20110407.180111.086913
50000 04-07T12

FILED006.20110407.175229.081672
48939 04-07T12
626 04-06T21
427 04-06T20
6 04-07T11
2 04-07T07


FILED006.20110407.175316.081673
32721 04-06T21
8895 04-06T20
8383 04-07T12
1 04-06T16

# 6  
Old 04-07-2011
Code:
awk '{sub(/\..*/,"")}/^FILED/{file=$0;while((getline y)>0 && y~/^[0-9]/) {split(y,a,FS);b[file FS a[2]]+=a[1]}}END{for(i in b) {split(i,c,FS);print c[1],b[i],c[2]|"sort -k1,1 -k2,2rn"}}'
FILED001 99995 04-07T12
FILED001 4 04-07T07
FILED001 1 04-07T17
FILED006 57322 04-07T12
FILED006 33347 04-06T21
FILED006 9322 04-06T20
FILED006 6 04-07T11
FILED006 2 04-07T07
FILED006 1 04-06T16

Last edited by yinyuemi; 04-07-2011 at 07:15 PM..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Automate splitting of files , scp files as each split completes and combine files on target server

i use the split command to split a one terabyte backup file into 10 chunks of 100 GB each. The files are split one after the other. While the files is being split, I will like to scp the files one after the other as soon as the previous one completes, from server A to Server B. Then on server B ,... (2 Replies)
Discussion started by: malaika
2 Replies

2. Shell Programming and Scripting

Diff 3 files, but diff only their 2nd column

Guys i have 3 files, but i want to compare and diff only the 2nd column path=`/home/whois/doms` for i in `cat domain.tx` do whois $i| sed -n '/Registry Registrant ID:/,/Registrant Email:/p' > $path/$i.registrant whois $i| sed -n '/Registry Admin ID:/,/Admin Email:/p' > $path/$i.admin... (10 Replies)
Discussion started by: kenshinhimura
10 Replies

3. Shell Programming and Scripting

combine multiple files by column into one files already sorted!

I have multiple files; each file contains a certain data in a column view simply i want to combine all those files into one file in columns example file1: a b c d file 2: 1 2 3 4 file 3: G (4 Replies)
Discussion started by: ahmedamro
4 Replies

4. Shell Programming and Scripting

Sending e-mail of record counts in 3 or more files

I am trying to load data into 3 tables simultaneously (which is working fine). Then when loaded, it should count the total number of records in all the 3 input files and send an e-mail to the user. The script is working fine, as far as loading all the 3 input files into the database tables, but... (3 Replies)
Discussion started by: msrahman
3 Replies

5. Shell Programming and Scripting

Get counts for multiple files

How do get the counts by excluding header and tailer. wc -l customer_data*.0826 31 customer_data_1.0826 57 customer_data_2.0826 456 customer_data_3.0826 668 customer_data_4.0826 789 customer_data_5.0826 2344 customer_data_6.0826 13457 customer_data_7.0826... (6 Replies)
Discussion started by: zooby
6 Replies

6. Shell Programming and Scripting

how to compare counts in two separate files

Hi all, what will be the code to compare count present in two seperate files for e.g file (a) contains counts 100 and file (b) contains records 90 since both these files have differnt count so it will display count didnt match and in case of success it display (5 Replies)
Discussion started by: jojo123
5 Replies

7. Shell Programming and Scripting

Find duplicates from multuple files with 2 diff types of files

I need to compare 2 diff type of files and find out the duplicate after comparing each types of files: Type 1 file name is like: file1.abc (the extension abc could any 3 characters but I can narrow it down or hardcode for 10/15 combinations). The other file is file1.bcd01abc (the extension... (2 Replies)
Discussion started by: ricky007
2 Replies

8. Shell Programming and Scripting

Comparing Counts Within Separate Files

Hey all, So I have this challenge where I am attempting to compare record counts from within several different log files. I want input and output counts for each file, and I want to compare that with the result of the input/output comparison from a separate--but related file. Example: ... (2 Replies)
Discussion started by: gator76
2 Replies

9. Shell Programming and Scripting

diff 2 files; output diff's to 3rd file

Hello, I want to compare two files. All records in file 2 that are not in file 1 should be output to file 3. For example: file 1 123 1234 123456 file 2 123 2345 23456 file 3 should have 2345 23456 I have looked at diff, bdiff, cmp, comm, diff3 without any luck! (2 Replies)
Discussion started by: blt123
2 Replies

10. UNIX for Dummies Questions & Answers

how to combine two files

i need to combine two file. These two files have the same line number, and i need to combine each corresponding line. I tried the paste, but i need coma as the delimeter. are there anyway to do it? thanks. (4 Replies)
Discussion started by: tao
4 Replies
Login or Register to Ask a Question