Consolidate files based on priority


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Consolidate files based on priority
# 1  
Old 01-29-2014
Consolidate files based on priority

please help, I would like to merge 4 files, all of them same format.
The first col is the key for all files,followed by variable number of fields so I want to check if the key is present in the files according to priority and then also add a column saying it came from which file .

So the priority is file1>file2>file3>file4. If a key is present in file1, it gets top priority and goes to the output, if the key is absent, file2 has to be checked , then 3 , then 4 .
Code:
file1
key1 description of keyf1
key2 descf1
key5 key descriptionf1


file 2
key2 description of keyf2
key3 descf2
key4 key descriptionf2


file 3
key5 description of keyf3
key6 descf3
key7 key descriptionf3

file4 

key6 description of keyf4
key7 descf4
key8 key descriptionf4

output 
key1 description of keyf1 file1
key2 descf1 file1
key3 descf2 file2
key4 key descriptionf2 file2
key5 key descriptionf1 file1
key6 descf3 file3
key7 key descriptionf3 file3
key8 key descriptionf4 file4

# 2  
Old 01-29-2014
Try
Code:
grep "" file[1-4] | tr : ' ' | sort -k2,2 | awk -F" " '!L[$2]++ {$(NF+1)=$1; $1="";  print}'
 key1 description of keyf1 file1
 key2 descf1 file1
 key3 descf2 file2
 key4 key descriptionf2 file2
 key5 key descriptionf1 file1
 key6 descf3 file3
 key7 key descriptionf3 file3
 key8 key descriptionf4 file4

If you need to get rid of the leading space, additional measures have to be taken.
# 3  
Old 01-29-2014
My approach would be to grab all the keys, then just look for the first occurrence of each key across all the files (listed in priority order).
Code:
#!/bin/sh
awk '{ print $1 }' file? | sort | while read key
do
  greptuple=`grep "^$key " file? | head -1` 
  filename=`echo "$greptuple" | cut -d ':' -f 1`
  description=`echo "$greptuple" | sed 's/^[^ ]* //'`
  echo "$key $description $file"
done

Not tested but should either work or get you close I'd think
# 4  
Old 01-29-2014
Try:
Code:
awk '!a[$1]{x=$1;$1="";a[x]=$0" "FILENAME}END{for (i in a) print i,a[i]}' file1 file2 file3 file4

# 5  
Old 01-29-2014
Another awk solution (keeping order same as it appears in input files)

Code:
awk '!($1 in F) {F[$1]; R[++l]=$0 OFS FILENAME} END{for(i=1;i<=l;i++) print R[i]}' file*

Edit: I think bartus11's solution can be simplified to:
Code:
awk '!a[$1]{a[$1]=$0" "FILENAME}END{for (i in a) print a[i]}' file1 file2 file3 file4


Last edited by Chubler_XL; 01-29-2014 at 06:54 PM..
Login or Register to Ask a Question

Previous Thread | Next Thread

7 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Providing virtual machine priority in kvm based virtual machines

Hi All, Is there any way I can prioritize my VMs when there is resource crunch in host machine so that some VMs will be allocated more vcpu, more memory than other VMs in kvm/qemu hypervisor based virtual machines? Lets say in my cloud environment my Ubuntu 16 compute hosts are running some... (0 Replies)
Discussion started by: SanjayK
0 Replies

2. Shell Programming and Scripting

Need help for automated pickup of file based on a priority

Hi experts, I am facing a problem right now.I have to automate the pickup of files based on a priority.The scenario is as below: 1) There will be files from Mon-Fri with Mon file being named as abc_def_01_YYYYMMDD and Tue file being abc_def_02_YYYYMMDD and so forth till Friday's file which... (1 Reply)
Discussion started by: vikramgk9
1 Replies

3. Shell Programming and Scripting

How to copy files from one location to another based on a priority?

Hi Gurus, I am a newbie to shell scripting and I am facing a problem right now.I have to automate the copy of files based on a priority.The scenario is as below: 1) There will be files from Mon-Fri with Mon file being named as abc_def_01_YYYYMMDD and Tue file being abc_def_02_YYYYMMDD and so... (4 Replies)
Discussion started by: vikramgk9
4 Replies

4. Shell Programming and Scripting

Multiple lines consolidate

This post is start for me ... I stumped at something that I not sure as to how start on ... I tried so of your script that i honestly lost mind looking and looking here ... please help COL1 COl2 COL3 12222 AUH FLUEH 12222 SSC OPERA 12222 SSC ... (8 Replies)
Discussion started by: Sebastian.Thoma
8 Replies

5. Shell Programming and Scripting

Script to consolidate files as mails,groupid

Hi, I'm newbie in Bash scripting. Here is scenario I've a file which containslist of 100 mailgroup and in a directory i have 100 files with each group's respective mail addresses. I'd like to make one CSV file with this information as follows. consolidated files with mailid,groupname. ... (7 Replies)
Discussion started by: manish123456
7 Replies

6. Shell Programming and Scripting

How to consolidate values in one column from different rows into one?

Hi Guys, Thank you all for helping me with my different queries and I continue to get better at scripting because of help from all of you! I have a file that would look something like - ID SUB ID VALUE 1 10 5 2 18 7 1 ... (1 Reply)
Discussion started by: sncoupons
1 Replies

7. Shell Programming and Scripting

consolidate file in unix

hi, i am trying to consolodate the files in the unix with the '>>' i have some 50 or 60 files.is there any another way of consolidating the alll 50 or 60 files in to one file. actually the way i m doing creaating the problem while loading the file with teradat tpump and fasload. so if there is... (3 Replies)
Discussion started by: narang.mohit
3 Replies
Login or Register to Ask a Question