Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

awk to match and apply condtions to matchijng files in directories

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting awk to match and apply condtions to matchijng files in directories
# 15  
Old 10-16-2016
Quote:
Originally Posted by Scrutinizer
Hi Don, that does not seem to be an accurate statement.

The $ is still required for variable expansions within double bracket expressions (as well as within single brackets (test commands); a difference would be the double quote protection that would be needed in the case of single brackets)

A situation where $-signs are not required for basic variable expansions are within arithmetic expressions, but that is not the case here.

So IMO greet_sed was right after all.
Yes. You are absolutely correct. Thank you for catching this.

I have updated post #11, striking out the incorrect statements.
This User Gave Thanks to Don Cragun For This Post:
cmccabe
# 16  
Old 10-16-2016
Here is the command that produces the attached two output files:

Code:
#!/bin/bash

for file in /home/cmccabe/Desktop/comparison/missing/*.txt
do
    file1=${file##*/}    # Strip off directory.
    file1="/home/cmccabe/Desktop/comparison/test_tvc/${file1%.txt}_tvc.bed"
    if [[ -f "$file1" ]]
    then
         awk 'FNR==NR{A[$2]=$0;Q=FILENAME;next} ($2 in A){if($10>30 && $11>49){print A[$2] >> "/home/cmccabe/Desktop/comparison/Match_in_both_files_and_meet_criteria";print "Match found in both the files named " Q " and " FILENAME " is: " A[$2];delete A[$2]}} END{print "NON-matched lines between file named "Q " and " FILENAME " are: ";for(i in A){print A[i] >> "/home/cmccabe/Desktop/comparison/out_no_match_found_values";print A[i]}}'  $file $file1 > /home/cmccabe/Desktop/comparison/final/Output_final_file.txt
    fi
done

Can the filename that was being compared be included in each output line? Alternatively, each pair of files being compared have one file with matched and a non-matched included?

For example,
F113.txt is being compared to F113_tvc.bed and the output of that comparison is saved asprefix_final in /home/cmccabe/Desktop/comparison/final with a Match in both files and meet criteria: and Missing in comparison: section (this would be for non matches and lines where the criteria wasn't met). So using the below data from post 1 (each line is a newline):

file1 (F113.txt)
Code:
 Missing in IDP but found in Reference: 
2   166848646G   A   exonic  SCN1A   68  13  16;20   0;0 17;15   0;0 0;0 0;0     c.[5139C>T]+[=] 52.94 2   166245888   G   A   exonic  SCN1A   68  13  16;20   0;0 17;15   0;0 0;0 0;0     c.[5500G>T]+[=] 32

file2 (F113.bed)
Code:
 Chrom    Position    Gene Sym    Target ID    Type    Zygosity    Genotype    Ref    Variant    Var Freq    Qual    Coverage    Ref Cov    Var Cov 
chr2    166245425   SCN2A   AMPL5155065355  SNP Het C/T C   T   54  100   50    23  27 chr2    166848646   SCN1A   AMPL1543060606  SNP Het        G/A   G  A   52.9411764706   100 68  32  36

desired output (F113_final.txt)
Code:
 Match in both files and meet criteria: 
2   166848646   G   A   exonic  SCN1A   68  13  16;20   0;0 17;15   0;0 0;0 0;0     c.[5139C>T]+[=] 52.94 
Missing in comparison: 2   166245888   G   A   exonic  SCN1A   68  13  16;20   0;0 17;15   0;0 0;0 0;0     c.[5500G>T]+[=] 32

Thank all for your help Smilie
Match_in_both_files_and_meet_criteria.txt (569 Bytes) 
out_no_match_found_values.txt (1.7 KB) 
Last edited by cmccabe; 10-16-2016 at 09:33 AM.. Reason: fixed format, added details
# 17  
Old 10-16-2016
Hey ,


Thanks Ravindra for the feedback.

I have couple of points which I just thought of mentioning :

1. Regarding the command "
Code:
echo $j | grep "^i "

--> I did this , because as $j will take value lets say F113_abc_xyz.beds and $i will take the value F113, so I am checking if $j has value with line starting with F113 and if it is successful, it will return exit status 0.

I have provided the test I had done for this:

Code:
bash-4.3$ export i=F113_abc_efg.beds                                                                                                                          
bash-4.3$ echo $i                                                                                                                                             
F113_abc_efg.beds           
bash-4.3$ z=F113                                                                                                                                              
bash-4.3$ echo "$i" | grep "^$z"                                                                                                                              
F113_abc_efg.beds                                                                                                                                             
bash-4.3$ echo $?                                                                                                                                             
0           
bash-4.3$

2. Regarding the 2nd point,thanks for the idea.I thought that $10 and $11 are being positional parameters.

3.Regarding 3rd point, really appreciate , while read is better option than for loops.

Thanks,
Sanghamitra
Last edited by Scrutinizer; 10-16-2016 at 12:40 PM.. Reason: code tags
This User Gave Thanks to Sanghamitra C. For This Post:
cmccabe
# 18  
Old 10-16-2016
Hi cmccabe,

Following minor change helps, hope i got you correctly Smilie

Code:
file1=${file##*/}
getprefix=${file1%%.txt}
<rest of the code is same>
awk <same code> $file $file1 > ${getprefix}_final_file.txt

I have verified with given input samples using RavinderSingh13 solution and it gets your desired output as per post#16.
This User Gave Thanks to greet_sed For This Post:
cmccabe
# 19  
Old 10-18-2016
Thank you all Smilie
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

awk move select fields to match file prefix in two directories

In the awk below I am trying to use the file1 as a match to file2. In file2 the contents of $5,&6,and $7 (always tab-delimited) and are copied to the output under the header Quality metrics. The below executes but the output is empty. I have added comments to help and show my thinking. Thank you... (0 Replies)
Discussion started by: cmccabe
0 Replies

2. Shell Programming and Scripting

awk to match field between two files and use conditions on match

I am trying to look for $2 of file1 (skipping the header) in $2 of file2 (skipping the header) and if they match and the value in $10 is > 30 and $11 is > 49, then print the line from file1 to a output file. If no match is foung the line is not printed. Both the input and output are tab-delimited.... (3 Replies)
Discussion started by: cmccabe
3 Replies

3. Shell Programming and Scripting

sed - pattern match - apply substitution

Greetings Experts, I am on AIX and in process of creating a re-startable script that connects to Oracle and executes the statements. The sample contents of the file1 is CREATE OR REPLACE VIEW DB_V.TAB1 AS SELECT * FROM DB_T.TAB1; .... CREATE OR REPLACE VIEW DB_V.TAB10 AS SELECT * FROM... (9 Replies)
Discussion started by: chill3chee
9 Replies

4. Shell Programming and Scripting

awk - Compare files in two different directories

Hi, My script works fine when I have both input files in the same directory but when I put on of the input file in another directory, the output does not show up. SCRIPT: awk ' BEGIN { OFS="\t" out = "File3.txt"} NR==FNR && NF {a=$0; next} function print_77_99() { if... (3 Replies)
Discussion started by: High-T
3 Replies

5. Homework & Coursework Questions

Finding the directories with same permission and then apply some default UNIX commands

Write a Unix shell script named 'mode' that accepts two or more arguments, a file mode, a command and an optional list of parameters and performs the given command with the optional parameters on all files with that given mode. For example, mode 644 ls -l should perform the command ls -l on all... (5 Replies)
Discussion started by: femchi
5 Replies

6. Shell Programming and Scripting

Finding the directories with same permission and then apply some default UNIX commands

HI there. My teacher asked us to write a code for this question Write a Unix shell script named 'mode' that accepts two or more arguments, a file mode, a command and an optional list of parameters and performs the given command with the optional parameters on all files with that given mode. ... (1 Reply)
Discussion started by: femchi
1 Replies

7. Shell Programming and Scripting

apply record separator to multiple files within a directory using awk

Hi, I have a bunch of records within a directory where each one has this form: (example file1) 1 2 50 90 80 90 43512 98 0909 79869 -9 7878 33222 8787 9090 89898 7878 8989 7878 6767 89 89 78676 9898 000 7878 5656 5454 5454 and i want for all of these files to be... (3 Replies)
Discussion started by: amarn
3 Replies

8. UNIX for Dummies Questions & Answers

Do UNIX Permission apply to sub directories?

Hi Guys, Can you tell me if unix permissions apply to sub dirs? Dir is /home/ops/batch/files/all /home is rwxrwxrwx ops is rwxrwxrwx batch is rwxr-wr-w files is rwxrwxrwx all is rwxrwxrwx Having problems writing to all (does the userid nee to be the batch owner... (1 Reply)
Discussion started by: Grueben
1 Replies

9. Shell Programming and Scripting

Apply 'awk' to all files in a directory or individual files from a command line

Hi All, I am using the awk command to replace ',' by '\t' (tabs) in a csv file. I would like to apply this to all .csv files in a directory and create .txt files with the tabs. How would I do this in a script? I have the following script called "csvtabs": awk 'BEGIN { FS... (4 Replies)
Discussion started by: ScKaSx
4 Replies

10. Shell Programming and Scripting

AWK Script - Count Files In Directories

Hey, I'm very new to AWK and am trying to write a script that counts the number of files in all subdirectories. So, basically, my root has many subdirectories, and each subdirectory has many files. How can I get the total count? I haven't been able to figure out how to loop through the... (1 Reply)
Discussion started by: beefeater267
1 Replies
Login or Register to Ask a Question