awk comparison and substitution


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting awk comparison and substitution
# 1  
Old 11-15-2011
awk comparison and substitution

Hi,
here's my - not so easy to describe - problem: I want to compare the values of one file (FileA) with a cutoff-value and, if this comparison is true, substitute those values with those in the second file (FileB). However, there are many FileA's (FileA[1->200]), whereas there is only one FileB. Every FileA has three lines, each containing one value.
Code:
3.4
3.5
3.6

FileB has 3 columns and > 200 lines.
Code:
0.0154    0.0139    0.0227
0.0198    0.0259    0.0231
0.0126    0.0216    0.0174
0.0115    0.0145    0.0237
...            ...            ...

The three values of FileA1 should now be compared line-by-line with the cutoff-value. If true, the corresponding value of FileB should be assigned. However, those corresponding values are contained within a line of FileB. So I now need some kind of script which substitutes line x of FileA[1] (if value > or < cutoff) with the field in line[1] and cloumn x.
My script so far:
Code:
# getting number of line of FileB in which values of FileA are contained

n=`echo "$1" |sed 's/.*\([0-9]\{1,3\}\).*/\1/'`

# comparison and substitution

awk -v val=$n '{
    getline < "$1"
        for(i=1; i<=NF; i++){
            if($i >= 3.5){
                print $i 
            }
        else{
            getline < "FileB.txt"
            NR==n {print $i}
            }
        }
        } ' $1 FileB.txt > $1_new.txt

since i'm a beginner in awk, it's very intuituve aaaand - of course - doesn't work.

output should look something like this:
Code:
0.0154
3.5
3.6

Any help would be greatly appreciated!

waddle
# 2  
Old 11-15-2011
I still don't understand, why 3.4 is choiced, and replaced by 0.0154. Why not 3.5 or 3.6. What's the cutoff value in each line in fileB

can you explain more detail?
# 3  
Old 11-15-2011
The script should chek FileA[1] for a line containing a value greater or equal to a value, in this example 3.5. If this is true, the original value should be printed, if it's false, this specific value (in this example in line 1 of FileA[1]) should be replaced with the corresponding value in FileB, here line1 (since its FileA[1]), column 1 (first value in FileA). Again, notice that the corresponding values are listed in one column in FileA and one line in FileB.

I hope, it's better to understand now...
# 4  
Old 11-15-2011
Try this:
Code:
# cat FileA1
3.4
3.5
3.6

Code:
# cat FileA2
3.1
3.3
3.8

Code:
awk -v val=3.5 '
NR==FNR{o[NR""1]=$1
        o[NR""2]=$2
        o[NR""3]=$3
        next}
{
if ( FILENAME  != lFN ) 
   L++
val+=0
cmp=$1+0
if ( val <= cmp ) 
   print 
else 
   print o[L""FNR] 
lFN=FILENAME}' FileB FileA*
0.0154
3.5
3.6
0.0198
0.0259
3.8

# 5  
Old 11-15-2011
Thanks for your time and work Klashxx,

the code works, though I now have some new problems
  1. i don't really understand it (but i'll try to)
  2. the code only works when i paste it into the shell, not when i try to run the script (but that's not the main point)
  3. i have to type in all FileA's (>200) consecutively (s. 4.) (also not the main point)
  4. i can't compare one FileA individually: if I take FileA[145], the values (if necessary) become substituted with those of line 1 from FileB, not with line 145
Thanks again,
waddle
# 6  
Old 11-15-2011
The basic thing here is the naming of the files, you need a constant pattern , say FileA1,FileA2,FileA3,...FileA200.
Code:
# cat FileA4
3.6
3.3
3.8

Code:
# cat ren.sh             
#!/usr/bin/ksh

value="${1}"
patFileA="${2}"
FileB="${3}"

awk -v val="${value}" '
NR==FNR{o[NR""1]=$1
        o[NR""2]=$2
        o[NR""3]=$3
        next}
{
if ( FILENAME  != lFN ) 
   extF=substr(FILENAME,match(FILENAME,/[0-9]/))
val+=0
cmp=$1+0
if ( val <= cmp ) 
   print 
else 
   print o[extF""FNR] 
lFN=FILENAME}' ${FileB} ${patFileA}*

Code:
# ren.sh 3.5 FileA4 FileB
3.6
0.0145
3.8

Use
Code:
ren.sh 3.5 FileA FileB

to process all the files.

Last edited by Klashxx; 11-15-2011 at 11:15 AM..
# 7  
Old 11-15-2011
Hi Klashxx,
I took your code and modified it slightly for my purposes. The files indeed have a pattern in their naming: [0-9]{1,3}[A-Z]{3}. Your code gives me correct output solely for FileA's, in which the comparison is true.
For FileA no. 2 however, I always get the same output: in case of comparison is true, i get the three values contained in this file plus as much "2"s as FileB has lines. In case that the comparison is not true, i get as many empty lines as FileB has lines.
Do you have any explanation for this finding?
cheers,
waddle
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

File comparison using awk

Hi All, i have two files file1 ,file 2 file 1 col1|col2|col3|col4|col5|col6|col7|col8 11346925|0|2009-09-20|9999-12-31|100|0 11346925|0|2009-09-20|9999-12-31|120|0 12954311|0|2009-09-11|9999-12-31|100|0 12954311|0|2009-07-23|2999-12-31|120|0 12954312|0|2009-09-11|9999-12-31|100|0... (9 Replies)
Discussion started by: mohanalakshmi
9 Replies

2. Shell Programming and Scripting

awk substitution

Hi all, I need some help with substitution in awk. Is it possible to substitute field from awk output with string from file? For example: zcat /SMS/CDR/cdr_TC/callLogs*_*_2013092710*.gz | sed 's/:/;/g' | awk -F";" '{if($2==1 && $10~/389123456789/) print $36";"$37}' 2;19733248 I want... (6 Replies)
Discussion started by: vasil
6 Replies

3. Shell Programming and Scripting

awk comparison not working

Can you please help me on belw awk comparsion which doest not work cat employee_list NAME Last-login Jack 03/25/2013 Maneul 03/26/2013 Eric 03/26/2013 Samuel 03/28/2013 loak 03/29/2013 zac 03/29/2013 this is my awk .. it gives me error cat employee_list | awk '(($2=='date... (3 Replies)
Discussion started by: Sara_84
3 Replies

4. Shell Programming and Scripting

Need help with simple comparison in AWK

Hi, I'm new to AWK and I'm having problems comparing a field to a string variable. /ARTIST/ {x = $2} $1 ~ x {print $0}My code tries to find a record with the string "ARTIST". Once it finds it, it stores the second field of the record into a variable. I don't know what the problem is for the... (7 Replies)
Discussion started by: klusps
7 Replies

5. Shell Programming and Scripting

AWK string comparison

Hey guys.. New in linux scripting and need some help on some scripting with history command. I managed to export the command history into a file and now i'm trying to select from that file some specific commands that were made in a certain period. Here's what i got so far echo -n... (2 Replies)
Discussion started by: mishu_cgm
2 Replies

6. Shell Programming and Scripting

awk comparison

Hello all, Probably a very simple question, I am stuck with a small part of a code: I am trying to do a comparison to get the maximum value of column 6 if columns 1, 4 and 5 of two or more rows match. Here is what I am doing: awk -F'\t' '{if ($6 > a)a=$6}END{for (i in a) print i"\t"a}' ... (4 Replies)
Discussion started by: jaysean
4 Replies

7. Shell Programming and Scripting

Comparison and editing of files using awk.(And also a possible bug in awk for loop?)

I have two files which I would like to compare and then manipulate in a way. File1: pictures.txt 1.1 1.3 dance.txt 1.2 1.4 treehouse.txt 1.3 1.5 File2: pictures.txt 1.5 ref2313 1.4 ref2345 1.3 ref5432 1.2 ref4244 dance.txt 1.6 ref2342 1.5 ref2352 1.4 ref0695 1.3 ref5738 1.2... (1 Reply)
Discussion started by: linuxkid
1 Replies

8. Shell Programming and Scripting

Substitution in AWK

I am trying to use AWK to replace dallinux02 to dallinux03 everywhere in the servers.txt file and move it over to "awk2". Here is my script "awk2.awk": gsub(/dallinux02/, "dallinux03"); print > "awk2" I am trying to run this using the following: $ awk -f awk2.awk... (3 Replies)
Discussion started by: ora_umair
3 Replies

9. Shell Programming and Scripting

help with awk substitution

Hi again. A have a CSV-file in the following format: 2008.09.01,15:17:42,9227096485,9233175320,CTC10,SMS,0901151742098314,Target_MSIS DN_is_blacklisted I want to have an awk command that will say: If the first 3 digits of $4 does not begin with 922 or 923, then make $8 say "Invalid... (3 Replies)
Discussion started by: daytripper1021
3 Replies

10. Shell Programming and Scripting

AWK substitution

I need to copy field 2 to field 3 for only those records that have the 1st field equal to account e.g. file account|123|789|xxx|yyy|zzz|... account_group|444|555|xxx|yy|zz|.... account|456|901|aaa|bbb|ccc|..... after running awk script should look like account|123|123|xxx|yyy|zzz|...... (4 Replies)
Discussion started by: klut
4 Replies
Login or Register to Ask a Question