Grepping for one variable while using awk to parse an associated variable


 
Thread Tools Search this Thread
Top Forums UNIX for Beginners Questions & Answers Grepping for one variable while using awk to parse an associated variable
# 1  
Old 01-03-2020
Grepping for one variable while using awk to parse an associated variable

Im trying to search for a single variable in the first field and from that output use awk to extract out the lines that contain a value less than a value stored in another variable. Both the variables are associated with each other.

Any guidance is appreciated.

File that contains the variables
variables.txt

Code:
01001 63.9
01003 66.8
01005 64.8
01007 62.8

file-to-search.txt

Code:
01001 02  1900 65.6
01001 02  1901 60.6
01001 02  1902 62.6
....
01003  02  1900 65.1
01003  02  1901 65.3
01003  02  1902 67.3
....
01005  02  1900 64.7
01005  02  1901 65.2
01005  02  1902 65.2
...
01007  02  1900 64.8
01007  02  1901 63.8
01007  02  1902 62.7
....

I was using a for loop but realized that the resulting output was not correct.
Code:
county=$(awk '{ print $1 }' variables.txt)
val=$(awk '{ print $2 }' variables.txt)

for id in $county
do
grep "$id" file-to-search.txt | awk '$4 < $val { print $0 }' | tail -1 | awk '{print $1" " $3" " $4}'
done

What im hoping for is a file that looks like this
Code:
01001 1902 62.6
01003 1901 65.3
01005 1900 64.7
01007 1902 62.7

# 2  
Old 01-03-2020
Cannot really correlate your explanation with your sample output, but.... to start with:
Code:
awk '
FNR==NR { f1[$1]=$2; next}
$1 in f1 && $4 < f1[$1]' variables.txt file-to-search.txt

# 3  
Old 01-03-2020
Just guessing - do you want to print the line with the greatest $4 value less than the one in variables.txt (why else would 01001 02 1901 60.6 and 01003 02 1900 65.1 be missing in your desired output)? Try
Code:
sort -k1,1 -k4,4r file2 |
awk '
FNR==NR         {T[$1] = $2
                 next
                }
($1 in T) &&
($4 <= T[$1])   {print $1, $3, $4
                 delete T[$1]
                }
' file1 -
01001 1902 62.6
01003 1901 65.3
01005 1900 64.7
01007 1902 62.7

If that's not the case, PLEASE become way clearer and more precise in your specifications!
# 4  
Old 01-03-2020
For each ID in column 1 of the avgs.txt file, I need to match id's to find the lines in file-to-search.txt whose value in column 4 is the next lowest to the value in column 2 of the avgs.txt file when sorted by column 3 (column 3 contains the year and I need to know the closest year, to 2019, whose value in column 4 contains the next lowest value). I need to print all columns and the output is only the line that contains the next lowest value (in column 4) when compared to the first file (column 2) assuming the year is sorted properly.

Last edited by ncwxpanther; 01-04-2020 at 10:15 AM..
# 5  
Old 01-04-2020
Code:
cat p.py
#!/usr/bin/env python3

data = {}
with open("file-to-search.txt") as f2s:
    for line in f2s:
        fields = line.split()
        fields.pop(1)

        if fields[0] in data:
            data[fields[0]].update({fields[2]: " ".join(fields)})
        else:
            data[fields[0]] = {fields[2]: " ".join(fields)}

points = {}
for entry in data:
    scores = []
    for s in data[entry]:
        scores.append(s)
    scores.sort(reverse=True)
    points[entry] = scores

with open("variable.txt") as s2f:
    for line in s2f:
        target, score = line.split()
        for s in points[target]:
            if score > s:
                print(data[target][s])
                break

Run:
Code:
python3 p.py
01001 1902 62.6
01003 1901 65.3
01005 1900 64.7
01007 1902 62.7

# 6  
Old 01-05-2020
Rudi

Id like to print the greatest $4 value less than the one in variables.txt but searching from the 2019 value in $3 in decreasing order, i.e., 2019, 2018, 2017...
# 7  
Old 01-05-2020
I found a solution that works for me. In my previous attempts I was not using an awk variable with -v. Thanks.

Code:
for id in {`cat counties.txt`}; do grep $id county-annual-average.txt > $id.txt; val=$(grep $id avgs.txt | awk '{print $2 }'); awk -v val="$val" '$4 < val' ${id}.txt | tail -1 | awk '{print $1" " $3" "$4}'; done

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

How can I assign awk's variable to shell script's variable?

I have the following script, and I want to assign the output ($10 and $5) from awk to N and L: grdinfo data.grd | awk '{print $10,$5}'| read N L output from gridinfo data.grd is: data.grd 50 100 41 82 -2796 6944 0.016 0.016 3001 2461. where N and L is suppose to be 3001 and 100. I use... (8 Replies)
Discussion started by: geomarine
8 Replies

2. Shell Programming and Scripting

How to handle grepping variable data containing wildcards?

I have a lot of files with keywords and unique names. I'm using a shell script to refer to a simple pattern file with comma separated values in order to match on certain keywords. The problem is that I don't understand how to handle the wildcard values when I want to skip over the unique names. ... (5 Replies)
Discussion started by: abercrom
5 Replies

3. Shell Programming and Scripting

Grepping the file into a variable after SSH into a server

Hi All, When i am logged into a server , i am able to assign grep value to a variable as follows. VAR=`grep 'Listener stopped' /logs/perf.log` However, when i log out of the server, and try to execute the following command by first SSHing into server, it throws error. $ VAR=`ssh Server... (4 Replies)
Discussion started by: srkmish
4 Replies

4. Shell Programming and Scripting

awk print variable then fields in variable

i have this variable: varT="1--2--3--5" i want to use awk to print field 3 from this variable. i dont want to do the "echo $varT". but here's my awk code: awk -v valA="$varT" "BEGIN {print valA}" this prints the entire line. i feel like i'm so close to getting what i want. i... (4 Replies)
Discussion started by: SkySmart
4 Replies

5. Shell Programming and Scripting

Grepping for variable in brackets

I have a script which reads a number out of a log file. The pertinent line is this: cat /tmp/listofnumbers When I run cat /tmp/listofnumbers what I am seeing is on each line. I am trying to make the script read from that file and grep for a variable like the following line should do: ... (4 Replies)
Discussion started by: newbie2010
4 Replies

6. Shell Programming and Scripting

Help about parse the variable

I'm using bash the code is QEMU_CMD="qemu-system-x86_64 -smp 2 -m 512 $QEMU_PARAMETER -hda $GUEST_IMAGE -kernel $GUEST_KERNEL -append \"root=/dev/hda rw console=ttyS0,115200 ip=$IP_PARAMETER \" -nographic" echo "..............................." echo "qemu command is... (9 Replies)
Discussion started by: yanglei_fage
9 Replies

7. Shell Programming and Scripting

using awk for setting variable but change the output of this variable within awk

Hi all, Hope someone can help me out here. I have this BASH script (see below) My problem lies with the variable path. The output of the command find will give me several fields. The 9th field is the path. I want to captured that and the I want to filter this to a specific level. The... (6 Replies)
Discussion started by: Cowardly
6 Replies

8. Shell Programming and Scripting

Grepping Date Variable

Hello, I would like for the user to input the date for a particular log file, then have the input sent to a variable, which is then used via grep to extra the logs for the specific date the user request. I did some searching, but I still don't understand why I'm not seeing any result. ... (4 Replies)
Discussion started by: ravzter
4 Replies

9. Shell Programming and Scripting

assign awk's variable to shell script's variable?

Dear All, we have a command output which looks like : Total 200 queues in 30000 Kbytes and we're going to get "200" and "30000" for further process. currently, i'm using : numA=echo $OUTPUT | awk '{print $2}' numB=echo $OUTPUT | awk '{print $5}' my question is : can I use just one... (4 Replies)
Discussion started by: tiger2000
4 Replies

10. UNIX for Dummies Questions & Answers

grepping a variable

I need to pass a parameter that will then be grepped. I need it to grep /paramater and then have a space so if 123 was passed my grep would be grep '/123 ' sample.log this works fine from the command line but when i try and set it searchThis="/$2 " and then run grep $searchThis... (6 Replies)
Discussion started by: magnia
6 Replies
Login or Register to Ask a Question