Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

awk sed issue

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting awk sed issue
# 1  
Old 04-11-2012
awk sed issue
Hi,

I am having some difficulty piping the results of an awk search to sed and am hoping for some help. Essentially, I have a file that contains a key and value and another file that contains the key, I want to replace the key in the second file with the value for the first.

It seems rediculous to write a perl script for this, but my awk/sed is killing me.

Here is my current iteration:

Code:
gawk '{print $1, $2 }' test2.references | sed -i 's/\1/\2/' test.bln.txt

With associated error:

Code:
sed: -e expression #1, char 8: Invalid back reference

I have tried $1 and $2 with and without single and double quotes in the sed part and get variations on Smilie

Thanks in advance,
Bob

Moderator's Comments:
Mod Comment Video tutorial on how to use code tags in The UNIX and Linux Forums.
Last edited by radoulov; 04-11-2012 at 06:03 PM..
# 2  
Old 04-11-2012
I can't figure out what those variables/backreferences are supposed to be, either. Show the input you have and the output you want. We can't figure out your intent from a program which doesn't do what you want.
# 3  
Old 04-11-2012
Yes, I should have done that.

File 1:
Code:
ctg7180000468805 ctg7180000468805_511
ctg7180000468807 ctg7180000468807_490
ctg7180000468809 ctg7180000468809_530
ctg7180000468810 ctg7180000468810_531
ctg7180000468811 ctg7180000468811_482
ctg7180000468813 ctg7180000468813_575
ctg7180000468814 ctg7180000468814_500
ctg7180000468816 ctg7180000468816_507
ctg7180000468817 ctg7180000468817_516
ctg7180000468818 ctg7180000468818_430

File 2:
Code:
Transcript_1 ctg7180000468805 219 5449 5667 1e-108 396
Transcript_2 ctg7180000468807 129 5540 5667 2e-57 224

So, again, file 1 contains 2 fields. A reference name and a slightly modified version of the reference. File 2 has as the second field, references. I want to replace the reference name in file 2 with the modified version in file 1.

The new file 2 would change to:
Code:
Transcript_1 ctg7180000468805_511  219 5449 5667 1e-108 396
Transcript_2 ctg7180000468807_490  129 5540 5667 2e-57 224

Bob
Moderator's Comments:
Mod Comment Code tags for code, please.
# 4  
Old 04-11-2012
Okay, I think I see what you want.

The NR==FNR thing is a bit of a trick -- NR is the number of records, and FNR is the number of records relative to the start of the file, so they're guaranteed to be the same ONLY for the first file. So you can use that as a 'loader' section to slurp the contents of file1 into an array for reference. Afterwards, when FNR != NR, you can look up the contents of the second column, replace, and print (the 1 on the end is an implied 'print')
Code:
awk 'NR==FNR { A[$1]=$2; next }; A[$2] { $2=A[$2] } 1' file1 file2 > file2_new

You shouldn't read and write to the same file simultaneously. Overwrite the old file afterwards if need be (but check first)
# 5  
Old 04-11-2012
Code:
awk 'NR == FNR {
  f1[$1] = $2
  next
  }
$2 in f1 {
  $2 = f1[$2]
  }1' file1 file2

You may loose some formatting when assigning a value to a field
(for example, consecutive white space characters will be squeezed).
Let us know if this is an issue.
Last edited by radoulov; 04-11-2012 at 06:22 PM..
# 6  
Old 04-11-2012
Hmmm. That did the trick, although I would love to be able to read through the logic of that awk line.
# 7  
Old 04-11-2012
Quote:
Originally Posted by Corona688
[...]
Code:
awk 'NR==FNR { A[$1]=$2; next }; A[$2] { $2=A[$2] } 1' file1 file2 > file2_new

[...]
Just a side note - in the above script the expression A[$2] will:

1. Create a new array element if one doesn't exist.
2. Will not produce the desired result when/if it evaluates false (if the associated value is 0 for instance).
Last edited by radoulov; 04-11-2012 at 06:47 PM..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

sed and awk giving error ./sample.sh: line 13: sed: command not found

Hi, I am running a script sample.sh in bash environment .In the script i am using sed and awk commands which when executed individually from terminal they are getting executed normally but when i give these sed and awk commands in the script it is giving the below errors :- ./sample.sh: line... (12 Replies)
Discussion started by: satishmallidi
12 Replies

2. Shell Programming and Scripting

Variable value substitution issue with awk command issue

Hi All, I am using the below script which has awk command, but it is not returing the expected result. can some pls help me to correct the command. The below script sample.ksh should give the result if the value of last 4 digits in the variable NM matches with the variable value DAT. The... (7 Replies)
Discussion started by: G.K.K
7 Replies

3. Shell Programming and Scripting

sed issue

I'm trying to change a date in a couple of large files using SED. The problem is when I use the -n parameter, it doesn't actually change the file. When I leave out the -n, it sends the whole file to the screen, but it does appear to change it. The problem is, these files are very large and it... (8 Replies)
Discussion started by: Drenhead
8 Replies

4. Shell Programming and Scripting

sed issue

Hi guys. Can somone advise as to what the problem is with the following sed command? 1) read -p "Please enter new username you wish to replace old: " new_username sed "s/$username/$new_username/" information_file ;; This is one of the case statements included but I'm... (1 Reply)
Discussion started by: jjb1989
1 Replies

5. Shell Programming and Scripting

one more issue last- sed

hi i have following sed command this replaces "** in filename1 with octal value 007 filename2 when i put it in script it wont work but it works from command line my OS is sun OS ---------- Post updated at 06:38 PM ---------- Previous update was at 06:14 PM ---------- i... (10 Replies)
Discussion started by: er_zeeshan05
10 Replies

6. Shell Programming and Scripting

Sed Issue

Hi, I am trying to use 3 sed statements in a shell script, but it get foll error. sed : garbage after command. If I use only two sed statements, the script works well. Is there any restriction for sed usage or is there some catch which I am missing. Sample Script is as follows : ... (3 Replies)
Discussion started by: sameersalve
3 Replies

7. Shell Programming and Scripting

sed issue

Hi All I'm getting this error while executing a sed script sed: 0602-404 Function /</ i\ File from New Cube: cannot be parsed. sed "/</ i\ File from New Cube: />/ i\ File from Old Cube:" difference1.txt > Difference.txt I've a file like this < Y2008 Dec ..... .... ... 345 I want... (6 Replies)
Discussion started by: Celvin VK
6 Replies

8. Shell Programming and Scripting

Sed Issue....

Can someone help me "port" this to AIX sed? sed '/nas/{n;s/true/false/}' I know it doesn't like the ; but i don't know how else to do it.... never had to sed on an AIX box :D (7 Replies)
Discussion started by: DeviousPete
7 Replies

9. Shell Programming and Scripting

Issue with a sed one liner variant - sed 's/ ; /|/g' $TMP1 > $TMP

Execution of the following segment is giving the error - Script extract:- OUT=$DATADIR/sol_rsult_orphn.bcp TMP1=${OUT}_tmp1 TMP=${OUT}_tmp ( isql -w 400 $dbConnect_OPR <<EOF select convert(char(10), s.lead_id) +'|' + s.pho_loc_type, ";", s.sol_rsult_cmnt, ";", +'|'+ s.del_ind... (3 Replies)
Discussion started by: kzmatam
3 Replies

10. UNIX for Dummies Questions & Answers

SED Issue

Can anyone tell me ...on the below listed command cat /mnt/winbox/list_measurement/ds1c/ds1_f.rome_27A03A 2>> error_log | sed -e '1,3d;s/^/27A03A,/' | sed -e "s#\(.*\)#\1 ,$(date +%Y-%m-%d)#g" > /SBS/ds1_f.rome_27A03A The outcome is this: ,2005-08-29 Forestdale,3:02 am MON AUG 29,... (9 Replies)
Discussion started by: Redg
9 Replies
Login or Register to Ask a Question