Replacing matched patterns in multiple files with awk


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Replacing matched patterns in multiple files with awk
# 1  
Old 01-12-2016
Replacing matched patterns in multiple files with awk

Hello all,

I have since given up trying to figure this out and used sed instead, but I am trying to understand awk and was wondering how someone might do this in awk.

I am trying to match on the first field of a specific file with the first field on multiple files, and append the second field of the first file on multiple files (preferably inplace).

Example:
file1
Code:
1000/3333,20150101-96
1000/4444,20150102-02
1000/5555,20150103-29
1000/6666,20150104-67

file2
Code:
1000/3333
9999/9999

file3
Code:
1000/6666
8888/8888

Preferred output:
file2
Code:
1000/3333,20150101-96
9999/9999

file3
Code:
1000/6666,20150104-67
8888/8888

I have seen plenty of examples on how to work with two files, or storing one field in an array and only acting on that one field.

I stole the code below and modified it for my purpose, but I still don't understand how I could perform this on multiple files or if it's possible to edit in-place (gawk -i inplace):
Code:
#!/bin/bash
INPUTFILE="/home/username/blah/file1"
DATAFILE="/home/username/blah/file2"
OUTFILE="/home/username/pleasework.out"

awk 'BEGIN {
while (getline < "'"$INPUTFILE"'")
{
split($0,a,",");
name=a[1];
date=a[2];

key=date
data=name
nameofarray[data]=key;
}
close("'"$INPUTFILE"'");

while (getline < "'"$DATAFILE"'")
{
var=nameofarray[$0];
print $0","var > "'"$OUTFILE"'"; 
}
}'

# 2  
Old 01-12-2016
Maybe you wanted something more like:
Code:
#!/bin/bash
# Move to the directory where the input files are located.
cd "/home/username/blah"

awk '
BEGIN {	# Set input and output field separators.
	FS = OFS = ","
}
FNR == NR {
	# Get keys from 1st input file.
	key[$1] = $2
	next
}
function copyback() {
	# After we have read the last line of input from a file, rewrite it
	# with updated contents if anything was changed.
	if(nc) {
		for(i = 1; i <= lines; i++)
			print d[i] > filename
		close(filename)
	}
}
FNR == 1 {
	# When we see the first line of subsequent files, update the previous
	# input file...
	copyback()
	# and get ready for the current input file...
	filename = FILENAME
	lines = nc = 0
}
{	# Gather data from current input file...
	if($1 in key) {
		# Add data gathered from first input file
		d[++lines] = $1 OFS key[$1]
		# Increment the number of changes made.
		nc++
	} else {	# Just save the input line unchanged.
		d[++lines] = $0
	}
}
END {	# When we hit EOF on the last input file, update the last input file...
	copyback()
}' file1 file2 file3

If you want to try this on a Solaris/SunOS system, change awk to /usr/xpg4/bin/awk or nawk.
This User Gave Thanks to Don Cragun For This Post:
# 3  
Old 01-13-2016
This is amazing. Thank you very much Don, and I really do appreciate the details in the comments!! Smilie Smilie
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Issue with search and replacing multiple items in multiple files

Im having an issue when trying to replace the first column with a new set of values in multiple files. The results from the following code only replaces the files with the last set of values in val.txt. I want to replace all the files with all the values. for date in {1..31} do for val in... (1 Reply)
Discussion started by: ncwxpanther
1 Replies

2. Shell Programming and Scripting

sed parser behaving strange on replacing multiple words in multiple files

I have 4000 files like $cat clus_grp_seq10_g.phy 18 1002 anig_OJJ65951_1 ATGGTTTCGCAGCGTGATAGAGAATTGTTTAGGGATGATATTCGCTCGCGAGGAACGAAGCTCAATGCTGCCGAGCGCGAGAGTCTGCTAAGGCCATATCTGCCAGATCCGTCTGACCTTCCACGCAGGCCACTTCAGCGGCGCAAGAAGGTTCCTCG aver_OOF92921_1 ... (1 Reply)
Discussion started by: sammy777888
1 Replies

3. Shell Programming and Scripting

Replacing multiple line patterns with awk

Hi forum, Can you please help me understand how to look for and replace the below pattern (containing line breaks) and return a new result? Rules: Must match the 3 line pattern and return a 1 line result. I have found solutions with sed, but it seems that sed installed in my system is... (5 Replies)
Discussion started by: demmel
5 Replies

4. Shell Programming and Scripting

Find matched patterns and print them with other patterns not the whole line

Hi, I am trying to extract some patterns from a line. The input file is space delimited and i could not use column to get value after "IN" or "OUT" patterns as there could be multiple white spaces before the next digits that i need to print in the output file . I need to print 3 patterns in a... (3 Replies)
Discussion started by: redse171
3 Replies

5. Shell Programming and Scripting

Matched multiple patterns that could be in a same line

Hi, I need help to match pattern started with "RW" in file 1 and with pattern in $1 in file 2 as follows:- File 1 BH /TOTAL=466(423); /POSITIVE=300(257); /UNKNOWN=25(25); BH /F_P=141(141); /F_N=136; /P=4; CC /TAX=!?; /MAX-R=2; CC /VER=2; RW P9610, AR_BSU , T; PAE25, AE_E57... (10 Replies)
Discussion started by: redse171
10 Replies

6. Shell Programming and Scripting

Find matched patterns in a column of 2 files with different size and merge them

Hi, i have input files like below:- input1 Name Seq_ID NewID Scores MT1 A0QZX3 1.65 277.4 IVO A0QZX3 1.65 244.5 HPO A0QZX3 1.65 240.5 RgP A0Q3PP 5.32 241.0 GX1 LPSZ3S 96.1 216.9 MEL LPSS3X 4.23 204.1 LDD LPSS3X 4.23 100.2 input2 Fac AddName NewID ... (9 Replies)
Discussion started by: redse171
9 Replies

7. Shell Programming and Scripting

Find matched patterns in multiple files

Hi, I need help to find matched patterns in 30 files residing in a folder simultaneously. All these files only contain 1 column. For example, File1 Gr_1 st-e34ss-11dd bt-wwd-fewq pt-wq02-ddpk pw-xsw17-aqpp Gr_2 srq-wy09-yyd9 sqq-fdfs-ffs9 Gr_3 etas-qqa-dfw ddw-ppls-qqw... (10 Replies)
Discussion started by: redse171
10 Replies

8. Shell Programming and Scripting

How to group matched patterns in different files

Hi, I have a master file that i need to split into multiple files based on matched patterns. sample of my data as follows:- scaff_1 a e 123 130 c_scaff_100 scaff_1 a e 132 138 c_scaff_101 scaff_1 a e 140 150 ... (2 Replies)
Discussion started by: redse171
2 Replies

9. Shell Programming and Scripting

Search multiple patterns in multiple files

Hi, I have to write one script that has to search a list of numbers in certain zipped files. For eg. one file file1.txt contains the numbers. File1.txt contains 5,00,000 numbers and I have to search each number in zipped files(The number of zipped files are around 1000 each file is 5 MB) I have... (10 Replies)
Discussion started by: vsachan
10 Replies

10. UNIX for Dummies Questions & Answers

best method of replacing multiple strings in multiple files - sed or awk? most simple preferred :)

Hi guys, say I have a few files in a directory (58 text files or somthing) each one contains mulitple strings that I wish to replace with other strings so in these 58 files I'm looking for say the following strings: JAM (replace with BUTTER) BREAD (replace with CRACKER) SCOOP (replace... (19 Replies)
Discussion started by: rich@ardz
19 Replies
Login or Register to Ask a Question