Sponsored Content
Top Forums Shell Programming and Scripting Need help with filtering records in a file Post 303009757 by Don Cragun on Tuesday 19th of December 2017 12:24:06 PM
Old 12-19-2017
Quote:
Originally Posted by manid
I just want to know , is it possible to add column to the end of each record with the filename like below?


Code:
echo $row | grep -F "setting application" | grep -vF "@msad" | awk -v var="$file" '{print $0 var}'  >> test.txt

Code:
[123] [AGENT] [abc] [cde] [fgh] [asd] [asd] setting application ABC for user [user1@ldap1] file1.txt
[345] [AGENT] [abc] [cde] [fgh] [asd] [asd] setting application CDE for user [user2@ldap2] file1.txt

If you are reading a file line-by-line as suggested by MadeInGermany in post #2, please never feed those lines through a three element pipeline. Doing so is GROSSLY inefficient!

If you are processing multiple files (which would be a logical reason for adding the name of the file each record came from in the output you produce), you can easily modify MadeInGermany's shell suggestion from post #2 with something like:
Code:
for file in file*.txt
do	while IFS="" read row
	do	case $row in
		(*" setting application "*)
			case $row in
			(*"@msad"*)
				;;
			(*)	printf "%s %s\n" "$row" "$file";;
			esac;;
		esac
	done < "$file"
done > tempfile.txt

(which just uses shell built-ins and doesn't need to invoke any external utilities) or extend my awk suggestion from post #3 to something like:
Code:
awk '/ setting application / && ! /@msad/ { print $0, FILENAME }' file*.txt > tempfile.txt

(which just invokes awk once). Note the comma between $0 and FILENAME in the print statement; without it, there won't be any separation between the input lines and the name of the file from which it was extracted.

The code that you showed us in post #4 invokes two copies of grep and one copy of awk for every line read from each of the files you process.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Count No of Records in File without counting Header and Trailer Records

I have a flat file and need to count no of records in the file less the header and the trailer record. I would appreciate any and all asistance Thanks Hadi Lalani (2 Replies)
Discussion started by: guiguy
2 Replies

2. UNIX for Dummies Questions & Answers

Use records from one file to delete records in another file

file_in_1: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 file_in_2: 9 10 11 12 21 22 23 24 1 2 3 4 17 18 19 20 file_out: (5 Replies)
Discussion started by: kenneth.mcbride
5 Replies

3. UNIX for Dummies Questions & Answers

Filtering records of a file based on a value of a column

Hi all, I would like to extract records of a file based on a condition. The file contains 47 fields, and I would like to extract only those records that match a certain value in one of the columns, e.g. COL1 COL2 COL3 ............... COL47 1 XX 45 ... (4 Replies)
Discussion started by: risk_sly
4 Replies

4. Shell Programming and Scripting

Issues with filtering duplicate records using gawk script

Hi All, I have huge trade file with milions of trades.I need to remove duplicate records (e.g I have following records) 30/10/2009,trdeId1,..,.. 26/10/2009.tradeId1,..,..,, 30/10/2009,tradeId2,.. In the above case i need to filter duplicate recods and I should get following output.... (2 Replies)
Discussion started by: nmumbarkar
2 Replies

5. Shell Programming and Scripting

filtering records based on numeric field value in 8th position

I have a ";" delimited file.Whcih conatins a number fileds of length 4 charcters in 8th position But there is a alphanumeric charcters like : space, ";" , "," , "/" , "23-1" , "23 1" , "aqjhdj" , "jun-23" , "APR-04" , "4:00AM" , "-234" , "56784 ", "." , "+" "_" , "&" , "*" , "^" , "%" , "!"... (2 Replies)
Discussion started by: indusri
2 Replies

6. UNIX for Dummies Questions & Answers

Grep specific records from a file of records that are separated by an empty line

Hi everyone. I am a newbie to Linux stuff. I have this kind of problem which couldn't solve alone. I have a text file with records separated by empty lines like this: ID: 20 Name: X Age: 19 ID: 21 Name: Z ID: 22 Email: xxx@yahoo.com Name: Y Age: 19 I want to grep records that... (4 Replies)
Discussion started by: Atrisa
4 Replies

7. UNIX for Dummies Questions & Answers

Filtering records from 1 file based on some manipulation doen on second file

Hi, I am looking for an awk script which should help me to meet the following requirement: File1 has records in following format INF: FAILEd RECORD AB1234 INF: FAILEd RECORD PQ1145 INF: FAILEd RECORD AB3215 INF: FAILEd RECORD AB6114 ............................ (2 Replies)
Discussion started by: mintu41
2 Replies

8. Shell Programming and Scripting

Deleting duplicate records from file 1 if records from file 2 match

I have 2 files "File 1" is delimited by ";" and "File 2" is delimited by "|". File 1 below (3 record shown): Doc1;03/01/2012;New York;6 Main Street;Mr. Smith 1;Mr. Jones Doc2;03/01/2012;Syracuse;876 Broadway;John Davis;Barbara Lull Doc3;03/01/2012;Buffalo;779 Old Windy Road;Charles... (2 Replies)
Discussion started by: vestport
2 Replies

9. Shell Programming and Scripting

Separate records of a file on 2 types of records

Hi I am new to shell programming in unix Please if I can provide help. I have a file structure of a header record and "N" detail records. The header record will be the total number of detail records I need to split the file in 2: One for the header Another for all detail records Could... (1 Reply)
Discussion started by: jamcogar
1 Replies

10. UNIX for Beginners Questions & Answers

Filtering records of a csv file based on a value of a column

Hi, I tried filtering the records in a csv file using "awk" command listed below. awk -F"~" '$4 ~ /Active/{print }' inputfile > outputfile The output always has all the entries. The same command worked for different users from one of the forum links. content of file I was... (3 Replies)
Discussion started by: sunilmudikonda
3 Replies
All times are GMT -4. The time now is 11:14 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy