Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Need help with filtering records in a file

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Need help with filtering records in a file
# 1  
Old 12-19-2017
Need help with filtering records in a file
Hi,

I have following records in a file

more file1.txt
Code:
[123] [AGENT] [abc] [cde] [fgh] [asd] [asd] setting applicaction ABC for user [user1@ldap1]
[345] [AGENT] [abc] [cde] [fgh] [asd] [asd] setting applicaction CDE for user [user2@ldap2]
[678] [AGENT] [abc] [cde] [fgh] [asd] [asd] setting applicaction XXX for user [user3@msad]
[628] [AGENT] [abc] [cde] [fgh] [asd] [asd] logging applicaction XXX for user [user3@ldap1]


I need to filter out records which have strings " setting application " and records which don't have @msad user.(elilminate records with uses@msad)

Output:

I tried below code

Code:
while read row;
do
echo $row | egrep "setting application" | awk -F@ldap '{print $0}' >> tempfile.txt

done < file1.txt

output am getting:

Code:
[123] [AGENT] [abc] [cde] [fgh] [asd] [asd] setting applicaction ABC for user [user1@ldap1]
[345] [AGENT] [abc] [cde] [fgh] [asd] [asd] setting applicaction CDE for user [user2@ldap2]
[678] [AGENT] [abc] [cde] [fgh] [asd] [asd] setting applicaction XXX for user [user3@msad]


Desired output:

Code:
[123] [AGENT] [abc] [cde] [fgh] [asd] [asd] setting applicaction ABC for user [user1@ldap1]
[345] [AGENT] [abc] [cde] [fgh] [asd] [asd] setting applicaction CDE for user [user2@ldap2]



But this is not working as expected, its printing all the records.
Last edited by manid; 12-19-2017 at 01:02 AM..
# 2  
Old 12-19-2017
Your awk command sets a field delimiter that accordingly splits the line into $1 and $2 but then you split the whole line $0.

Most simple is a combination of grep and grep -v
Code:
egrep " setting application " file1.txt | egrep -v "@msad" > tempfile.txt

Also doable with shell built-ins.
Code:
while IFS="" read row
do
  case $row in
  *" setting application "*)
    case $row in
    *"@msad"*)
    ;;
    *)
      printf "%s\n" "$row"
    ;;
    esac
  ;;
  esac
done < file1.txt > tempfile.txt

# 3  
Old 12-19-2017
If you're going to use two greps, with the fixed strings needed for this project, grep -F (or fgrep) will be faster than egrep (or grep -E). I.e., try:
Code:
grep -F ' setting application ' file1.txt | grep -vF '@mead' > tempfile.txt

If you don't want to do this just with shell built-ins as MadeInGermany suggested, you could also use either of the following awk scripts (that just need one invocation of awk):
Code:
awk -F'@msad' 'NF == 1 && / setting application /' file1.txt > tempfile.txt
awk '/ setting application / && ! /@msad/' file1.txt > tempfile.txt

Note, however, that with the supplied sample data from post #1 in file1.txt, none of the above produce any output.

All of these (and the sample code shown in post #1 are looking for setting application , but the desired lines in that sample data have applicaction instead of application

To get the output requested in post #1 from the input provided in post #1, you would need to use one of the following:
Code:
grep -F ' setting applicaction ' file1.txt | grep -vF '@mead' > tempfile.txt
awk -F'@msad' 'NF == 1 && / setting applicaction /' file1.txt > tempfile.txt
awk '/ setting applicaction / && ! /@msad/' file1.txt > tempfile.txt

This User Gave Thanks to Don Cragun For This Post:
manid
# 4  
Old 12-19-2017
I just want to know , is it possible to add column to the end of each record with the filename like below?


Code:
echo $row | grep -F "setting application" | grep -vF "@msad" | awk -v var="$file" '{print $0 var}'  >> test.txt

Code:
[123] [AGENT] [abc] [cde] [fgh] [asd] [asd] setting application ABC for user [user1@ldap1] file1.txt
[345] [AGENT] [abc] [cde] [fgh] [asd] [asd] setting application CDE for user [user2@ldap2] file1.txt


Moderator's Comments:
Mod Comment Please use CODE tags as required by forum rules!
Last edited by RudiC; 12-19-2017 at 12:37 PM.. Reason: Added CODE tags.
# 5  
Old 12-19-2017
What happens if you try that (provided that the file shell variable is defined correctly)?
# 6  
Old 12-19-2017
Quote:
Originally Posted by manid
I just want to know , is it possible to add column to the end of each record with the filename like below?


Code:
echo $row | grep -F "setting application" | grep -vF "@msad" | awk -v var="$file" '{print $0 var}'  >> test.txt

Code:
[123] [AGENT] [abc] [cde] [fgh] [asd] [asd] setting application ABC for user [user1@ldap1] file1.txt
[345] [AGENT] [abc] [cde] [fgh] [asd] [asd] setting application CDE for user [user2@ldap2] file1.txt

If you are reading a file line-by-line as suggested by MadeInGermany in post #2, please never feed those lines through a three element pipeline. Doing so is GROSSLY inefficient!

If you are processing multiple files (which would be a logical reason for adding the name of the file each record came from in the output you produce), you can easily modify MadeInGermany's shell suggestion from post #2 with something like:
Code:
for file in file*.txt
do	while IFS="" read row
	do	case $row in
		(*" setting application "*)
			case $row in
			(*"@msad"*)
				;;
			(*)	printf "%s %s\n" "$row" "$file";;
			esac;;
		esac
	done < "$file"
done > tempfile.txt

(which just uses shell built-ins and doesn't need to invoke any external utilities) or extend my awk suggestion from post #3 to something like:
Code:
awk '/ setting application / && ! /@msad/ { print $0, FILENAME }' file*.txt > tempfile.txt

(which just invokes awk once). Note the comma between $0 and FILENAME in the print statement; without it, there won't be any separation between the input lines and the name of the file from which it was extracted.

The code that you showed us in post #4 invokes two copies of grep and one copy of awk for every line read from each of the files you process.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Filtering records of a csv file based on a value of a column

Hi, I tried filtering the records in a csv file using "awk" command listed below. awk -F"~" '$4 ~ /Active/{print }' inputfile > outputfile The output always has all the entries. The same command worked for different users from one of the forum links. content of file I was... (3 Replies)
Discussion started by: sunilmudikonda
3 Replies

2. Shell Programming and Scripting

Separate records of a file on 2 types of records

Hi I am new to shell programming in unix Please if I can provide help. I have a file structure of a header record and "N" detail records. The header record will be the total number of detail records I need to split the file in 2: One for the header Another for all detail records Could... (1 Reply)
Discussion started by: jamcogar
1 Replies

3. Shell Programming and Scripting

Deleting duplicate records from file 1 if records from file 2 match

I have 2 files "File 1" is delimited by ";" and "File 2" is delimited by "|". File 1 below (3 record shown): Doc1;03/01/2012;New York;6 Main Street;Mr. Smith 1;Mr. Jones Doc2;03/01/2012;Syracuse;876 Broadway;John Davis;Barbara Lull Doc3;03/01/2012;Buffalo;779 Old Windy Road;Charles... (2 Replies)
Discussion started by: vestport
2 Replies

4. UNIX for Dummies Questions & Answers

Filtering records from 1 file based on some manipulation doen on second file

Hi, I am looking for an awk script which should help me to meet the following requirement: File1 has records in following format INF: FAILEd RECORD AB1234 INF: FAILEd RECORD PQ1145 INF: FAILEd RECORD AB3215 INF: FAILEd RECORD AB6114 ............................ (2 Replies)
Discussion started by: mintu41
2 Replies

5. UNIX for Dummies Questions & Answers

Grep specific records from a file of records that are separated by an empty line

Hi everyone. I am a newbie to Linux stuff. I have this kind of problem which couldn't solve alone. I have a text file with records separated by empty lines like this: ID: 20 Name: X Age: 19 ID: 21 Name: Z ID: 22 Email: xxx@yahoo.com Name: Y Age: 19 I want to grep records that... (4 Replies)
Discussion started by: Atrisa
4 Replies

6. Shell Programming and Scripting

filtering records based on numeric field value in 8th position

I have a ";" delimited file.Whcih conatins a number fileds of length 4 charcters in 8th position But there is a alphanumeric charcters like : space, ";" , "," , "/" , "23-1" , "23 1" , "aqjhdj" , "jun-23" , "APR-04" , "4:00AM" , "-234" , "56784 ", "." , "+" "_" , "&" , "*" , "^" , "%" , "!"... (2 Replies)
Discussion started by: indusri
2 Replies

7. Shell Programming and Scripting

Issues with filtering duplicate records using gawk script

Hi All, I have huge trade file with milions of trades.I need to remove duplicate records (e.g I have following records) 30/10/2009,trdeId1,..,.. 26/10/2009.tradeId1,..,..,, 30/10/2009,tradeId2,.. In the above case i need to filter duplicate recods and I should get following output.... (2 Replies)
Discussion started by: nmumbarkar
2 Replies

8. UNIX for Dummies Questions & Answers

Filtering records of a file based on a value of a column

Hi all, I would like to extract records of a file based on a condition. The file contains 47 fields, and I would like to extract only those records that match a certain value in one of the columns, e.g. COL1 COL2 COL3 ............... COL47 1 XX 45 ... (4 Replies)
Discussion started by: risk_sly
4 Replies

9. UNIX for Dummies Questions & Answers

Use records from one file to delete records in another file

file_in_1: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 file_in_2: 9 10 11 12 21 22 23 24 1 2 3 4 17 18 19 20 file_out: (5 Replies)
Discussion started by: kenneth.mcbride
5 Replies

10. Shell Programming and Scripting

Count No of Records in File without counting Header and Trailer Records

I have a flat file and need to count no of records in the file less the header and the trailer record. I would appreciate any and all asistance Thanks Hadi Lalani (2 Replies)
Discussion started by: guiguy
2 Replies
Login or Register to Ask a Question