Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Get output of multiple pattern match from first field to a file

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Get output of multiple pattern match from first field to a file
# 1  
Old 01-26-2017
Get output of multiple pattern match from first field to a file
Hi All,
Greetings!

I have a file of 40000+ lines with different entries, I need matching entries filterd out to their files based on first filed pattern for the matching :

For example:
All server1 entries (in field1) to come together with its path in 2nd field.

The best output I want to have it should generate filename for each:
Say server1.out file: which would be having "first field" and "second filed" of server1.
And so on for all the serverX .


Code:
datafile.txt  : 

server1	/usr/file1
server1 /usr/fileA
server2 /usr1/fileB
server2	/usr2/fileca
server3 /usr/DB/fileA
server3 /usr1/fileA
serverA /usr1/data1
server1 /usr3/data2
server2 /usr2/data2
server2 /path1/data2
serverA /pathb/data3

Code:
Desired output to be with each filename :


server1.out
server1	/usr/file1
server1 /usr/fileA
server1 /usr3/data2


Code:
File: server2.out 

server2 /usr1/fileB
server2 /usr2/filec
server2 /usr2/data2

Code:
File: server3.out 
server3 /usr/DB/fileA
server3 /usr1/fileA


Code:
File: serverA.out
serverA /usr1/data1
serverA /pathb/data3


Thanks ..
# 2  
Old 01-26-2017
Try:
Code:
sort -k1,1 -k2 datafile.txt | awk '$1!=s{s=$1; close(f); f=s ".out"} {print>f}'

Last edited by Scrutinizer; 01-26-2017 at 01:50 PM..
This User Gave Thanks to Scrutinizer For This Post:
rveri
# 3  
Old 01-26-2017
Code:
#!/bin/sh
pserver=""
while read server path
do
  if [ "$server" != "$pserver" ]; then
    exec 3>>"$server".out
    pserver=$server
  fi
  echo "$server $path" >&3
done < datafile.txt

If the input file is sorted, it can become more efficient
Code:
#!/bin/sh
pserver=""
sort -k1,1 datafile.txt |
while read server path
do
  if [ "$server" != "$pserver" ]; then
    exec 3>"$server".out
    pserver=$server
  fi
  echo "$server $path" >&3
done < datafile.txt

# 4  
Old 01-27-2017
Thank you both,
I checked the code from Scrutinizer , worked great ...
MadeInGermany also worked great ...

Both the code hard to understand .. but looks like same flow they are using , with Aws and shell..

Appreciate if you both give some explanation , how merging the lines based on first field if you get a chance ,..

Great code , thanks again .....
# 5  
Old 01-27-2017
My second sample
works the same as Scrutinizer's sample:
sort the file on the 1st field and pipe the result to awk or a while loop. The awk automatically loops over each input line, so the code only handles the per-line action.
Detailed description follows.
In shell the while loop reads line by line; the 1st field goes to $server variable, the rest to $path variable.
If $server is different from $pserver (true in line 1), it creates a new file "$server.out" with a descriptor 3, and $server is saved in $pserver. Then the "$server $path" is written to the descriptor 3.
If $server is unchanged (equal to $pserver) it writes further "$server $path" to the descriptor 3.
If $server is different from $pserver again, it creates a new file, again using the descriptor 3 that automatically closes the old file.
In contrast, the awk code needs an explicit close(), does not show the file descriptor, and creates the file automatically with the first write.
This User Gave Thanks to MadeInGermany For This Post:
rveri
# 6  
Old 01-31-2017
Thanks MadeInGermany,
It worked for a small line of code, when I executed against the 44000 lines , it did not do what it was intended to do, both the script awk and shell version, only had 1 line in the output files. not sure what happened.

Then I used shell array and grep to get the desired out put files , and working ,
Below is the code I made with grep ,

Code:
F=datafile.txt
cat $F | grep "^[a-z]" |awk '{print $1}' | sort | uniq -c| sort -rnk1 > hostlist
H=hostlist  #number of uniqe host filterd out in the file. 

#Stored each unique host names in array AA ..
typeset -i i=0 k=0 ; for j in `cat $H`; do  AA[$i]=$j ;  i=`expr $i + 1`  ; done

     ## AC=ArrayCount /Iteration needed
      AC=$(echo ${#AA[@]} ) 
...


    #AH :ArrayHost Name     #AHF: ArrayHostFile desired output filename.
    while [[ ${k} -le ${AC} ]]
    do
     
      AH=${AA[${k}]} ; AHF=${AH}.out
      grep ${AH} $F > ${AHF} ; ls -l ${AHF}    
      k=`expr $k + 1`
    done
...

It generated around 200 individual files from the big datafile . Again Many Thanks.. to both, It was kind of stuck when I started with this.., thanks..
Last edited by rveri; 01-31-2017 at 07:37 PM..
# 7  
Old 02-01-2017
Please get rid of the old style call of the expr program
Code:
i=`expr $i + 1`
k=`expr $i + 1`

And use the shell-builtins instead
Code:
i=$((i+1))
k=$((k+1))

or
Code:
((i+=1))
((k+=1))

And please avoid useless cat, and here you can even avoid grep
Code:
awk '/^[a-z]/ {print $1}' $F | sort ...

This User Gave Thanks to MadeInGermany For This Post:
rveri
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Modify text file if found multiple pattern match for every line.

Looking for help, i have input file like below and want to modify to expected output, if can without create additional file, hope can direct modify it. have 2 thing need do. 1st is adding a word (testplan generation off) after ! ! IPG: Tue Aug 07 14:31:17 2018 2nd is adding... (16 Replies)
Discussion started by: kttan
16 Replies

2. Shell Programming and Scripting

Removing multiple lines from input file, if multiple lines match a pattern.

GM, I have an issue at work, which requires a simple solution. But, after multiple attempts, I have not been able to hit on the code needed. I am assuming that sed, awk or even perl could do what I need. I have an application that adds extra blank page feeds, for multiple reports, when... (7 Replies)
Discussion started by: jxfish2
7 Replies

3. Shell Programming and Scripting

Command/script to match a field and print the next field of each line in a file.

Hello, I have a text file in the below format: Source Destination State Lag Status CQA02W2K12pl:D:\CAQA ... (10 Replies)
Discussion started by: pocodot
10 Replies

4. Shell Programming and Scripting

Remove multiple lines that match pattern

Not sure how I can accomplish this. I would like to remove all interfaces that have the commands I would like to see: switchport port-security, spanning-tree portfast. One line is no problem. interface FastEthernet0/8 spanning-tree portfast interface FastEthernet0/9 spanning-tree... (4 Replies)
Discussion started by: mrlayance
4 Replies

5. Shell Programming and Scripting

Match Pattern and print pattern and multiple lines into one line

Hello Experts , require help . See below output: File inputs ------------------------------------------ Server Host = mike id rl images allocated last updated density vimages expiration last read <------- STATUS ------->... (4 Replies)
Discussion started by: tigerhills
4 Replies

6. Shell Programming and Scripting

Multiple pattern match and print the output in a single line

I need to match two patterns in a log file and need to get the next line of the one of the pattern (out of two patterns) that is matched, finally need to print these three values in a single line. Sample Log: 2013/06/11 14:29:04 <0999> (725102) Processing batch 02_1231324 2013/06/11... (4 Replies)
Discussion started by: rpm120
4 Replies

7. UNIX for Dummies Questions & Answers

Match pattern in a field, print pattern only instead of the entire field

Hi ! I have a tab-delimited file, file.tab: Column1 Column2 Column3 aaaaaaaaaa bbtomatoesbbbbbb cccccccccc ddddddddd eeeeappleseeeeeeeee ffffffffffffff ggggggggg hhhhhhtomatoeshhh iiiiiiiiiiiiiiii ... (18 Replies)
Discussion started by: lucasvs
18 Replies

8. Shell Programming and Scripting

AWK: Pattern match between 2 files, then compare a field in file1 as > or < field in file2

First, thanks for the help in previous posts... couldn't have gotten where I am now without it! So here is what I have, I use AWK to match $1 and $2 as 1 string in file1 to $1 and $2 as 1 string in file2. Now I'm wondering if I can extend this AWK command to incorporate the following: If $1... (4 Replies)
Discussion started by: right_coaster
4 Replies

9. Shell Programming and Scripting

how do i pattern match a field with awk?

hi, let's say $numbers = "324 350 587" an so on... what i'm trying to do is this: awk -v numbers="$numbers" '{if (numbers ~ /$2/) print $0, "bla bla"}' file # file looks like this: 214 ..... 215 ... 216 .... 250 ... 324 325 ... 350 something ... ... 587 ... (4 Replies)
Discussion started by: someone123
4 Replies

10. UNIX for Dummies Questions & Answers

Output Multiple Field from dataBase file

I am fairly new in unix I was wondering if anybody can help me out with this: I am trying to output to a file the following fields; Field1 Field2 Field4 From a database file dataBase1. this is how the file looks: dataBase1 TABLE DATA Example ================== Table ... (3 Replies)
Discussion started by: Dennz
3 Replies
Login or Register to Ask a Question