How to pick a group of data using awk/ksh


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting How to pick a group of data using awk/ksh
# 1  
Old 04-20-2012
How to pick a group of data using awk/ksh

Hi gurus,
I have data coming in as shown below. And in each case, I need to pick the data in the last group as shown below.

Data Set 1:

Code:
DC | 18161621
LA | 15730880
NY | 16143237
DC | 18161621
LA | 17316397
NY | 17915905
DC | 18161621
LA | 17993534
NY | 18161621
DC | 18161621
LA | 17993534
NY | 18161621

in this above case (a group is defined as the bottom most unique rows), I need to pick the last group of data which is :
Code:
DC | 18161621
LA | 17993534
NY | 18161621


Data Set 2:

Code:
DC | 18161621
LA | 15730880
NY | 16143237
SF | 10000000
DC | 18161621
LA | 17316397
NY | 17915905
SF | 10000011
DC | 18161621
LA | 17993534
NY | 18161621
SF | 10000111
DC | 18161621
LA | 17993534
NY | 18161621
SF | 10009090

in this above case (a group is defined as the bottom most unique rows), I need to pick the last group of data which is :

Code:
DC | 18161621
LA | 17993534
NY | 18161621
SF | 10009090


So, the number of lines/records in a group can be just one or two or three or more but always I need to pick the last group and echo it to a file.

I was hoping to achieve it with an awk but no go so far. Any help into this is appreciated. Let me know if I am not clear.
Thanks,
Carl.

Last edited by Scrutinizer; 04-20-2012 at 02:45 PM.. Reason: code tags
# 2  
Old 04-20-2012
Hi calredd,

It can be an easy task or a bit more difficult. Can you identify each group? I mean, does the group always begins in DC or can vary?
# 3  
Old 04-20-2012
The names in the group can be anything, not necessarily start with a DC or NY. But the data will always come in groups :

FIRST_NUMBER|100
SECOND_NUMBER|199
FIRST_NUMBER|200
SECOND_NUMBER|299
FIRST_NUMBER|300
SECOND_NUMBER|399

In this above case, I need to pick the last group:

FIRST_NUMBER|300
SECOND_NUMBER|399

Let me know if you need more clarification.

---------- Post updated at 02:38 PM ---------- Previous update was at 02:31 PM ----------

FYI... one solution that I am trying to implement is to first determine number of unique lines/rows based on the first field only (as pipe is always a delimiter) :

For ex:
FIRST_NUMBER|100
SECOND_NUMBER|199
FIRST_NUMBER|200
SECOND_NUMBER|299
FIRST_NUMBER|300
SECOND_NUMBER|399

In the above case, that would be 2.

then just do a tail -2 of the above data would hopefully give me the last two lines but my solution isnt working yet.
# 4  
Old 04-20-2012
one way can be:
Code:
tail -n  $(sort -u -t"|" -k1,1 infile | wc -l) infile


Last edited by 47shailesh; 04-20-2012 at 04:58 PM.. Reason: added syntax for pipe delimited file
This User Gave Thanks to 47shailesh For This Post:
# 5  
Old 04-20-2012
Quote:
Originally Posted by 47shailesh
one way can be:
Code:
tail -n  $(sort -u -t"|" -k1,1 infile | wc -l) infile

I get your point but the syntax isnt working. How do I pass a negative number via variable to tail commad...

tail -2 infile works fine but
tail -${n} infile doesnt work assuming n=2
# 6  
Old 04-20-2012
Certainly it does, if n=2. If it doesn't work, I suspect you've got spaces or something as well as that digit inside n, messing up your statement.
# 7  
Old 04-20-2012
which part is not working. can you post error here.

tail -2 is equivalent to tail -n 2 so try tail -n ${n} infile and tail -${n} works for me
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

awk - Pick last value from set of rows

Input data COL_1,COL_2,COL_3,COL_4,COL_5,COL_6,COL_7,COL_8,COL_9,COL_10,COL_11,COL_12,COL_13 C,ABC,ABCD,3,ZZ,WLOA,2015-12-01,2016-12-01,975.73,ZZZ,P,111111.00,Y1 **GROUP1** C,ABC,ABCD,3,ZZ,WLOA,2015-12-01,2016-12-01,975.73,ZZZ,P,222222.00,Y1 **GROUP1**... (2 Replies)
Discussion started by: Ads89
2 Replies

2. Shell Programming and Scripting

Pick the column value including comma from csv file using awk

Source 1 column1 column2 column 3 column4 1,ganesh,1,000,1 222,ram,2,000,5 222,ram,50,000,5 33,raju,5,000,7 33,raju,5,000,7 33,raju,5,000,8 33,raju,5,000,4 33,raju,5,000,1 In my .csv file, third column is having price value with comma (20,300), it has to be considered 1,000 as... (1 Reply)
Discussion started by: Ganesh L
1 Replies

3. Shell Programming and Scripting

use awk pick value from lines as condition for grep

Hi Folks! I have a file like this 000000006 dist:0.0 FILE ./MintRoute/MultiHopWMEWMA.nc LINE:305:1 NODE_KIND:131 nVARs:4 NUM_NODE:66 TBID:733 TEID:758 000000000 dist:0.0 FILE ./Route/MultiHopLEPSM.nc LINE:266:1 NODE_KIND:131 nVARs:4 NUM_NODE:66 TBID:601 TEID:626 000000001 ... (2 Replies)
Discussion started by: jackoverflow
2 Replies

4. UNIX Desktop Questions & Answers

awk to pick out more than one line

This really is a dummy question but I'm stuck and out of time... I have a large file and out of it I only want to pick out lines starting with either "Pressure" or "N". I still need these lines to be in their original order. example of text Pressure 3 N 2 N 3 bla bla bla bla Pressure 4... (3 Replies)
Discussion started by: jenjen_mt
3 Replies

5. Shell Programming and Scripting

Use awk to pick out zip code

Hi, Suppose I have a csv file, each line look like this: ABC Company, 1999, March, caucasian owned, 123 BroadWay NY 92939-2222 How do I create two new columns at the end, one for state, one for zip. So that the line is ABC Company, 1999, March, caucasian owned, 123 BroadWay NY... (2 Replies)
Discussion started by: grossgermany
2 Replies

6. Shell Programming and Scripting

AWK or KSH : Sort, Group and extract from 3 files

Hi, I've the following two CSV files: File1.csv File2.csv Class,Student# Student#,Marks 1001,6001 6002,50 1001,6002 6001,60 1002,7000 ... (3 Replies)
Discussion started by: Matrix2682
3 Replies

7. Shell Programming and Scripting

Manipulating Pick multi dimensional data with awk.

Hi. I am reasonably new to awk, but have done quite a lot of unix scripting in the past. I have resolved the issues below with unix scripting but it runs like a dog. Moved to awk for speed and functionality but running up a big learning curve in a hurry, so hope there is some help here. I... (6 Replies)
Discussion started by: mike.strategis
6 Replies

8. Shell Programming and Scripting

awk help required to group output and print a part of group line and original line

Hi, Need awk help to group and print lines to format the output as shown below INPUT FORMAT set echo on set heading on set spool on /* SCHEMA1 */ CREATE TABLE T1; /* SCHEMA1 */ CREATE TABLE T2; /* SCHEMA1 */ CREATE TABLE T3; /* SCHEMA1 */ CREATE TABLE T4; /* SCHEMA1 */ CREATE TABLE T5;... (5 Replies)
Discussion started by: rajan_san
5 Replies

9. UNIX for Dummies Questions & Answers

converting a tabular format data to comma seperated data in KSH

Hi, Could anyone help me in changing a tabular format output to comma seperated file pls in K-sh. Its very urgent. E.g : username empid ------------------------ sri 123 to username,empid sri,123 Thanks, Hema:confused: (2 Replies)
Discussion started by: Hemamalini
2 Replies

10. Shell Programming and Scripting

awk/sed/ksh script to cleanup /etc/group file

Many of my servers' /etc/group file have many userid's that does not exist in /etc/passwd file and they need to be deleted. This happened due to manual manipulation of /etc/passwd files. I need to do this for 40 servers. Can anyone help me in achieving this? Even reducing a step or two will be... (6 Replies)
Discussion started by: pdtak
6 Replies
Login or Register to Ask a Question

Featured Tech Videos