Find lines in text file with certain data in first field


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Find lines in text file with certain data in first field
# 1  
Old 05-21-2009
Find lines in text file with certain data in first field

Hi all,

Sorry for the title, I was unsure how to word my issue. I'll get right to the issue. In my text file, I need to find all lines with the same data in the first field. Then I need to create a file with the matching lines merged into one. So my original file will look something like this:

a1 test1
a1 test2
b2 test1
b2 test2
a2 test1
a2 test2
b1 test1
b1 test2
...


And I want to output a new file to look like this:

a1 test1, test2
b2 test1, test2
a2 test1, test2
b1 test1, test2
...

Please help!!

Last edited by rstev39147; 05-21-2009 at 07:47 PM..
# 2  
Old 05-21-2009
if you have Python,
Code:
#!/usr/bin/env python
d={}
for line in open("file"):
    line = line.strip().split()
    d.setdefault(line[0],[])
    d[line[0]].append(line[-1])
for i,j in d.iteritems():
    print i,','.join(j)

output
Code:
# ./test.py
a1 test1,test2
a2 test1,test2
b1 test1,test2
b2 test1,test2

# 3  
Old 05-22-2009
Given your sample you may try something like this:
(use gawk, nawk or /usr/xpg4/bin/awk on Solaris)

Code:
awk 'END { print f, r[f] }
f && $1 != f { print f, r[f] } 
{ r[f = $1] = r[$1] ? r[$1] ", " $2 : $2 }
' infile

# 4  
Old 05-22-2009
Or, if order doesn't matter, here's a shorter version:

Code:
awk '{r[$1]=r[$1] ", " $2} END {for (f in r) print f, substr(r[f], 3)}' infile

If order does matter, you can always pipe the result through to 'sort'.
# 5  
Old 05-22-2009
Actually,
the array in my previous post is not needed:

Code:
awk 'END { print r } 
f && $1 != f { print r; r = x } 
{ r = r ? r ", " $2 : (f = $1) FS $2 }
' infile

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Modifying text file records, find data in one place in the record and print it elsewhere

Hello, I have some text data that is in the form of multi-line records. Each record ends with the string $$$$ and the next record starts on the next line. RDKit 2D 15 14 0 0 0 0 0 0 0 0999 V2000 5.4596 2.1267 0.0000 C 0 0 0 0 0 0 0 0 0 0 0 ... (5 Replies)
Discussion started by: LMHmedchem
5 Replies

2. Shell Programming and Scripting

awk to skip lines find text and add text based on number

I am trying to use awk skip each line with a ## or # and check each line after for STB= and if that value in greater than or = to 0.8, then at the end of line the text "STRAND BIAS" is written in else "GOOD". So in the file of 4 entries attached. awk tried: awk NR > "##"' "#" -F"STB="... (6 Replies)
Discussion started by: cmccabe
6 Replies

3. Shell Programming and Scripting

Text file to CSV with field data separated by blank lines

Hello, I have some data in a text file where fields are separated by blank lines. There are only 6 fields however some fields have several lines of data as I will explain. Also data in a particular field is not consistently the same size but does end on a blank line. The first field start with... (6 Replies)
Discussion started by: vestport
6 Replies

4. Shell Programming and Scripting

find string(s) in text file and nearby data, export to list help

Hi, So I'm kinda new to shell scripts and the like. I've picked up quite a bit of use from browsing the forums here but ran into a new one that I can't seem to find an answer for. I'm looking to parse/find a string AND the next 15 or so charachters that follow the string within a text file... (1 Reply)
Discussion started by: kar23me
1 Replies

5. Shell Programming and Scripting

Extracting specific lines of data from a file and related lines of data based on a grep value range?

Hi, I have one file, say file 1, that has data like below where 19900107 is the date, 19900107 12 144 129 0.7380047 19900108 12 168 129 0.3149017 19900109 12 192 129 3.2766666E-02 ... (3 Replies)
Discussion started by: Wynner
3 Replies

6. Shell Programming and Scripting

Find and replace data in text file with data in same file

OK I will do my best to explain what I need help with. I am trying to format an ldif file so I can import it into Oracle oid. I need the file to look like this example. Keep in mind there are 3000 of these in the file. changetype: modify replace: userpassword dn:... (0 Replies)
Discussion started by: timothyha22
0 Replies

7. Shell Programming and Scripting

Find pattern, and then last field from subsequent lines

I've got a log file, of the format Name: network1 Dropped packets: 15618 Dropped packets for IPv6: 27 Dropped packets: 74 Dropped packets for IPv6: 0 Failed RADIUS Authentication procedures: 0 Failed RADIUS Accounting procedures: 0 Name: network2 Dropped packets: 1117 ... (18 Replies)
Discussion started by: Yorkie99
18 Replies

8. Shell Programming and Scripting

Shell script to read lines in a text file and filter user data

hi all, I have this file with some user data. example: $cat myfile.txt FName|LName|Gender|Company|Branch|Bday|Salary|Age aaaa|bbbb|male|cccc|dddd|19900814|15000|20| eeee|asdg|male|gggg|ksgu|19911216||| aara|bdbm|male|kkkk|acke|19931018||23| asad|kfjg|male|kkkc|gkgg|19921213|14000|24|... (4 Replies)
Discussion started by: srimal
4 Replies

9. UNIX for Dummies Questions & Answers

find uniq lines in file, using the first field of line

Hello all, new to unix and have just found the forum. I think I will be here quite often, and hope that in time i will be able to provide soem help, role on not being a newbie anymore :) I have a question which iI am hoping someone could help me with. If i have a file with lines in in thus... (8 Replies)
Discussion started by: grom
8 Replies

10. Shell Programming and Scripting

shellscript to find a line in between a particular set of lines of a text file

i have a file a.txt and following is only one portion. I want to search <branch value="/dev36/AREA/" include="yes"></branch> present in between <template_file name="Approve External" path="core/approve/bin" and </template_file> where the no of lines containing "<branch value= " is increasing ... (2 Replies)
Discussion started by: millan
2 Replies
Login or Register to Ask a Question