Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Merge multiple lines in same file with common key using awk
Top Forums Shell Programming and Scripting Merge multiple lines in same file with common key using awk Post 302735711 by protosd on Monday 26th of November 2012 03:55:42 AM
Old 11-26-2012
AMAZING!

Thank you, that works!

I wish I could learn AWK, I know it's really powerful. I've coded in many different languages and can usually work out what stuff does, but AWK is so cryptic!

I'm looking at what you posted trying to understand. If you're up giving me a little tutorial, I'd love to understand.

Where does p1 come from?

Is prev a built-in variable?

Code:
# While NR (number of records) is not equal to 1 and p1 is not equal to the first field (the ID)
NR!=1 && p1!=$1

# Print the line and append a null so the line will join with the next
{print prev;prev=""}

# p1 gets the value of the ID field
{p1=$1;

# This is a bit cryptic, but it looks like you're assigning/merging the contents
# of previous line with the next one and stripping the first field??
prev=(prev"")?prev FS substr($0,index($0,$2)):$0}

# Also a bit cryptic, but guessing we've reached the EOF and spitting out what's left?
END{if(prev"") print prev}'

---------- Post updated at 12:52 AM ---------- Previous update was at 12:48 AM ----------

---------- Post updated at 12:55 AM ---------- Previous update was at 12:52 AM ----------

Quote:
Originally Posted by pamu
Try

Code:
awk -F "," 's != $1 || NR ==1{s=$1;if(p){print p};p=$0;next}
{sub($1,"",$0);p=p""$0;}END{print p}' file

That works well also!

Thanks!


The great thing about Unix and AWK is there are so many different ways to do stuff.

The thing about AWK is it's hard to tell what special meaning certain things have and where they come from.

I wish I'd learned it a long time ago.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

merge multiple lines from flat file

Hi, I have a tab delimited flat file like this: 189 Guide de lutilisateur sur lappel conférence à trois au moyen d'adaptateurs téléphoniques <TABLE><TBODY><TR><TD><DIV class=subheader>La fonction Appel conférence à trois </DIV></TD> \ <TD><?php print $navTree;?> vous permet de tenir un appel... (4 Replies)
Discussion started by: hnhegde
4 Replies

2. Shell Programming and Scripting

merge based on common, awk help

All, $ cat x.txt z 11 az x 12 ax y 13 ay $ cat y.txt ay TT ax NN Output required: y 13 ay TT x 12 ax NN (3 Replies)
Discussion started by: jkl_jkl
3 Replies

3. Shell Programming and Scripting

Merge lines in a file with Awk - incorrect output

Hi, I would like: FastEthernet0/0 is up, line protocol is up 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored 0 output errors, 0 collisions, 0 interface resets Serial1/0:0 is up, line protocol is up 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 0... (14 Replies)
Discussion started by: mv652
14 Replies

4. Shell Programming and Scripting

Using Perl to Merge Multiple Lines in a File

I've hunted and hunted but nothing seems to apply to what I need. Any help will be much appreciated! My input file looks like (Unix): marker,allele1,allele2 RS1002244,1,1 RS1002244,1,3 RS1002244,3,3 RS1003719,2,2 RS1003719,2,4 RS1003719,4,4 Most markers are listed 3 times but a few... (2 Replies)
Discussion started by: Peggy White
2 Replies

5. Shell Programming and Scripting

"Join" or "Merge" more than 2 files into single output based on common key (column)

Hi All, I have working (Perl) code to combine 2 input files into a single output file using the join function that works to a point, but has the following limitations: 1. I am restrained to 2 input files only. 2. Only the "matched" fields are written out to the "matched" output file and... (1 Reply)
Discussion started by: Katabatic
1 Replies

6. Shell Programming and Scripting

Gawk / Awk Merge Lines based on Key

Hi Guys, After windows died on my netbook I installed Lubuntu and discovered Gawk about a month ago. After using Excel for 10+ years I'm amazed how quick and easily Gawk can process data but I'm stuck with a little problem merging data from multiple lines. I'm an SEO Consultant and provide... (9 Replies)
Discussion started by: Jamesfirst
9 Replies

7. UNIX for Dummies Questions & Answers

Writing a loop to merge multiple files by common column

I have 100 data files labelled 250.1.txt through 250.100.txt. The second column of the data files partially match (there is about %90 overlap). Each data file has 4 columns. I want the merge all these text files by the matching values in the second column. In the output, the first column should... (1 Reply)
Discussion started by: evelibertine
1 Replies

8. Shell Programming and Scripting

Merge multiple files with common header

Hi all, Say i have multiple files x1 x2 x3 x4, all with common header (date, time, year, age),, How can I merge them to one singe file "X" in shell scripting Thanks for your suggestions. (2 Replies)
Discussion started by: msarguru
2 Replies

9. UNIX for Dummies Questions & Answers

Merge selective columns from files based on common key

Hi, I am trying to selectively merge two files based on keys reported in the 1st column. File1: #file1-header1 file1-header2 111 qwe rtz uio 198 asd fgh jkl 165 yxc 789 poi uzt rew 89 lkj File2: #file2-header2 file2-header2 165 ghz nko2 ... (2 Replies)
Discussion started by: dovah
2 Replies

10. Shell Programming and Scripting

Join and merge multiple files with duplicate key and fill void columns

Join and merge multiple files with duplicate key and fill void columns Hi guys, I have many files that I want to merge: file1.csv: 1|abc 1|def 2|ghi 2|jkl 3|mno 3|pqr file2.csv: (5 Replies)
Discussion started by: yjacknewton
5 Replies

LEARN ABOUT CENTOS

prev

PREV(1) 							     [nmh-1.5]								   PREV(1)

NAME

       prev - show the previous message

SYNOPSIS

       prev [+folder] [-showproc program] [-showmimeproc program] [-header | -noheader] [-checkmime | -nocheckmime] [switches for showproc or
	    showmimeproc] [-version] [-help]

DESCRIPTION

       Prev performs a show on the previous message in the specified (or current) folder.  Like show, it passes any switches  on  to  the  program
       named by showproc or showmimeproc, which is called to list the message.	This command is almost exactly equivalent to "show prev".  Consult
       the manual entry for show(1) for all the details.

FILES

       $HOME/.mh_profile	  The user profile

PROFILE COMPONENTS

       Path:		    To determine the user's nmh directory
       Current-Folder:	    To find the default current folder
       showproc:	    Program to show non-MIME messages
       showmimeproc:	    Program to show MIME messages

SEE ALSO

       show(1), next(1)

DEFAULTS

       `+folder' defaults to the current folder
       `-checkmime'
       `-header'

CONTEXT

       If a folder is specified, it will become the current folder.  The message that is shown (i.e.,  the  previous  message  in  sequence)  will
       become the current message.

BUGS

       prev  is  really  a link to the show program.  As a result, if you make a link to prev and that link is not called prev, your link will act
       like show instead.  To circumvent this, add a profile-entry for the link to your nmh profile and add the argument prev to the entry.

MH.6.8								   11 June 2012 							   PREV(1)
All times are GMT -4. The time now is 01:34 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy