Unix/Linux Go Back    


Shell Programming and Scripting BSD, Linux, and UNIX shell scripting — Post awk, bash, csh, ksh, perl, php, python, sed, sh, shell scripts, and other shell scripting languages questions here.

Merge multiple lines into a single line

Shell Programming and Scripting


Tags
awk, sed, solved

Reply    
 
Thread Tools Search this Thread Display Modes
    #1  
Old Unix and Linux 3 Weeks Ago
dwdnet dwdnet is offline
Registered User
 
Join Date: Jul 2017
Last Activity: 21 July 2017, 10:51 AM EDT
Posts: 12
Thanks: 17
Thanked 0 Times in 0 Posts
Merge multiple lines into a single line

Hi all, I'm relatively new to scripting, I can do pretty basic things. I have a daily log file that looks like:


Code:
timestamp=2017-06-28-01.01.35.080576;
     event status=0;
     userid=user1;
   authid=user1;
     application id=10.10.10.10.11111.12345678901;
     application name=GUI;

  
timestamp=2017-06-28-01.01.36.096486;
     event status=0;
     userid=user1;
     authid=user1;
     application id=10.10.10.10.11111.12345678901;
     application name=GUI;
     statement text=SELECT table.field, table.field, table.field from database where table.field = value

There is a blank line between each log entry. I need to combine each log entry into one line and preferably remove the final semi-colon if there is one. I'm guessing that awk or sed could help. I've asked elsewhere, still no luck. Hopefully someone can help me out here.

Last edited by vbe; 3 Weeks Ago at 01:31 PM.. Reason: code tags...
Sponsored Links
    #2  
Old Unix and Linux 3 Weeks Ago
MadeInGermany MadeInGermany is online now Forum Advisor  
Registered User
 
Join Date: May 2012
Last Activity: 27 July 2017, 2:26 PM EDT
Location: Simplicity
Posts: 3,642
Thanks: 285
Thanked 1,217 Times in 1,100 Posts
With sed (multi-liner with comments)

Code:
sed '
#loop
:L
# last line? Then jump to E (leave the loop)
$bE
# append next line to the line buffer with \n in between
N
# empty line? Not then jump to L
/\n *$/!bL
:E
# replace all empbedded \n and the surrounding space with one space
s/ *\n */ /g
# delete leading space
s/^ *//
# delete a trailing ; and following space
s/; *$//
# default print of line buffer
' logfile

Compare this practical exercise with the man sed! You might learn something.

Last edited by MadeInGermany; 3 Weeks Ago at 02:25 PM..
The Following 2 Users Say Thank You to MadeInGermany For This Useful Post:
dwdnet (3 Weeks Ago), RavinderSingh13 (3 Weeks Ago)
Sponsored Links
    #3  
Old Unix and Linux 3 Weeks Ago
RudiC RudiC is offline Forum Staff  
Moderator
 
Join Date: Jul 2012
Last Activity: 27 July 2017, 1:40 AM EDT
Location: Aachen, Germany
Posts: 11,066
Thanks: 284
Thanked 3,411 Times in 3,141 Posts
Try also

Code:
awk 'gsub (/;/, "\t") && $1 = $1' RS= file
timestamp=2017-06-28-01.01.35.080576	 event status=0	 userid=user1	 authid=user1	 application id=10.10.10.10.11111.12345678901	 application name=GUI	
timestamp=2017-06-28-01.01.36.096486	 event status=0	 userid=user1	 authid=user1	 application id=10.10.10.10.11111.12345678901	 application name=GUI	 statement text=SELECT table.field, table.field, table.field from database where table.field = value

The Following User Says Thank You to RudiC For This Useful Post:
dwdnet (3 Weeks Ago)
    #4  
Old Unix and Linux 3 Weeks Ago
RavinderSingh13 RavinderSingh13 is online now Forum Advisor  
Registered User
 
Join Date: May 2013
Last Activity: 27 July 2017, 2:54 PM EDT
Location: Chennai
Posts: 2,563
Thanks: 564
Thanked 1,209 Times in 1,088 Posts
Hello MadeInGermany,

Nice code sir Linux.


Hello dwdnet,

As you haven't shown the expected sample output, so based on your explanation of problem only, let me know if following helps you.

Code:
awk '{$1=$1;sub(/;$/,"");print}' RS=   Input_file

Hello Rudi sir,

Nice code, I think only thing could be improved is OP needs to remove the last occurrence of semi colon in each line.

Thanks,
R. Singh

Last edited by RavinderSingh13; 3 Weeks Ago at 03:05 PM..
The Following 2 Users Say Thank You to RavinderSingh13 For This Useful Post:
dwdnet (3 Weeks Ago), RudiC (3 Weeks Ago)
Sponsored Links
    #5  
Old Unix and Linux 3 Weeks Ago
RudiC RudiC is offline Forum Staff  
Moderator
 
Join Date: Jul 2012
Last Activity: 27 July 2017, 1:40 AM EDT
Location: Aachen, Germany
Posts: 11,066
Thanks: 284
Thanked 3,411 Times in 3,141 Posts
Thanks for pointing this out; I misread it in post#1. Try

Code:
awk 'sub (/;$/, "")+1 && $1=$1' RS= file
timestamp=2017-06-28-01.01.35.080576; event status=0; userid=user1; authid=user1; application id=10.10.10.10.11111.12345678901; application name=GUI
timestamp=2017-06-28-01.01.36.096486; event status=0; userid=user1; authid=user1; application id=10.10.10.10.11111.12345678901; application name=GUI; statement text=SELECT table.field, table.field, table.field from database where table.field = value

The Following User Says Thank You to RudiC For This Useful Post:
dwdnet (3 Weeks Ago)
Sponsored Links
    #6  
Old Unix and Linux 3 Weeks Ago
dwdnet dwdnet is offline
Registered User
 
Join Date: Jul 2017
Last Activity: 21 July 2017, 10:51 AM EDT
Posts: 12
Thanks: 17
Thanked 0 Times in 0 Posts
Thanks for the quick response guys. I'm not seeing the same results as you are. The only line that I am seeing is the very last 'statement text' line.


Code:
statement text=SELECT table.field, table.field, table.field from database where table.field = value

I'm running Debian Stretch. Could it be possible that the environment can cause this?

Moderator's Comments:
Merge multiple lines into a single line Please use CODE tags when displaying sample input, output, and code segments (as required by UNIX & Linux Forum rules).

Last edited by Don Cragun; 3 Weeks Ago at 05:53 PM..
Sponsored Links
    #7  
Old Unix and Linux 3 Weeks Ago
Don Cragun's Unix or Linux Image
Don Cragun Don Cragun is offline Forum Staff  
Administrator
 
Join Date: Jul 2012
Last Activity: 23 July 2017, 6:19 PM EDT
Location: San Jose, CA, USA
Posts: 10,412
Thanks: 527
Thanked 3,638 Times in 3,104 Posts
One might guess that the file you are processing has DOS line terminators (<carriage-return><newline>) instead of the expected UNIX line terminators (<newline>) and that the last line in your output is overwriting earlier lines in your output because of the embedded <carriage-return> characters.

You didn't mention which code suggestion you are using. Please try it again and pipe the output produced through od as shown below and show us the output (in CODE tags):

Code:
current command you are using | od -bc

If that shows some carriage returns in the output (displayed as \r) we can suggest ways to get around that problem if you tell us which code you are using.
The Following User Says Thank You to Don Cragun For This Useful Post:
dwdnet (3 Weeks Ago)
Sponsored Links
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Linux More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Need help combining txt files w/ multiple lines into csv single cell - also need data merge jetsetter UNIX for Dummies Questions & Answers 2 02-04-2013 09:12 AM
Merge a group of lines into single line prasanth_babu UNIX for Advanced & Expert Users 8 10-18-2010 01:01 PM
merge lines into single line based on symbol \t repinementer Shell Programming and Scripting 5 07-15-2009 09:16 AM
Help on Merge multi-lines into one single line happyday Shell Programming and Scripting 5 05-25-2009 07:30 AM
Merge multi-lines into one single line happyday Shell Programming and Scripting 4 05-09-2009 01:00 PM



All times are GMT -4. The time now is 02:55 PM.