Unix/Linux Go Back    


Shell Programming and Scripting BSD, Linux, and UNIX shell scripting — Post awk, bash, csh, ksh, perl, php, python, sed, sh, shell scripts, and other shell scripting languages questions here.

Merge multiple lines into a single line

Shell Programming and Scripting


Tags
awk, sed, solved

Reply    
 
Thread Tools Search this Thread Display Modes
    #1  
Old Unix and Linux 07-04-2017   -   Original Discussion by dwdnet
dwdnet dwdnet is offline
Registered User
 
Join Date: Jul 2017
Last Activity: 21 July 2017, 10:51 AM EDT
Posts: 12
Thanks: 17
Thanked 0 Times in 0 Posts
Merge multiple lines into a single line

Hi all, I'm relatively new to scripting, I can do pretty basic things. I have a daily log file that looks like:


Code:
timestamp=2017-06-28-01.01.35.080576;
     event status=0;
     userid=user1;
   authid=user1;
     application id=10.10.10.10.11111.12345678901;
     application name=GUI;

  
timestamp=2017-06-28-01.01.36.096486;
     event status=0;
     userid=user1;
     authid=user1;
     application id=10.10.10.10.11111.12345678901;
     application name=GUI;
     statement text=SELECT table.field, table.field, table.field from database where table.field = value

There is a blank line between each log entry. I need to combine each log entry into one line and preferably remove the final semi-colon if there is one. I'm guessing that awk or sed could help. I've asked elsewhere, still no luck. Hopefully someone can help me out here.

Last edited by vbe; 07-04-2017 at 02:31 PM.. Reason: code tags...
Sponsored Links
    #2  
Old Unix and Linux 07-04-2017   -   Original Discussion by dwdnet
MadeInGermany MadeInGermany is offline Forum Staff  
Moderator
 
Join Date: May 2012
Last Activity: 18 November 2017, 2:49 AM EST
Location: Simplicity
Posts: 3,822
Thanks: 318
Thanked 1,281 Times in 1,159 Posts
With sed (multi-liner with comments)

Code:
sed '
#loop
:L
# last line? Then jump to E (leave the loop)
$bE
# append next line to the line buffer with \n in between
N
# empty line? Not then jump to L
/\n *$/!bL
:E
# replace all empbedded \n and the surrounding space with one space
s/ *\n */ /g
# delete leading space
s/^ *//
# delete a trailing ; and following space
s/; *$//
# default print of line buffer
' logfile

Compare this practical exercise with the man sed! You might learn something.

Last edited by MadeInGermany; 07-04-2017 at 03:25 PM..
The Following 2 Users Say Thank You to MadeInGermany For This Useful Post:
dwdnet (07-05-2017), RavinderSingh13 (07-04-2017)
Sponsored Links
    #3  
Old Unix and Linux 07-04-2017   -   Original Discussion by dwdnet
RudiC RudiC is offline Forum Staff  
Moderator
 
Join Date: Jul 2012
Last Activity: 17 November 2017, 6:42 AM EST
Location: Aachen, Germany
Posts: 11,616
Thanks: 320
Thanked 3,615 Times in 3,320 Posts
Try also

Code:
awk 'gsub (/;/, "\t") && $1 = $1' RS= file
timestamp=2017-06-28-01.01.35.080576	 event status=0	 userid=user1	 authid=user1	 application id=10.10.10.10.11111.12345678901	 application name=GUI	
timestamp=2017-06-28-01.01.36.096486	 event status=0	 userid=user1	 authid=user1	 application id=10.10.10.10.11111.12345678901	 application name=GUI	 statement text=SELECT table.field, table.field, table.field from database where table.field = value

The Following User Says Thank You to RudiC For This Useful Post:
dwdnet (07-05-2017)
    #4  
Old Unix and Linux 07-04-2017   -   Original Discussion by dwdnet
RavinderSingh13 RavinderSingh13 is offline Forum Advisor  
Registered User
 
Join Date: May 2013
Last Activity: 18 November 2017, 2:48 AM EST
Location: Chennai
Posts: 2,670
Thanks: 588
Thanked 1,272 Times in 1,145 Posts
Hello MadeInGermany,

Nice code sir Linux.


Hello dwdnet,

As you haven't shown the expected sample output, so based on your explanation of problem only, let me know if following helps you.

Code:
awk '{$1=$1;sub(/;$/,"");print}' RS=   Input_file

Hello Rudi sir,

Nice code, I think only thing could be improved is OP needs to remove the last occurrence of semi colon in each line.

Thanks,
R. Singh

Last edited by RavinderSingh13; 07-04-2017 at 04:05 PM..
The Following 2 Users Say Thank You to RavinderSingh13 For This Useful Post:
dwdnet (07-05-2017), RudiC (07-04-2017)
Sponsored Links
    #5  
Old Unix and Linux 07-04-2017   -   Original Discussion by dwdnet
RudiC RudiC is offline Forum Staff  
Moderator
 
Join Date: Jul 2012
Last Activity: 17 November 2017, 6:42 AM EST
Location: Aachen, Germany
Posts: 11,616
Thanks: 320
Thanked 3,615 Times in 3,320 Posts
Thanks for pointing this out; I misread it in post#1. Try

Code:
awk 'sub (/;$/, "")+1 && $1=$1' RS= file
timestamp=2017-06-28-01.01.35.080576; event status=0; userid=user1; authid=user1; application id=10.10.10.10.11111.12345678901; application name=GUI
timestamp=2017-06-28-01.01.36.096486; event status=0; userid=user1; authid=user1; application id=10.10.10.10.11111.12345678901; application name=GUI; statement text=SELECT table.field, table.field, table.field from database where table.field = value

The Following User Says Thank You to RudiC For This Useful Post:
dwdnet (07-05-2017)
Sponsored Links
    #6  
Old Unix and Linux 07-04-2017   -   Original Discussion by dwdnet
dwdnet dwdnet is offline
Registered User
 
Join Date: Jul 2017
Last Activity: 21 July 2017, 10:51 AM EDT
Posts: 12
Thanks: 17
Thanked 0 Times in 0 Posts
Thanks for the quick response guys. I'm not seeing the same results as you are. The only line that I am seeing is the very last 'statement text' line.


Code:
statement text=SELECT table.field, table.field, table.field from database where table.field = value

I'm running Debian Stretch. Could it be possible that the environment can cause this?

Moderator's Comments:
Merge multiple lines into a single line Please use CODE tags when displaying sample input, output, and code segments (as required by UNIX & Linux Forum rules).

Last edited by Don Cragun; 07-04-2017 at 06:53 PM..
Sponsored Links
    #7  
Old Unix and Linux 07-04-2017   -   Original Discussion by dwdnet
Don Cragun's Unix or Linux Image
Don Cragun Don Cragun is online now Forum Staff  
Administrator
 
Join Date: Jul 2012
Last Activity: 18 November 2017, 3:01 AM EST
Location: San Jose, CA, USA
Posts: 10,651
Thanks: 571
Thanked 3,728 Times in 3,182 Posts
One might guess that the file you are processing has DOS line terminators (<carriage-return><newline>) instead of the expected UNIX line terminators (<newline>) and that the last line in your output is overwriting earlier lines in your output because of the embedded <carriage-return> characters.

You didn't mention which code suggestion you are using. Please try it again and pipe the output produced through od as shown below and show us the output (in CODE tags):

Code:
current command you are using | od -bc

If that shows some carriage returns in the output (displayed as \r) we can suggest ways to get around that problem if you tell us which code you are using.
The Following User Says Thank You to Don Cragun For This Useful Post:
dwdnet (07-05-2017)
Sponsored Links
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Linux More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Need help combining txt files w/ multiple lines into csv single cell - also need data merge jetsetter UNIX for Dummies Questions & Answers 2 02-04-2013 10:12 AM
Merge a group of lines into single line prasanth_babu UNIX for Advanced & Expert Users 8 10-18-2010 02:01 PM
merge lines into single line based on symbol \t repinementer Shell Programming and Scripting 5 07-15-2009 10:16 AM
Help on Merge multi-lines into one single line happyday Shell Programming and Scripting 5 05-25-2009 08:30 AM
Merge multi-lines into one single line happyday Shell Programming and Scripting 4 05-09-2009 02:00 PM



All times are GMT -4. The time now is 04:24 AM.