Merge multiple lines into a single line


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Merge multiple lines into a single line
# 1  
Old 07-04-2017
Merge multiple lines into a single line

Hi all, I'm relatively new to scripting, I can do pretty basic things. I have a daily log file that looks like:

Code:
timestamp=2017-06-28-01.01.35.080576;
     event status=0;
     userid=user1;
   authid=user1;
     application id=10.10.10.10.11111.12345678901;
     application name=GUI;

  
timestamp=2017-06-28-01.01.36.096486;
     event status=0;
     userid=user1;
     authid=user1;
     application id=10.10.10.10.11111.12345678901;
     application name=GUI;
     statement text=SELECT table.field, table.field, table.field from database where table.field = value

There is a blank line between each log entry. I need to combine each log entry into one line and preferably remove the final semi-colon if there is one. I'm guessing that awk or sed could help. I've asked elsewhere, still no luck. Hopefully someone can help me out here.

Last edited by vbe; 07-04-2017 at 02:31 PM.. Reason: code tags...
# 2  
Old 07-04-2017
With sed (multi-liner with comments)
Code:
sed '
#loop
:L
# last line? Then jump to E (leave the loop)
$bE
# append next line to the line buffer with \n in between
N
# empty line? Not then jump to L
/\n *$/!bL
:E
# replace all empbedded \n and the surrounding space with one space
s/ *\n */ /g
# delete leading space
s/^ *//
# delete a trailing ; and following space
s/; *$//
# default print of line buffer
' logfile

Compare this practical exercise with the man sed! You might learn something.

Last edited by MadeInGermany; 07-04-2017 at 03:25 PM..
These 2 Users Gave Thanks to MadeInGermany For This Post:
# 3  
Old 07-04-2017
Try also
Code:
awk 'gsub (/;/, "\t") && $1 = $1' RS= file
timestamp=2017-06-28-01.01.35.080576	 event status=0	 userid=user1	 authid=user1	 application id=10.10.10.10.11111.12345678901	 application name=GUI	
timestamp=2017-06-28-01.01.36.096486	 event status=0	 userid=user1	 authid=user1	 application id=10.10.10.10.11111.12345678901	 application name=GUI	 statement text=SELECT table.field, table.field, table.field from database where table.field = value

This User Gave Thanks to RudiC For This Post:
# 4  
Old 07-04-2017
Hello MadeInGermany,

Nice code sir Smilie.


Hello dwdnet,

As you haven't shown the expected sample output, so based on your explanation of problem only, let me know if following helps you.
Code:
awk '{$1=$1;sub(/;$/,"");print}' RS=   Input_file

Hello Rudi sir,

Nice code, I think only thing could be improved is OP needs to remove the last occurrence of semi colon in each line.

Thanks,
R. Singh

Last edited by RavinderSingh13; 07-04-2017 at 04:05 PM..
These 2 Users Gave Thanks to RavinderSingh13 For This Post:
# 5  
Old 07-04-2017
Thanks for pointing this out; I misread it in post#1. Try
Code:
awk 'sub (/;$/, "")+1 && $1=$1' RS= file
timestamp=2017-06-28-01.01.35.080576; event status=0; userid=user1; authid=user1; application id=10.10.10.10.11111.12345678901; application name=GUI
timestamp=2017-06-28-01.01.36.096486; event status=0; userid=user1; authid=user1; application id=10.10.10.10.11111.12345678901; application name=GUI; statement text=SELECT table.field, table.field, table.field from database where table.field = value

This User Gave Thanks to RudiC For This Post:
# 6  
Old 07-04-2017
Thanks for the quick response guys. I'm not seeing the same results as you are. The only line that I am seeing is the very last 'statement text' line.

Code:
statement text=SELECT table.field, table.field, table.field from database where table.field = value

I'm running Debian Stretch. Could it be possible that the environment can cause this?

Moderator's Comments:
Mod Comment Please use CODE tags when displaying sample input, output, and code segments (as required by UNIX & Linux Forum rules).

Last edited by Don Cragun; 07-04-2017 at 06:53 PM..
# 7  
Old 07-04-2017
One might guess that the file you are processing has DOS line terminators (<carriage-return><newline>) instead of the expected UNIX line terminators (<newline>) and that the last line in your output is overwriting earlier lines in your output because of the embedded <carriage-return> characters.

You didn't mention which code suggestion you are using. Please try it again and pipe the output produced through od as shown below and show us the output (in CODE tags):
Code:
current command you are using | od -bc

If that shows some carriage returns in the output (displayed as \r) we can suggest ways to get around that problem if you tell us which code you are using.
This User Gave Thanks to Don Cragun For This Post:
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Merge multi-lines into one single line using shell script or Linux command

Hi, Can anyone help me for merge the following multi-line log which beginning with a " and line ending with ": into one line. *****Original Log***** 087;2008-12-06;084403;"mc;;SYHLR6AP1D\LNZW;AD-703;1;12475;SYHLR6AP1B;1.1.1.1;0000000062;HGPDI:MSISDN=12345678,APNID=1,EQOSID=365;... (3 Replies)
Discussion started by: rajeshlinux2010
3 Replies

2. UNIX for Dummies Questions & Answers

Need help combining txt files w/ multiple lines into csv single cell - also need data merge

:confused:Hello -- i just joined the forums. I am a complete noob -- only about 1 week into learning how to program anything... and starting with linux. I am working in Linux terminal. I have a folder with a bunch of txt files. Each file has several lines of html code. I want to combine... (2 Replies)
Discussion started by: jetsetter
2 Replies

3. Shell Programming and Scripting

Merge multiple lines in one line

Hi guys, So i have a input file with several sequences aligned (fasta) >NC_005930 241 bp MNMINIFIINNIFDQFIPVKLSIFSLTSVGSIIA LSWVWINTKTHWAISRSNTP-SLLLNSL WTLLITNL-NEKTNPWAPWLFSLFLLCFSFNIMSLI-PYTF-SQ TSHLSFTFGLSLPIWIMVNIAGFKNNWKKKISHLLPQGTPIYLVPVMII IETISLFIQPLTLGFRLGANLLAGHLLIFLCSCTIWE... (6 Replies)
Discussion started by: andreia
6 Replies

4. Shell Programming and Scripting

Merge multiple lines to one line when line starts with and ends with

example: comment Now_TB.table column errac is for error messages 1 - first 2 - second 3 -third ; in this example I need to be able to grab the comment as first word and ; as the last word and it might span a few lines. I need it to be put all in one line without line breaks so I can... (4 Replies)
Discussion started by: wambli
4 Replies

5. Shell Programming and Scripting

Multiple lines in a single column to be merged as a single line for a record

Hi, I have a requirement with, No~Dt~Notes 1~2011/08/1~"aaa bbb ccc ddd eee fff ggg hhh" Single column alone got splitted into multiple lines. I require the output as No~Dt~Notes 1~2011/08/1~"aaa<>bbb<>ccc<>ddd<>eee<>fff<>ggg<>hhh" mean to say those new lines to be... (1 Reply)
Discussion started by: Bhuvaneswari
1 Replies

6. UNIX for Advanced & Expert Users

Merge a group of lines into single line

Hi Everybody, Below are the contents of the a text file .., SN = 8 MSI = 405027002277133 IKVALUE = DE6AA6A11D42B69DF6398D44B17BC6F2 K4SNO = 2 CARDTYPE = SIM ALG = COMP128_3 SN = 8 MSI = 405027002546734 IKVALUE = 1D9F8BAA73973D8FBF8CBFB01436D822 K4SNO = 2 CARDTYPE = SIM ALG =... (8 Replies)
Discussion started by: prasanth_babu
8 Replies

7. Shell Programming and Scripting

merge lines into single line based on symbol \t

The symbols are \t and \t\t (note: not tab) If the line starts with \t merge them into a single line upto symbol \t\t \t\t to end and start new line I able to join in a single line but not ending at \t\t and I completely confused help would be appreciated:b::D Input \ta tab XXXXXXXXXX \te... (5 Replies)
Discussion started by: repinementer
5 Replies

8. Shell Programming and Scripting

Help on Merge multi-lines into one single line

Hello, Can anyone let me know how to use Perl script to Merge following multi-lines into one single line... ***** Multi-line***** FILE_Write root OK Tue Jul 01 00:00:00 2008 cl_get_path file descriptor = 1 FILE_Write root OK ... (5 Replies)
Discussion started by: happyday
5 Replies

9. Shell Programming and Scripting

Merge multi-lines into one single line

Hi, Can anyone help me for merge the following multi-line log which beginning with a number and time: into one line. For each line need to delete the return and add a space. Please see the red color line. *****Original Log*****... (4 Replies)
Discussion started by: happyday
4 Replies

10. Shell Programming and Scripting

Removing end of line to merge multiple lines

I'm sure this will be an easy question for you experts out there, but I have been searching the forum and working on this for a couple hours now and can't get it right. I have a very messy data file that I am trying to tidy up - one of the issues is some records are split into multiple lines: ... (4 Replies)
Discussion started by: tink
4 Replies
Login or Register to Ask a Question