Merge multiple lines into a single line


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Merge multiple lines into a single line
# 15  
Old 07-05-2017
Smilie Don, that definitely worked for all but a small portion of the log. That really helps. In a copy of the actual log, I ran the code you gave me and it did merge most lines.

I ran the od -bc command against it and I am seeing \n in some lines in the statement text= field. So when I view the log, some of the statement text= field content is still on their own line.

It sounds like the \n means new line. Would it be possible to remove the \n if it does not precede the timestamp= field if that is the case?

Last edited by dwdnet; 07-05-2017 at 06:20 PM.. Reason: Spelling mistake.
# 16  
Old 07-05-2017
You said originally that blank lines separate records. From what you have said in post #15 it sounds like you have some blank lines in the middle of some records. It isn't clear how you want the data represented by those blank input lines to appear in your output.

Please show us a couple of sample input records (in CODE tags) that do not work with the awk code suggested in post #12, show us the output that code is currently producing (in CODE tags), and show us the output that you want to produce from those input records (also in CODE tags).
This User Gave Thanks to Don Cragun For This Post:
# 17  
Old 07-10-2017
I'll have to create a mock up of a sample of the records as I can't put up live data. Hopefully that will suffice. Thanks.

---------- Post updated at 01:58 PM ---------- Previous update was at 11:29 AM ----------

I'm not sure this will work. Some of the statement text= lines are longer than 4096 characters. Quite complex database statements. I'll have to try and approach this from a different angle. I sincerely appreciate all the help though. Thanks, you guys are great.
# 18  
Old 07-10-2017
If the input is a text file, you don't care that the output might not be a text file (due to line length limitations), can clearly describe the rules for combining input lines into output lines, and you can provide representative (not actual) sample input and corresponding sample output, then the fact the some of the output lines are long should not be a problem.

Some versions of the awk utility can't write more than 4K bytes at a time, but all can write a single output line by writing several partial lines as long as each segment of the line being written is less that 4K bytes.

Are you saying that your input didn't have any blank lines in the middle of any records, but instead had some records that were more than 4K bytes long?
This User Gave Thanks to Don Cragun For This Post:
# 19  
Old 07-10-2017
Some of the records are longer than 4K. There are no blank lines in the long record however there are some line feeds, which is what is breaking the record into different lines. I am able to manually edit the text file and remove the LF though.
# 20  
Old 07-10-2017
You can, of course, edit the files manually. Or, you can show us representative sample input and the corresponding output you want from that input and we might then be able to help you modify the code shown in post #12 to get what you want programmatically.
# 21  
Old 07-11-2017
Manually editing would be very time consuming so I'll attempt to explain using the example below. Running your initial function helped quite a bit and re-formated the majority of records. There were some fields, within the record, (statement text=) that still were broken up with a LF.

Sample Input
Code:
timestamp=2017-06-28-01.01.35.080576; event status=0; userid=user1; authid=user1; application id=10.10.10.10.11111.12345678901; application name=GUI
timestamp=2017-06-28-01.01.36.096486; event status=0; userid=user1; authid=user1; application id=10.10.10.10.11111.12345678901; application name=GUI; statement text=SELECT table.field, table.field, table.field, table.field, table.field, table.field, table.field, table.field, table.field, table.field, table.field, table.field, table.field, table.field, table.field, from database where table.field = value
timestamp=2017-06-28-01.01.35.080576; event status=0; userid=user1; authid=user1; application id=10.10.10.10.11111.12345678901; application name=GUI
timestamp=2017-06-28-01.01.36.096486; event status=0; userid=user1; authid=user1; application id=10.10.10.10.11111.12345678901; application name=GUI; statement text=SELECT table.field, table.field, table.field from database where table.field = value
timestamp=2017-06-28-01.01.35.080576; event status=0; userid=user1; authid=user1; application id=10.10.10.10.11111.12345678901; application name=GUI
timestamp=2017-06-28-01.01.36.096486; event status=0; userid=user1; authid=user1; application id=10.10.10.10.11111.12345678901; application name=GUI; statement text=SELECT table.field, table.field, table.field, table.field, table.field, table.field, table.field, table.field, table.field, table.field, 
table.field, table.field, from database where table.field = value
timestamp=2017-06-28-01.01.35.080576; event status=0; userid=user1; authid=user1; application id=10.10.10.10.11111.12345678901; application name=GUI
timestamp=2017-06-28-01.01.36.096486; event status=0; userid=user1; authid=user1; application id=10.10.10.10.11111.12345678901; application name=GUI; statement text=SELECT table.field, table.field, table.field from database where table.field = value
timestamp=2017-06-28-01.01.35.080576; event status=0; userid=user1; authid=user1; application id=10.10.10.10.11111.12345678901; application name=GUI
timestamp=2017-06-28-01.01.36.096486; event status=0; userid=user1; authid=user1; application id=10.10.10.10.11111.12345678901; application name=GUI; statement text=SELECT table.field, table.field, table.field from database where table.field = value
timestamp=2017-06-28-01.01.35.080576; event status=0; userid=user1; authid=user1; application id=10.10.10.10.11111.12345678901; application name=GUI
timestamp=2017-06-28-01.01.36.096486; event status=0; userid=user1; authid=user1; application id=10.10.10.10.11111.12345678901; application name=GUI; statement text=SELECT table.field, table.field, table.field from database where table.field = value

Sample Output
Code:
timestamp=2017-06-28-01.01.35.080576; event status=0; userid=user1; authid=user1; application id=10.10.10.10.11111.12345678901; application name=GUI
timestamp=2017-06-28-01.01.36.096486; event status=0; userid=user1; authid=user1; application id=10.10.10.10.11111.12345678901; application name=GUI; statement text=SELECT table.field, table.field, table.field, table.field, table.field, table.field, table.field, table.field, table.field, table.field, table.field, table.field, table.field, table.field, table.field, from database where table.field = value
timestamp=2017-06-28-01.01.35.080576; event status=0; userid=user1; authid=user1; application id=10.10.10.10.11111.12345678901; application name=GUI
timestamp=2017-06-28-01.01.36.096486; event status=0; userid=user1; authid=user1; application id=10.10.10.10.11111.12345678901; application name=GUI; statement text=SELECT table.field, table.field, table.field from database where table.field = value
timestamp=2017-06-28-01.01.35.080576; event status=0; userid=user1; authid=user1; application id=10.10.10.10.11111.12345678901; application name=GUI
timestamp=2017-06-28-01.01.36.096486; event status=0; userid=user1; authid=user1; application id=10.10.10.10.11111.12345678901; application name=GUI; statement text=SELECT table.field, table.field, table.field, table.field, table.field, table.field, table.field, table.field, table.field, table.field, table.field, table.field, from database where table.field = value
timestamp=2017-06-28-01.01.35.080576; event status=0; userid=user1; authid=user1; application id=10.10.10.10.11111.12345678901; application name=GUI
timestamp=2017-06-28-01.01.36.096486; event status=0; userid=user1; authid=user1; application id=10.10.10.10.11111.12345678901; application name=GUI; statement text=SELECT table.field, table.field, table.field from database where table.field = value
timestamp=2017-06-28-01.01.35.080576; event status=0; userid=user1; authid=user1; application id=10.10.10.10.11111.12345678901; application name=GUI
timestamp=2017-06-28-01.01.36.096486; event status=0; userid=user1; authid=user1; application id=10.10.10.10.11111.12345678901; application name=GUI; statement text=SELECT table.field, table.field, table.field from database where table.field = value
timestamp=2017-06-28-01.01.35.080576; event status=0; userid=user1; authid=user1; application id=10.10.10.10.11111.12345678901; application name=GUI
timestamp=2017-06-28-01.01.36.096486; event status=0; userid=user1; authid=user1; application id=10.10.10.10.11111.12345678901; application name=GUI; statement text=SELECT table.field, table.field, table.field from database where table.field = value

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Merge multi-lines into one single line using shell script or Linux command

Hi, Can anyone help me for merge the following multi-line log which beginning with a " and line ending with ": into one line. *****Original Log***** 087;2008-12-06;084403;"mc;;SYHLR6AP1D\LNZW;AD-703;1;12475;SYHLR6AP1B;1.1.1.1;0000000062;HGPDI:MSISDN=12345678,APNID=1,EQOSID=365;... (3 Replies)
Discussion started by: rajeshlinux2010
3 Replies

2. UNIX for Dummies Questions & Answers

Need help combining txt files w/ multiple lines into csv single cell - also need data merge

:confused:Hello -- i just joined the forums. I am a complete noob -- only about 1 week into learning how to program anything... and starting with linux. I am working in Linux terminal. I have a folder with a bunch of txt files. Each file has several lines of html code. I want to combine... (2 Replies)
Discussion started by: jetsetter
2 Replies

3. Shell Programming and Scripting

Merge multiple lines in one line

Hi guys, So i have a input file with several sequences aligned (fasta) >NC_005930 241 bp MNMINIFIINNIFDQFIPVKLSIFSLTSVGSIIA LSWVWINTKTHWAISRSNTP-SLLLNSL WTLLITNL-NEKTNPWAPWLFSLFLLCFSFNIMSLI-PYTF-SQ TSHLSFTFGLSLPIWIMVNIAGFKNNWKKKISHLLPQGTPIYLVPVMII IETISLFIQPLTLGFRLGANLLAGHLLIFLCSCTIWE... (6 Replies)
Discussion started by: andreia
6 Replies

4. Shell Programming and Scripting

Merge multiple lines to one line when line starts with and ends with

example: comment Now_TB.table column errac is for error messages 1 - first 2 - second 3 -third ; in this example I need to be able to grab the comment as first word and ; as the last word and it might span a few lines. I need it to be put all in one line without line breaks so I can... (4 Replies)
Discussion started by: wambli
4 Replies

5. Shell Programming and Scripting

Multiple lines in a single column to be merged as a single line for a record

Hi, I have a requirement with, No~Dt~Notes 1~2011/08/1~"aaa bbb ccc ddd eee fff ggg hhh" Single column alone got splitted into multiple lines. I require the output as No~Dt~Notes 1~2011/08/1~"aaa<>bbb<>ccc<>ddd<>eee<>fff<>ggg<>hhh" mean to say those new lines to be... (1 Reply)
Discussion started by: Bhuvaneswari
1 Replies

6. UNIX for Advanced & Expert Users

Merge a group of lines into single line

Hi Everybody, Below are the contents of the a text file .., SN = 8 MSI = 405027002277133 IKVALUE = DE6AA6A11D42B69DF6398D44B17BC6F2 K4SNO = 2 CARDTYPE = SIM ALG = COMP128_3 SN = 8 MSI = 405027002546734 IKVALUE = 1D9F8BAA73973D8FBF8CBFB01436D822 K4SNO = 2 CARDTYPE = SIM ALG =... (8 Replies)
Discussion started by: prasanth_babu
8 Replies

7. Shell Programming and Scripting

merge lines into single line based on symbol \t

The symbols are \t and \t\t (note: not tab) If the line starts with \t merge them into a single line upto symbol \t\t \t\t to end and start new line I able to join in a single line but not ending at \t\t and I completely confused help would be appreciated:b::D Input \ta tab XXXXXXXXXX \te... (5 Replies)
Discussion started by: repinementer
5 Replies

8. Shell Programming and Scripting

Help on Merge multi-lines into one single line

Hello, Can anyone let me know how to use Perl script to Merge following multi-lines into one single line... ***** Multi-line***** FILE_Write root OK Tue Jul 01 00:00:00 2008 cl_get_path file descriptor = 1 FILE_Write root OK ... (5 Replies)
Discussion started by: happyday
5 Replies

9. Shell Programming and Scripting

Merge multi-lines into one single line

Hi, Can anyone help me for merge the following multi-line log which beginning with a number and time: into one line. For each line need to delete the return and add a space. Please see the red color line. *****Original Log*****... (4 Replies)
Discussion started by: happyday
4 Replies

10. Shell Programming and Scripting

Removing end of line to merge multiple lines

I'm sure this will be an easy question for you experts out there, but I have been searching the forum and working on this for a couple hours now and can't get it right. I have a very messy data file that I am trying to tidy up - one of the issues is some records are split into multiple lines: ... (4 Replies)
Discussion started by: tink
4 Replies
Login or Register to Ask a Question