Login or Register to Ask a Question and Join Our Community


UNIX for Dummies Questions & Answers

Concatenate three lines into one

 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Concatenate three lines into one
# 1  
Old 08-13-2014
Concatenate three lines into one
Hi. I'm new to this forum.

I am attempting to parse an Audit Log from Cognos/TM1, selecting only Event IDs of "client" which are found on the "start-tag" record. These Logs are in a pseudo-XML format but not a true XML format. I want to FTP an Audit Log File from the Cognos server to our UNIX server. Then I want to look for "client" at position 28 in a given line, and upon finding that, write the current line and the next two lines after it as one output line. (We have Shell Scripts already that FTP data from Windows to UNIX, so I don't need assistance with that.)

These lines have CRLF (I think) after each one. When I view them in Windows-based editors (like TextPad), they are separate lines.

I'm a Unix script newbie. I've searched the Web for possible solutions but have not found any. I have found examples of merging two lines by patterns using sed and one using sed (I think) and a line range, but the sample had a hard-coded line range.

We run the AIX Flavor of UNIX at my company. We use KSH as the Shell Script most of the time.

Pseudo-code:

IF current-line-number line has "client" at position 28, write current-line-number concatenate current-line-number plus 1 concatenate current-line-number plus 2 FI

Thank you for your help.
Last edited by FredAtArrow; 08-13-2014 at 02:27 PM..
# 2  
Old 08-13-2014
Welcome to the UNIX and Linux Forums.
Thank you for telling us what OS and shell you're using. That always makes it easier for the volunteers here who try to help answer your questions suggest solutions that will work in your environment.

Please provide a short sample input file and a corresponding output file showing what you want to be produced by your script for that sample input. (Please use CODE tags when showing us sample input, sample output, and code fragments.)

- Don
# 3  
Old 08-13-2014
Thank you for your reply, Mr. Cragun.

Input file sample (I put in "(crlf)" to show where crlf's are):

Code:
<Commit ts="20140812153817" client="AAAAAA">(crlf)
"144","AAAAAA","00.00.000.000","User 'AAAAAA' successfully logged in from address '00.00.000.000'."crlf
</Commit>crlf
<Commit ts="20140812160347" client="BBBBBB">(crlf)
"144","BBBBBB","00.0.000.00","User 'BBBBBB' successfully logged in from address '00.0.000.00'."(crlf)
</Commit>(crlf)
<Commit ts="20140812170838" client="CCCCCC">(crlf)
"144","CCCCCC","00.0.000.00","User 'CCCCCC' successfully logged in from address '00.0.000.00'."(crlf)
</Commit>(crlf)

Output file sample (again I put "(crlf)" to show where crlf's should be):

Code:
<Commit ts="20140812153817" client="AAAAAA">"144","AAAAAA","00.00.000.000","User 'AAAAAA' successfully logged in from address '00.00.000.000'."</Commit>(crlf)
<Commit ts="20140812160347" client="BBBBBB">"144","BBBBBB","00.0.000.00","User 'BBBBBB' successfully logged in from address '00.0.000.00'."</Commit>(crlf)
<Commit ts="20140812170838" client="CCCCCC">"144","CCCCCC","00.0.000.00","User 'CCCCCC' successfully logged in from address '00.0.000.00'."</Commit>(crlf)

Once in a single-record-per-event format, I can parse it using our ETL Tools.
# 4  
Old 08-13-2014
Code:
awk '{ gsub(/[\r\n]/, ""); L=L$0 } /<\/Commit>/ { print L; L="" }' inputfile > outputfile

This User Gave Thanks to Corona688 For This Post:
FredAtArrow
# 5  
Old 08-13-2014
I think Corona688 missed the fact that you want CRLF line terminators in your output too. To do that, this trivial modification to his code:
Code:
awk '{ gsub(/[\r\n]/, ""); L=L$0 } /<\/Commit>/ { print L "\r"; L="" }' inputfile > outputfile

should do what you want as long as the 1st line in every set of 3 adjacent lines in your input file contains "client" starting in column 29 (columns are numbered starting from 1; not 0; in awk. If you want to discard other types of records, this more complicated script should work:
Code:
awk '
cnt {	out = out $0
	if(--cnt == 0) {
		gsub("\r", "", out)
		print out "\r"
	}
	next
}
substr($0, 29, 6) == "client" {
	cnt = 2
	out = $0
}' inputfile > outuptfile

These 2 Users Gave Thanks to Don Cragun For This Post:
Corona688 FredAtArrow
# 6  
Old 08-14-2014
Posted by DON:

Quote:
I think Corona688 missed the fact that you want CRLF line terminators in your output too. To do that, this trivial modification to his code:


Code:
awk '{ gsub(/[\r\n]/, ""); L=L$0 } /<\/Commit>/ { print L "\r"; L="" }' inputfile > outputfile

should do what you want as long as the 1st line in every set of 3 adjacent lines in your input file contains "client" starting in column 29 (columns are numbered starting from 1; not 0; in awk . If you want to discard other types of records, this more complicated script should work:


Code:
awk 'cnt { out = out $0 if(--cnt == 0) { gsub("\r", "", out) print out "\r" } next}substr($0, 29, 6) == "client" { cnt = 2 out = $0}' inputfile > outuptfile
Hello Don,

As per user's request I can see user wants crlf at last only.

Code:
awk '/^<Commit ts/ {gsub(/\(crlf\)/,X,$0);a=$0;getline;gsub(/\(crlf\)/,X,$0);gsub(/crlf/,X,$0);a=a $0;getline;a=a $0; print a}'  filename

Output will be as follows.

Code:
<Commit ts="20140812153817" client="AAAAAA">"144","AAAAAA","00.00.000.000","User 'AAAAAA' successfully logged in from address '00.00.000.000'."</Commit>crlf
<Commit ts="20140812160347" client="BBBBBB">"144","BBBBBB","00.0.000.00","User 'BBBBBB' successfully logged in from address '00.0.000.00'."</Commit>(crlf)
<Commit ts="20140812170838" client="CCCCCC">"144","CCCCCC","00.0.000.00","User 'CCCCCC' successfully logged in from address '00.0.000.00'."</Commit>(crlf)


Thanks,
R. Singh
This User Gave Thanks to RavinderSingh13 For This Post:
FredAtArrow
# 7  
Old 08-14-2014
Quote:
Originally Posted by RavinderSingh13
Posted by DON:



Hello Don,

As per user's request I can see user wants crlf at last only.

Code:
awk '/^<Commit ts/ {gsub(/\(crlf\)/,X,$0);a=$0;getline;gsub(/\(crlf\)/,X,$0);gsub(/crlf/,X,$0);a=a $0;getline;a=a $0; print a}'  filename

Output will be as follows.

Code:
<Commit ts="20140812153817" client="AAAAAA">"144","AAAAAA","00.00.000.000","User 'AAAAAA' successfully logged in from address '00.00.000.000'."</Commit>crlf
<Commit ts="20140812160347" client="BBBBBB">"144","BBBBBB","00.0.000.00","User 'BBBBBB' successfully logged in from address '00.0.000.00'."</Commit>(crlf)
<Commit ts="20140812170838" client="CCCCCC">"144","CCCCCC","00.0.000.00","User 'CCCCCC' successfully logged in from address '00.0.000.00'."</Commit>(crlf)


Thanks,
R. Singh
Hi Ravinder,
The (crlf) (and in two places) crlf (without the parentheses) is FredAtArrow's notation indicating that the input and output lines have Windows style <carriage-return><line-feed> (where <line-feed> is a synonym for <newline> on Linux/UNIX systems) line terminators instead of just the Linux/UNIX style <newline> line terminators. The:
Code:
gsub(/[\r\n]/, "")

that processed every input line in Corona688's proposal is overkill removing both <carriage-return>s and <newline>s (I say overkill, because awk already strips out the <newline> characters before it hands us a line as long as we keep the default value for the awk RS variable).

The code I suggested strips out the carriage returns (the awk escape sequence \r in a string) from the accumulated set of three input lines just before adding back the desired <carriage-return> at the end of the line when printing the results:
Code:
		gsub("\r", "", out)
		print out "\r"

and, as we both know, as long as we are using the default awk ORS variable, the print command adds a <newline> when it prints a string.
These 2 Users Gave Thanks to Don Cragun For This Post:
FredAtArrow RavinderSingh13
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to concatenate lines with specific pattern?

How to concatenate lines with specific pattern? I have data dumped from a table into text file. In some occurrence the data row is split into two rows. Example: 12345678|Global Test|Global Test Task|My Request|Date|Date|Date|1|1| 12345679|Global Test2|Global Test Task2|My... (8 Replies)
Discussion started by: nixtime
8 Replies

2. Shell Programming and Scripting

Need to concatenate spuriously separated lines

Given the pattern below: 3113296571|NULL|NULL|NULL||N| 1| 0| 926667| 1001036| 0| 3076120438|NULL|NULL|NULL|NULL|DUE FOR NEW CONSENT!|N|NULL| 10198318|2011-07-25-12.34.02.786000|NULL|NULL|NULL| 0 3113336478|NULL|NULL|NULL||N| 1| ... (16 Replies)
Discussion started by: lemele
16 Replies

3. Programming

Concatenate two lines in a fIle

Hi All, Can any one help me in finding the solution for concatenating two or more lines in a file and writing them to a temporary file. for Example: He is a wise student. So he got first rank. This is in a file i want the output as He is a wise student so he got first rank. into a file... (3 Replies)
Discussion started by: uday.sena.m
3 Replies

4. Shell Programming and Scripting

Concatenate lines between lines starting with a specific pattern

Hi, I have a file such as: --- >contig00001 length=35524 numreads=2944 gACGCCGCGCGCCGCGGCCAGGGCTGGCCCA CAGGCCGCGCGGCGTCGGCTGGCTGAG >contig00002 length=4242 numreads=43423 ATGCCGAAGGTCCGCCTGGGGCTGG CGCCGGGAGCATGTAGCG --- I would like to concatenate the lines not starting with ">"... (9 Replies)
Discussion started by: s052866
9 Replies

5. Shell Programming and Scripting

Concatenate multiple lines based.

Hello, I have been searching the forum for concatenation based on condition. I have been close enough but not got th exact one. infile: -----DB_Name ABC (X, Y,Z). DB_Name DEF (T). DB_Name GHI (U ,V,W). Desired Output file should be: ---------------------------DB_Name ABC... (8 Replies)
Discussion started by: indrajit_u
8 Replies

6. Shell Programming and Scripting

concatenate lines in pairs

Hi, I have a text file with the following contents /C=IT/O=INFN/OU=Personal Certificate/L=Napoli/CN=Some guy /C=IT/O=INFN/CN=INFN CA /O=Grid/O=NorduGrid/OU=uninett.no/CN=Another guy /O=Grid/O=NorduGrid/CN=NorduGrid Certification Authority /C=TW/O=AP/OU=GRID/CN=Someone else... (5 Replies)
Discussion started by: kerl
5 Replies

7. Shell Programming and Scripting

Concatenate two lines in one

Does anyone know a way I can fix my file below where it has lines that are cut? Just like the (notice the space before ms,c=PH).. dn: cn=english,ou=Messaging,ou=lang,o=subject,t=j ms,c=PH which should be dn: cn=english,ou=Messaging,ou=lang,o=subject,t=jms,c=PH The whole text file... (5 Replies)
Discussion started by: Orbix
5 Replies

8. Shell Programming and Scripting

concatenate lines using shell scripting

i have a mega file in this format: a, b, c, d, = a2, b2, c2, d2, = a3, b3 = i want to combine lines until = meet. the result should be : a,b,c,d, a2,b2,c3,d2 a3,b3 (13 Replies)
Discussion started by: dtdt
13 Replies

9. Shell Programming and Scripting

Concatenate lines in a single paragraph

Hi all, May you help me with this? After extract only some lines that I need from a file, I would like to concatenate the remaining lines in a single line (paragraph). Source file pattern: Event Type: Warning Event Source: Sorting Event Category: None Event ID: 1188 Date: 9/10/2008... (4 Replies)
Discussion started by: cgkmal
4 Replies

10. UNIX Desktop Questions & Answers

How to concatenate consecutive lines

I have a few lines like -- feature 1, subfeat 0, type 3, subtype 1, value 0, -- feature 1, subfeat 0, type 1, subtype 1, value 0, I would like to concatenate the... (1 Reply)
Discussion started by: shivi707
1 Replies
Login or Register to Ask a Question