Two Input Lines Into Single Output Line (CSV)


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Two Input Lines Into Single Output Line (CSV)
# 8  
Old 07-24-2011
Ok, I've found the bug that was causing the double comma. The only dummy input file that was without an ntp record was the last one, so I wasn't seeing that I had coded that incorrectly.

Have a go with this:
Code:
 awk '
    BEGIN { n = 0; k = 1;      # needed for bug fix
    }

    FILENAME != last {
        fn = FILENAME;
        gsub( ".txt", "", fn );
        printf( "%s%s%s", k ? "" : ",", n ? "\n" : "",  fn );  # bug fix here
        last = FILENAME;
        n = 1;     # signal need for newline before fn (prevent lead newline)
        k = 0;     # when set, signals no comma needed in previous printf
    }

    /ntp server/ || /ntp peer/{
        printf( ",%s", $0 );
        k = 1;
    }

    END {
        printf( "%s\n", k ? "" : "," );
    }'

I'm still perplexed with the newline when there is a second record because I'm not seeing that with my dummied up test data. I'll continue to play with this for a bit, but wanted to see that this should have made the difference with the double commas.

I assumed that the 'ntp peer' record was to be handled the same way as the ntp server record. If not, post how you need it and a sample of the text from the file.

And not to worry about not posting the o/s type. It can help, but in this case I don't think OSX and its tools are to blame.

Last edited by agama; 07-24-2011 at 01:17 AM.. Reason: clarification
# 9  
Old 07-24-2011
Quote:
Originally Posted by agama
Ok, I've found the bug that was causing the double comma. The only dummy input file that was without an ntp record was the last one, so I wasn't seeing that I had coded that incorrectly.
Me neither! Smilie (as if!)

Quote:
Originally Posted by agama

I'm still perplexed with the newline when there is a second record because I'm not seeing that with my dummied up test data. I'll continue to play with this for a bit, but wanted to see that this should have made the difference with the double commas.
Then it's obviously some important detail that I'm failing to share with you.

Quote:
Originally Posted by agama
I assumed that the 'ntp peer' record was to be handled the same way as the ntp server record. If not, post how you need it and a sample of the text from the file.
You assume correctly!

Quote:
Originally Posted by agama
And not to worry about not posting the o/s type. It can help, but in this case I don't think OSX and its tools are to blame.
Glad to hear it!

And I just learned something new about these forums...

I was refreshing the thread each time before I posted an edit to my last post with the idea I would know that you hadn't been back (or at least that you hadn't posted anything new). However, when I reloaded the thread from the forum index, I saw this post from you. Sorry for any confusion that may have introduced!!

---------- Post updated at 10:02 PM ---------- Previous update was at 09:52 PM ----------

OK, what is it about:

awk 'BEGIN (...)

I'm having trouble with??

I guess I should call it a night and start fresh in the morning. I can't thank you enough for what you've done to get me this far...
SmilieSmilie
# 10  
Old 07-24-2011
Quote:
Originally Posted by svermill
OK, what is it about:

awk 'BEGIN (...)

I'm having trouble with??

I guess I should call it a night and start fresh in the morning. I can't thank you enough for what you've done to get me this far...
SmilieSmilie
I didn't include the ls command and the pipe. My test script is still using xargs, so I only cut and pasted the awk portion.

I'm calling it a night; not sure when I can peek in tomorrow -- it will probably be early afternoon. You are most welcome for the help. I'm always surprised at what I end up learning as I figure out, or read other's solutions.
# 11  
Old 07-24-2011
Quote:
Originally Posted by agama
You are most welcome for the help. I'm always surprised at what I end up learning as I figure out, or read other's solutions.
Understood. I have spent countless hundreds (likely thousands) of hours over the years doing likewise for others on the network engineering side (I'm a CCIE if that means anything to you), and I've learned as much if not more from that process as anything.

Quote:
Originally Posted by agama
I didn't include the ls command and the pipe. My test script is still using xargs, so I only cut and pasted the awk portion.
I have tried a bunch of different things but I always get a ">" after the first line of whatever I paste into my shell. Likely my ignorance is start to seriously show if it hasn't already. Sleep will likely help too!

Good night!
# 12  
Old 07-24-2011
Do you have Perl on your Mac?
Code:
perl -nle 'printf "$ARGV," if $.==1;printf "$_," if /ntp server/;if (eof){$.=0;printf "\n"}' * > out.csv

# 13  
Old 07-24-2011
Quote:
Originally Posted by svermill
I have tried a bunch of different things but I always get a ">" after the first line of whatever I paste into my shell. Likely my ignorance is start to seriously show if it hasn't already.
I can explain the > --

The greather than sign is the 'secondary' prompt (also known as PS2) which is issued when a newline is entered before a terminating single quote. So, pasting the awk snipit would cause this.

The easy solution, or in my opinion the easiest, is to edit a file (test_script.ksh or somesuch) with your favorite editor and paste the code example into that. Save the file, and then from the command line execute the script using ksh or bash (or your preferred shell). Like this:

Code:
ksh test_script.ksh

The other advantage to this is that you can edit the file and easily tweek things if you want, and you have the template for a script when you have something that works.

You can also use Kshell's (or bash's) command line editing features that allow direct invocation of emacs or vi, but that's more difficult to explain if you aren't already familiar with it.
# 14  
Old 07-24-2011
Yeah, I definitely needed a good night's sleep. So I stayed up and smoked a cigar instead! Smilie

Anyway, with xargs using both bash and ksh, I was only getting the hostname of the very first device in the folder - nothing else (so no commas, no ntp commands, no nothin'). So I dropped xargs and did *.txt in awk. I get an output file and both 'ntp server' and 'ntp peer' commands are present. However, only my to two core devices have a single-line entry for multiple ntp matches; all others have a newline between first and second entries when there are two matches.

Now this may or may not be an interesting detail: I was unaware that my core devices actually have three 'ntp server' commands. And yes, they're strangely all on one single line in the output file. I don't see any real differences between the core configs and any of the other configs??

---------- Post updated at 10:49 AM ---------- Previous update was at 10:45 AM ----------

Quote:
Originally Posted by bartus11
Do you have Perl on your Mac?
Code:
perl -nle 'printf "$ARGV," if $.==1;printf "$_," if /ntp server/;if (eof){$.=0;printf "\n"}' * > out.csv

Hi bartus11,

Yes, I just installed ActivePerl the other day (and I believe that some stripped down/likely outdated version of Perl ships native to Mac OS X). I'm trying to ensure that I actually understand (as much as possible) what I'm doing, though, so I don't want to jump to Perl just yet - my head is too full of new stuff this weekend as it is! But I promise that once I get this other approach sorted out, I'll turn my attention to your example Perl code. It certainly looks nice and streamlined!

Thanks again to you both! Smilie

---------- Post updated at 10:53 AM ---------- Previous update was at 10:49 AM ----------

OK, I admit that I just pasted in your Perl script to see what would happen. Smilie

Here's an example of the output:


router1.txt,
router2.txt,ntp server 10.10.250.1
,
switch1.txt,ntp server 10.10.250.2
,ntp server 10.10.250.1 prefer
,

Still suffering from newline issues??
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Get an output of lines in pattern 1st line then 10th line then 11th line then 20th line and so on.

Input file: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 (6 Replies)
Discussion started by: Sagar Singh
6 Replies

2. Shell Programming and Scripting

ksh - Read input from file and output CSV i same row

Hello I have the following output and want the output to look: FROM: GigabitEthernet0/0 is up, line protocol is up 1 input errors, 0 CRC, 0 frame, 1 overrun, 0 ignored 275 output errors, 0 collisions, 3 interface resets GigabitEthernet0/1 is up, line protocol is up 0... (4 Replies)
Discussion started by: JayJay2018
4 Replies

3. Shell Programming and Scripting

How do I split a single-line input into five lines?

Example input: John:Shepherd:770-767-4040:U.S.A:New York Mo Jo:Jo Jo: 666-666-6666:U.S.A:Townsville Expected Output: First Name: John Last Name: Shepherd Phone Number: 770-767-4040 Country: U.S.A State: New York First Name: Mo Jo Last Name: Jo Jo Phone Number: 666-666-6666... (10 Replies)
Discussion started by: Camrikron
10 Replies

4. UNIX for Dummies Questions & Answers

Need help combining txt files w/ multiple lines into csv single cell - also need data merge

:confused:Hello -- i just joined the forums. I am a complete noob -- only about 1 week into learning how to program anything... and starting with linux. I am working in Linux terminal. I have a folder with a bunch of txt files. Each file has several lines of html code. I want to combine... (2 Replies)
Discussion started by: jetsetter
2 Replies

5. Shell Programming and Scripting

turning output of two lines into one CSV line

Hi, I am attempting to use sed on linux to do something trivial. I am also too embarassed to show you what I have tried so far! What I am trying to do should be trivial, if I knew what I was doing, but I don't. Would someone please help me? Here is my problemI have a ASCII file that has the... (4 Replies)
Discussion started by: Jon8987z
4 Replies

6. Shell Programming and Scripting

Input two variable on single line

Can we input two variable on single line that separate by space example user input "list jpg" it will list all jpg files in current directory (3 Replies)
Discussion started by: guidely
3 Replies

7. Shell Programming and Scripting

Multiple lines in a single column to be merged as a single line for a record

Hi, I have a requirement with, No~Dt~Notes 1~2011/08/1~"aaa bbb ccc ddd eee fff ggg hhh" Single column alone got splitted into multiple lines. I require the output as No~Dt~Notes 1~2011/08/1~"aaa<>bbb<>ccc<>ddd<>eee<>fff<>ggg<>hhh" mean to say those new lines to be... (1 Reply)
Discussion started by: Bhuvaneswari
1 Replies

8. Shell Programming and Scripting

Break lines up into single lines after each space in every line

It sounds a bit confusing but what I have is a text file like the example below (without the Line1, Line2, Line3 etc. of course) and I want to move every group of characters into a new line after each space. Example of text file; line1 .digg-widget-theme2 ul { background: rgb(0, 0, 0) none... (7 Replies)
Discussion started by: lewk
7 Replies

9. Shell Programming and Scripting

single line input to multiple line output with sed

hey gents, I'm working on something that will use snmpwalk to query the devices on my network and retreive the device name, device IP, device model and device serial. I'm using Nmap for the enumeration and sed to clean up the results for use by snmpwalk. Once i get all the data organized I'm... (8 Replies)
Discussion started by: mitch
8 Replies

10. Shell Programming and Scripting

Need output in different lines not in one single line

I am getting the coutput like this as show below in one single line, where as the command is executed is several lines and the output should also be requied in several lines, not in one single line. Anyone any idea? p4 opened -a | grep *locked* | awk '{ printf $8 }' >/tmp/aa $ cat... (1 Reply)
Discussion started by: csaha
1 Replies
Login or Register to Ask a Question