Two Input Lines Into Single Output Line (CSV)

Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Two Input Lines Into Single Output Line (CSV)
# 22  
Old 07-24-2011
Well, if you want to clean your files from those unwanted characters, run this:
perl -i -pe 's/\x0d$//' *

Then you will be able to run AWK code on them (probablySmilie)
# 23  
Old 07-24-2011
Looks like I cross posted...

Tweek above.

I suspected it might be something like a bloody carriage-return (ctl-m) but assumed that these files were generated on a UNIX box; most UNIX applications don't add the unnecessary character.

As for xargs, you might try something simple to see what is happening.

ls *txt | xargs echo "files:"

You should see one "files:" string followed by the list of files. If that seems to work, try it again with the awk programme. If you see something different, then have a peek at the man page.
# 24  
Old 07-24-2011
Yes, that works like a champ. This too completes in one second or less, so my lack of xargs isn't hurting me too much. But I will look into it and see if it's something I can get implemented on my Mac. If so, I'll take the above code and the rest of the script you earlier posted to arrive at a final solution.

I can't thank you both enough. If it's any comfort, I've been remotely helping someone to set up a network lab using the Dynamips hypervisor at the same time you've been helping me! I doubt I'd ever be of much help to anyone around here due to my entry-level knowledge of scripting. But I do try to give back in other ways. Thanks again.


---------- Post updated at 01:26 PM ---------- Previous update was at 01:19 PM ----------

Originally Posted by agama
As for xargs, you might try something simple to see what is happening.

ls *txt | xargs echo "files:"

You should see one "files:" string followed by the list of files. If that seems to work, try it again with the awk programme. If you see something different, then have a peek at the man page.
Strange. I do indeed see the expected output when I do the above. However, when I try to invoke that in ksh, I still just get the hostname of the very first device and then silence on the line. The full code at this point:


ls *.txt | xargs awk '
    BEGIN { n = 0; k = 1; }

    FILENAME != last {
        fn = FILENAME;
        gsub( ".txt", "", fn );
        printf( "%s%s%s", k ? "" : ",", n ? "\n" : "",  fn );
        last = FILENAME;
        n = 1;     # signal need for newline before fn (prevent lead newline)
        k = 0;     # when set, signals comma

    /ntp server/ || /ntp peer/ {
        gsub( "\r", "", $0 );           # ditch the charriage return
        printf( ",%s", $0 );
        k = 1;

    END {
        printf( "%s\n", k ? "" : "," );
    } ' > out5.txt

Am I screwing something up in the script that you can see at this point?
# 25  
Old 07-24-2011
Originally Posted by svermill
Am I screwing something up in the script that you can see at this point?
Glad it worked for you... always a pleasure to help.

I cut/pasted your code and it works fine for me, so from my point of view there's nothing that you've done wrong. For now your solution will work. Be aware that if your number of files grows large, you might get a message along the lines of "argument list too large." At that point you'd need to come back to the xargs issue.

You can try this simpler awk and see what the results are. Should print each filename on a new line:

ls *txt | xargs awk ' FILENAME != last { print FILENAME; last = FILENAME; }'

If it doesn't, I'm not sure what is up.
# 26  
Old 07-24-2011
Ah, one mystery solved and a new one introduced!

Your above abbreviated code led me to a discovery. It printed the very first hostname and then threw me an error on the very next file, which is something akin to "batchfile.txt." This is obviously not output from my TCL/Expect script, but rather is a big long list of TCL commands and Expect interactive stuff that I paste into my TCL shell to create all of the .txt files I'm performing post-analysis on with these new scripts you have helped me to create. At the end of every one of those TCL commands is something like "router1.txt," which is precisely what creates the .txt files with an embedded hostname in the file. Not sure how, but that was clearly a problem for xargs. I converted all non-router/switch text files to a .xtx extension and then ran the script and now I have something more along the lines I was hoping for. HOWEVER... Smilie

My output file right now is "out5.txt." Towards the very end of this file, I actually see an entry of "out5" as if it were a hostname, and then a very VERY long comma-separated line of 'ntp server' and 'ntp peer' strings. So somehow the script is actually catching up to itself before it completes. Right? I suppose I can just exclude output file from being evaluated as input, no?

---------- Post updated at 02:27 PM ---------- Previous update was at 02:00 PM ----------

Just dawned on me that I had stopped using .csv in my output file so that I wouldn't have to do "Open With" and could just open it directly in my text editor. I will, however, go back to .csv for production stuff and then this simply wouldn't be an issue! Smilie

OK, time for a nice cold beer! Smilie
# 27  
Old 07-24-2011
Very interesting discovery. What was the error?

Sounds like you've figured out the work around, but it would be interesting to see what was triggering it. Is batchfile.txt something that you can post?
# 28  
Old 07-24-2011
I would deny you no request at this moment in time! Of course there is no "router1" or "router2" or any of that - everything has been sanitized along the way. In keeping with that spirit, here is a single line from "batchfile" - this repeats as required by IP address for the given scenario...

tclsh router-config.tcl telnet commandlist.txt AAAusername AAApassword AAAenable_password > router1.txt

So I doubt that any of this requires explanation for you but just for the record, I have a long list of commands that get sent to the routers/switches via "commandlist.txt." Within the TCL script, the three AAA entries above get sent via Expect. The output file part is obvious. In this specific case, I only had one host output .txt file that alphanumerically preceded "batchfile" (and obviously that wasn't "router1"!) And also that one device lacked an 'ntp server/peer' command in it's config, hence this scenario where I was seeing the very first hostname in my out.txt/csv file and then nothing else!!

---------- Post updated at 07:43 PM ---------- Previous update was at 07:36 PM ----------

Oh, and you rightly asked me for the specific error. It looked like this:

svermill's-MacBook-Pro:folder$ ksh "NTP Script.ksh"
awk: can't open file batchfile
input record number 16, file batchfile
source line number 1

Hope that helps to understand what the issue was - I can see it in a very pixilated way but it's likely crystal-clear to yourself and many other code ninjas here...
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Get an output of lines in pattern 1st line then 10th line then 11th line then 20th line and so on.

Input file: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 (6 Replies)
Discussion started by: Sagar Singh
6 Replies

2. Shell Programming and Scripting

ksh - Read input from file and output CSV i same row

Hello I have the following output and want the output to look: FROM: GigabitEthernet0/0 is up, line protocol is up 1 input errors, 0 CRC, 0 frame, 1 overrun, 0 ignored 275 output errors, 0 collisions, 3 interface resets GigabitEthernet0/1 is up, line protocol is up 0... (4 Replies)
Discussion started by: JayJay2018
4 Replies

3. Shell Programming and Scripting

How do I split a single-line input into five lines?

Example input: John:Shepherd:770-767-4040:U.S.A:New York Mo Jo:Jo Jo: 666-666-6666:U.S.A:Townsville Expected Output: First Name: John Last Name: Shepherd Phone Number: 770-767-4040 Country: U.S.A State: New York First Name: Mo Jo Last Name: Jo Jo Phone Number: 666-666-6666... (10 Replies)
Discussion started by: Camrikron
10 Replies

4. UNIX for Dummies Questions & Answers

Need help combining txt files w/ multiple lines into csv single cell - also need data merge

:confused:Hello -- i just joined the forums. I am a complete noob -- only about 1 week into learning how to program anything... and starting with linux. I am working in Linux terminal. I have a folder with a bunch of txt files. Each file has several lines of html code. I want to combine... (2 Replies)
Discussion started by: jetsetter
2 Replies

5. Shell Programming and Scripting

turning output of two lines into one CSV line

Hi, I am attempting to use sed on linux to do something trivial. I am also too embarassed to show you what I have tried so far! What I am trying to do should be trivial, if I knew what I was doing, but I don't. Would someone please help me? Here is my problemI have a ASCII file that has the... (4 Replies)
Discussion started by: Jon8987z
4 Replies

6. Shell Programming and Scripting

Input two variable on single line

Can we input two variable on single line that separate by space example user input "list jpg" it will list all jpg files in current directory (3 Replies)
Discussion started by: guidely
3 Replies

7. Shell Programming and Scripting

Multiple lines in a single column to be merged as a single line for a record

Hi, I have a requirement with, No~Dt~Notes 1~2011/08/1~"aaa bbb ccc ddd eee fff ggg hhh" Single column alone got splitted into multiple lines. I require the output as No~Dt~Notes 1~2011/08/1~"aaa<>bbb<>ccc<>ddd<>eee<>fff<>ggg<>hhh" mean to say those new lines to be... (1 Reply)
Discussion started by: Bhuvaneswari
1 Replies

8. Shell Programming and Scripting

Break lines up into single lines after each space in every line

It sounds a bit confusing but what I have is a text file like the example below (without the Line1, Line2, Line3 etc. of course) and I want to move every group of characters into a new line after each space. Example of text file; line1 .digg-widget-theme2 ul { background: rgb(0, 0, 0) none... (7 Replies)
Discussion started by: lewk
7 Replies

9. Shell Programming and Scripting

single line input to multiple line output with sed

hey gents, I'm working on something that will use snmpwalk to query the devices on my network and retreive the device name, device IP, device model and device serial. I'm using Nmap for the enumeration and sed to clean up the results for use by snmpwalk. Once i get all the data organized I'm... (8 Replies)
Discussion started by: mitch
8 Replies

10. Shell Programming and Scripting

Need output in different lines not in one single line

I am getting the coutput like this as show below in one single line, where as the command is executed is several lines and the output should also be requied in several lines, not in one single line. Anyone any idea? p4 opened -a | grep *locked* | awk '{ printf $8 }' >/tmp/aa $ cat... (1 Reply)
Discussion started by: csaha
1 Replies
Login or Register to Ask a Question

Featured Tech Videos