Strange behaviour when output to terminal vs file (awk)


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Strange behaviour when output to terminal vs file (awk)
# 1  
Old 12-12-2012
Strange behaviour when output to terminal vs file (awk)

Hi all !

I noticed something very weird.

I have a large pipe delimited file (20 fields/3,000 records) that looks like that:
Code:
AAA|BBB|11111|22222|...|($NF of record 1)
CCC|DDD|33333|44444|...|($NF of record 2)
CCC|DDD|55555|66666|...|($NF of record 3)

For the lines with same 1st and 2nd fields, I write the 3rd up to 20th fields at the end of the previous line.
In order to obtain this:
Code:
AAA|BBB|11111|22222|...|($NF of record 1)
CCC|DDD|33333|44444|...|($NF of record 2)|55555|66666|...|$NF of record 3

I use this command:
Code:
gawk 'BEGIN{FS=OFS="|"}{a[$1$2]=(a[$1$2]?a[$1$2]:$1 FS $2) FS $3 FS $4 FS $5 FS $6 FS $7 FS $8 FS $9 FS $10 FS $11 FS $12 FS $13 FS $14 FS $15 FS $16 FS $17 FS $18 FS $19 FS $20}END{for(i in a) print a[i]}' input

When I redirect the command to my terminal (iTerm) it works perfectly (output has 200 records).
When I redirect the command to an output file and count (wc -l) the number of lines, it also gives me 200 records.

But when I open the file with my usual text editor (Smultron), the duplicated 1st and 2nd fields just disappeared and the remaining fields are written in a new line (giving me the same number of lines as the input):
Code:
AAA|BBB|11111|22222|...|($NF of record 1)
CCC|DDD|33333|44444|...|($NF of record 2)
|55555|66666|...|($NF of record 3)

I really don't understand what is wrong.
I used this command many times, it is the first time it does that. Plus I tested the command with an input file with just 3 rows (like the example above, and it woks).

Have you already got this problem?!!!

Thanks for your help.

---------- Post updated at 09:09 PM ---------- Previous update was at 08:46 PM ----------

***EDIT***
- Both iTerm and Smultron use Unicode (UTF-8)
- It worked properly when I don't go up to the last field
# 2  
Old 12-16-2012
Does your Smultron editor force a line break on long lines?
# 3  
Old 12-16-2012
Note that with 20 fields on each of 3,000 input records being combined into 200 output lines, your average output lines with have 300 fields and some lines could have many more. You don't give us any indication of what system you're using, nor of the contents of most of the input fields. The awk utility and most editors are only defined to work on text files, and by definition, lines in a text file can't be longer than LINE_MAX bytes (including the terminating newline character). (Try:
Code:
getconf LINE_MAX

to determine the value of LINE_MAX on your system. The standards only require that implementations support lines up to 2,048 bytes per line.) Are you sure that none of your output lines exceed LINE_MAX?

If you ask awk to print a line that is longer than LINE_MAX bytes long, the results are unspecified. If you use ed, ex, grep, sed, vi (or any of LOTS of other standard utilities that are described as processing text files) to read or write or create internal lines longer than LINE_MAX bytes long, the results are unspecified. There are very few standard text processing utilities that are defined to work on lines with arbitrary lengths (cut, fold, and paste).
# 4  
Old 12-16-2012
Found this !

Hello,

found this :

Smultron is an elegant and powerful text editor for Mac OS X that is easy to use.

So i think it's a MAC.

Regards
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Print Terminal Output Exactly how it Appears in the Terminal to a New Text File

Hello All, I have a text file containing output from a command that contains lots of escape/control characters that when viewed using vi or view, looks like jibberish. But when viewed using the cat command the output is formatted properly. Is there any way to take the output from the cat... (7 Replies)
Discussion started by: mrm5102
7 Replies

2. Shell Programming and Scripting

awk Strange behaviour in AIX

Can someone please explain the strange behaviour.. I was just trying a few things to learn awk.. in the below code when I start the braces in the same line, the output is as expected, when I start at next line, output is displayed twice. Please see the file, code I tried and output below. ... (2 Replies)
Discussion started by: Kulasekar
2 Replies

3. Shell Programming and Scripting

Strange behaviour of arrays in awk

Imagine 2 files f1 f2: file1_l1_c1 code_to_find file1_l1_c3 file1_l2_c1 file1_code2 file1_l2_c3 file1_l3_c1 file1_code3 file1_l3_c3 file2_l1_c1 file2_l1_c2 code_to_find file2_l2_c1 file2_l2_c2 file2_code5 file2_l3_c1 file2_l3_c2 file2_code3 Say we want to print lines from f2 having... (5 Replies)
Discussion started by: ripat
5 Replies

4. UNIX for Dummies Questions & Answers

Strange perl print output behaviour

Hi, Today I have found the following case in perl: print "length:$lengths\tsum:". $count{$lengths}+$count_pair{$lengths}."\tindi:$count{$lengths}\t$count_pair{$lengths}\n";This give output as That means the first part of print is not printing. Only the values after the additions are printed.... (5 Replies)
Discussion started by: gvj
5 Replies

5. Shell Programming and Scripting

Strange terminal behaviour after killing ssh

Hi, We have a problem where occasionally an ssh will hang for no apparent reason preventing the rest of the script continuing. To deal with this I am trying to write a wrapper script to kill a hung ssh command after a specified period. The scripts use a sleep command running in the... (2 Replies)
Discussion started by: RECrerar
2 Replies

6. Shell Programming and Scripting

Strange RegExp Behaviour

Hello, I was trying to identify lines who has a word of the following pattern "xyyx" (where x, and ys are different characters). I was trying the following grep - egrep '(\S)()\2\1' This pattern do catches the wanted pattern, but it also catches "GGGG" or "CCCC" patterns. I was trying to... (5 Replies)
Discussion started by: itskov
5 Replies

7. Shell Programming and Scripting

Strange behaviour with perl i/o?

Hi All, I got a strange problem here. I have a perl script which is fetching data from a database table and writing a file with that data. If i run that script from linux command line, the file it creates is a normal ascii text file without any binary character in it.But... (9 Replies)
Discussion started by: DILEEP410
9 Replies

8. UNIX for Dummies Questions & Answers

Strange Program behaviour

Had a strange thing going on with my code. It's ok I figured it out for myself.... (2 Replies)
Discussion started by: mrpugster
2 Replies

9. UNIX for Advanced & Expert Users

Strange sed behaviour

$ echo a.bc | sed -e "s/\|/\\|/g" |a|.|b|c| $ Is the behavior of the sed statement expected ? Or is this a bug in sed ? OS details Linux 2.6.9-55.0.0.0.2.ELsmp #1 SMP Wed May 2 14:59:56 PDT 2007 i686 i686 i386 GNU/Linux (8 Replies)
Discussion started by: vino
8 Replies

10. Shell Programming and Scripting

A Strange Behaviour!!!

Can some-one give me a view to this : I have a directory in an unix server, having permissions r-xr-xr-x .This directory is basically a source directory. Now there is another directory basically the destination directory which has all the permissions. Note:I log in as not the owner,but user... (5 Replies)
Discussion started by: navojit dutta
5 Replies
Login or Register to Ask a Question