UNIX for Dummies Questions & Answers

Hi,

I am trying to do some transformation on a large file and I am getting some troubles trying remove newlines only when the last character of a line is a symbol (in this case is a pipe "|").

I have tried with sed like this:

Code:

sed -i 's/|\n/|/g' my_file

or

Code:

sed -i 's/|$/|/gg' my_file

but as sed is a line operation command does not find newlines. I have also tried:

Code:

sed -i ':a;N;$!ba;s/|\n//g' my_file

This works for small files, but not for my "large" file (actually is only 4Gb).

I could do it using vi and %:s/|\n/|/g' but I need to be able to execute inside a script (also is a slow solution).

Any help is appreciatted.

Hello ngb,

AFSIK only GNU sed will take that \n substitution. Could you please try following and let me know if this helps.

Code:

 awk '{gsub("\n"/,X,$0);print}' Input_file > tmp_Input_file
 mv tmp_Input_file Input_file

You could first test awk command if that works successfully then you can try mv command to change tmp_Input_file to Input_file or if you want to change to a different file then you can change it accordingly. Also remove carriage characters if any in your file by doing tr -d '\r' < Input_file > Output_file.


Thanks,
R. Singh

Thank you RavinderSingh13 for your help.

I am getting this error:

awk: {gsub("\n"/,X,$0);print}
awk: ^ syntax error
awk: fatal: 0 is invalid as number of arguments for gsub


The error is pointing to the comma between / and X.

TrY:

Code:

sed -e :a -e '/|$/{$!N;s/\n//;};ta' file

With GNU sed you could also try

Code:

sed ':a;/|$/{$!N;s/\n//;};ta' file

The difference is that this works on a line-by-line basis, whereas the last example in post #1 first reads the entire file before it attempts to globally substitute...


--
Any POSIX compliant sed understands \n in the regex part of the s-function if it is not used within a square bracket expression.

Thank you Scrutinizer!!! That worked for the large file too. I will check further later but your command helped a lot.

I got a request to explain the sed statement in post #4:

Code:

-e :a        # define label a
-e '/|$/{    # if the last character is a vertical bar, then
$!N          # if it is not the last line then append the next line              
s/\n//;}     # and replace a newline character. 
ta           # if the last replacement was successful, then go to label a

The loop is necessary, so that it also works for multiple consecutive lines that end with a vertical bar..