Unix/Linux Go Back    


Shell Programming and Scripting BSD, Linux, and UNIX shell scripting — Post awk, bash, csh, ksh, perl, php, python, sed, sh, shell scripts, and other shell scripting languages questions here.

concatenate lines using shell scripting

Shell Programming and Scripting


Tags
shell script

Closed    
 
Thread Tools Search this Thread Display Modes
    #8  
Old Unix and Linux 09-06-2009   -   Original Discussion by dtdt
dtdt's Unix or Linux Image
dtdt dtdt is offline
Registered User
 
Join Date: Sep 2009
Last Activity: 24 January 2015, 10:15 PM EST
Posts: 33
Thanks: 1
Thanked 0 Times in 0 Posts
geeez, why it doesn't work on my file.

awk '/=/{$0="\n"}1' ORS="" $DSTDIR/ttt > $DSTDIR/ttt2

both ttt, ttt2 are still the same. what did i do wrong?

---------- Post updated at 01:51 PM ---------- Previous update was at 01:46 PM ----------

it doesn't do the trick for me. it duplicates para after para...

not concatnate those lines.

Quote:
Originally Posted by bakunin View Post
in sed it is easy: we have two types of lines, the ones reading "=" and the others. When we encounter a "="-line, we want to print out what we have so far, minus the newlines. If we encounter one of the other lines we want to store its contents until we encounter a "="-line.

sed has a so-called "hold space", think of it as a variable, where you can store things until you need them. We append everything to this hold space until we encounter a "="-line, then we recall the hold space, filter out all embedded newlines and print it, then start over.

In the following script i have put in comment for your understanding, remove them, because sed doesn't allow inline commenting in scripts. Furthermore, you can put the whole script on one line, replacing linefeeds with semicolons:



Code:
sed -n '/^=/ {                   # if a line starts with "="
          s/.*//                 # delete this lines content
          x                      # exchange the pattern space (empty) and the hold space
          s/\n//g                # delete newlines
          p                      # then print what you have
     }
     /^=/ ! {                    # if a line doesn't start with "="
          H                      # append it to the hold space
     }' /your/file > newfile

for short:



Code:
sed -n '/^=/{s/.*//;x;s/\n//gp};/^=/!{H}' /your/file > newfile

I hope this helps.

bakunin
Sponsored Links
    #9  
Old Unix and Linux 09-06-2009   -   Original Discussion by dtdt
bakunin's Unix or Linux Image
bakunin bakunin is offline Forum Staff  
Bughunter Extraordinaire
 
Join Date: May 2005
Last Activity: 22 January 2018, 12:33 PM EST
Location: In the leftmost byte of /dev/kmem
Posts: 5,678
Thanks: 112
Thanked 1,633 Times in 1,198 Posts
I just noticed that in your example text the trailing commata were also stripped off. To achieve this i would like to add a line to my script, the purpose should be evident:



Code:
sed -n '/^=/ {                   # if a line starts with "="
          s/.*//                 # delete this lines content
          x                      # exchange the pattern space (empty) and the hold space
          s/\n//g                # delete newlines
          s/[,;]$//
          p                      # then print what you have
     }
     /^=/ ! {                    # if a line doesn't start with "="
          H                      # append it to the hold space
     }' /your/file > newfile

I hope this helps.

bakunin
Sponsored Links
    #10  
Old Unix and Linux 09-06-2009   -   Original Discussion by dtdt
dtdt's Unix or Linux Image
dtdt dtdt is offline
Registered User
 
Join Date: Sep 2009
Last Activity: 24 January 2015, 10:15 PM EST
Posts: 33
Thanks: 1
Thanked 0 Times in 0 Posts
i guess i have some problem with file on unix/windows. i have to go out now. will check later when i come back. so far, no good.

thanks for all ur help

---------- Post updated at 10:32 PM ---------- Previous update was at 02:57 PM ----------

it's so freaking me out right now.

only protocomm's solution partially work. that means i can see it worked in window's word, but not on note pad. when i open the output file in liunx server, it's not working like note pad.

i have no idea what's happening. any suggestions?

thanks all
    #11  
Old Unix and Linux 09-07-2009   -   Original Discussion by dtdt
bakunin's Unix or Linux Image
bakunin bakunin is offline Forum Staff  
Bughunter Extraordinaire
 
Join Date: May 2005
Last Activity: 22 January 2018, 12:33 PM EST
Location: In the leftmost byte of /dev/kmem
Posts: 5,678
Thanks: 112
Thanked 1,633 Times in 1,198 Posts
Quote:
Originally Posted by dtdt View Post
i have no idea what's happening. any suggestions?
Since you don't show us any real example of what your file looks like we are left to guesses. Therefore the following is again - a guess (btw.: my code worked perfectly on AIX, Solaris and Ubuntu Linux - more Unix/Linux dialects i don't have at hand but probably nothing will change the result).

Files are different in Windows/DOS and UNIX. The reason is that newlines are encoded differently. In UNIX dialects a newline is encoded in a single character, a "^M" (control-m), the linefeed character. In DOS/Windows a newline is encoded in two characters, a CR (carriage return) and a LF (line feed).

Create a file under UNIX (issue "cat > unixfile", then start typing, press "control-D" to end the input) and get a file created under Windows. Now issue "od -ax dosfile | more" and "od -ax unixfile | more" in two windows and observe the difference.

If this is the case with your file try "ASCII mode" when you are transferring it via ftp from DOS to UNIX and vice versa. The ASCII mode (instead of "binary" mode, which is default) takes care of exactly this thing.

I hope this helps.

bakunin
Sponsored Links
    #12  
Old Unix and Linux 09-07-2009   -   Original Discussion by dtdt
dtdt's Unix or Linux Image
dtdt dtdt is offline
Registered User
 
Join Date: Sep 2009
Last Activity: 24 January 2015, 10:15 PM EST
Posts: 33
Thanks: 1
Thanked 0 Times in 0 Posts
would you please PM your email? I'd like to set you the file as attachment. this is the first time i met this and i completely lost.
Sponsored Links
    #13  
Old Unix and Linux 09-07-2009   -   Original Discussion by dtdt
durden_tyler's Unix or Linux Image
durden_tyler durden_tyler is offline Forum Advisor  
Registered User
 
Join Date: Apr 2009
Last Activity: 9 September 2017, 1:30 PM EDT
Posts: 2,083
Thanks: 21
Thanked 383 Times in 346 Posts
Quote:
Originally Posted by dtdt View Post
i have a mega file in this format:

a,
b,
c,
d,
=
a2,
b2,
c2,
d2,
=
a3,
b3
=

i want to combine lines until = meet. the result should be :
a,b,c,d,
a2,b2,c3,d2
a3,b3
...
One way to do it in Perl:



Code:
$ 
$ cat f2
a,
b,
c,
d,
=
a2,
b2,
c2,
d2,
=
a3,
b3
=
$ 
$ perl -ne 'chomp; $x.=$_; END{$x=~s/[,]*=/\n/g; print $x}' f2
a,b,c,d
a2,b2,c2,d2
a3,b3
$ 
$

If you use vi or vim to open your file in Unix/Linux and see "^M" characters at the end of each line, then that could be one of your problems.

tyler_durden
Sponsored Links
    #14  
Old Unix and Linux 09-07-2009   -   Original Discussion by dtdt
dtdt's Unix or Linux Image
dtdt dtdt is offline
Registered User
 
Join Date: Sep 2009
Last Activity: 24 January 2015, 10:15 PM EST
Posts: 33
Thanks: 1
Thanked 0 Times in 0 Posts
thanks all. the f* problem is the format with the file. it's really strange, i got the file after parsing a web page. i used the program hundreds of times, and this is the first time i have this issue. after i run a dos2unix convert, all works! thanks a lot.
Sponsored Links
Closed

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Linux More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
concatenate log file lines up to timestamp AlanC Shell Programming and Scripting 4 06-29-2009 12:38 PM
Concatenate lines in a single paragraph cgkmal Shell Programming and Scripting 4 05-17-2009 02:02 AM
sed / awk to concatenate lines until blank line dunstonrocks Shell Programming and Scripting 11 03-31-2009 12:11 AM
How to concatenate consecutive lines shivi707 UNIX Desktop Questions & Answers 1 01-12-2009 09:08 AM
need to concatenate two lines if the line doesnt end with quotes laxmi131 UNIX for Advanced & Expert Users 9 10-27-2008 08:22 AM



All times are GMT -4. The time now is 06:18 AM.