Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

how to delete \n in a large file with sed

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting how to delete \n in a large file with sed
# 1  
Old 06-15-2010
Question how to delete \n in a large file with sed
Hello

i have a big file with a specific format and delimiter is "§" :

Code:
§field1§$field2§$field3§$field4§$field5§$field6§$field§

in this file we have a field which are very long (more than 20000 chars !!!!)
so through vi i cant manipulate them.
despite this i managed to suppress lines that do not contain the '§' delimiter because we have a restriction of 250 chars.

so now i want to move up the end of the record to the previous line to complete correctly a record.
here is an example :

Code:
§IDNUM§$NAME§$SURNAME§$DATE§$COMMENTS:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxx
xxxx§$OPTION§$FLAG§

1- delete of the line without delimiter we have now this
Code:
§IDNUM§$NAME§$SURNAME§$DATE§$COMMENTS:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxx§$OPTION§$FLAG§

2- now i want this (suppress the \n to move up the end of the record)
Code:
§IDNUM§$NAME§$SURNAME§$DATE§$COMMENTS:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx§$OPTION§$FLAG§

How can i do that ?
We are on sun solaris SunOS 5.9
Last edited by vgersh99; 06-15-2010 at 01:10 PM.. Reason: code tags, please!
# 2  
Old 06-15-2010
Code:
# cat a
§IDNUM§$NAME§$SURNAME§$DATE§$COMMENTS:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxx§$OPTION§$FLAG§

Code:
# sed -e :jx -e '$!N;s/\n//;tjx' a
§IDNUM§$NAME§$SURNAME§$DATE§$COMMENTS:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx§$OPTION§$FLAG§

# 3  
Old 06-15-2010
great that works but in fact there is a problem

we dont want to move up records that start by "§" delimiter.
With this commandline all the records are in one line

example
Code:
§01§§NAME§§SURNAME§§DATE§§COMMENTS:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxx§§OPTION§§FLAG§
§02§§NAME§§SURNAME§§DATE§§COMMENTS:xxxxxxxxxxxxxxxxxxxxxxxxxx§§OPTION§§FLAG§

after update : all are in 1 line
Code:
§01§§NAME§§SURNAME§§DATE§§COMMENTS:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx§§OPTION§§FLAG§§02§§NAME§§SURNAME§§DATE§§COMMENTS:xxxxxxxxxxxxxxxxxxxxxxxxxx§§OPTION§§FLAG§

Last edited by vgersh99; 06-15-2010 at 01:09 PM.. Reason: code tags, please!
# 4  
Old 06-15-2010
Code:
# cat a
§IDNUM§$NAME§$SURNAME§$DATE§$COMMENTS:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxx
xxxx§$OPTION§$FLAG§
E   X   A   M    P   L   E
§IDNUM§$NAME§$SURNAME§$DATE§$COMMENTS:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxx
xxxx§$OPTION§$FLAG§

Code:
# sed '/§/,/FLAG/{;/§/!d;}' a | sed '/§/{N; s/\n//}'
§IDNUM§$NAME§$SURNAME§$DATE§$COMMENTS:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx§$OPTION§$FLAG§
E   X   A   M    P   L   E
§IDNUM§$NAME§$SURNAME§$DATE§$COMMENTS:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx§$OPTION§$FLAG§

# 5  
Old 06-16-2010
i have a problem can you help me ?

Code:
sed '/\247/,/FLAG/{;/\247/!d;}' head.temp  | sed '/\247/{N; s/\n//}'
sed: command garbled: /§/{N; s/\n//}

thanks a lot
# 6  
Old 06-16-2010
awk alternative:
Code:
awk -F§ 'NF>1{printf $0 (($NF=="")?RS:"")}' infile2

# 7  
Old 06-16-2010
Quote:
Originally Posted by ade05fr
i have a problem can you help me ?

Code:
sed '/\247/,/FLAG/{;/\247/!d;}' head.temp  | sed '/\247/{N; s/\n//}'
sed: command garbled: /§/{N; s/\n//}

thanks a lot
Try like this Smilie

Code:
 
sed '/\247/,/FLAG/{;/\247/!d;}' head.temp  | sed '/\247/{N; s/\n//;}'
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Script to compare files in 2 folders and delete the large file

Hello, my first thread here. I've been searching and fiddling around for about a week and I cannot find a solution.:confused: I have been converting all of my home videos to HEVC and sometimes the files end up smaller and sometimes they don't. I am currently comparing all the video files... (5 Replies)
Discussion started by: Josh52180
5 Replies

2. UNIX for Beginners Questions & Answers

sed awk: split a large file to unique file names

Dear Users, Appreciate your help if you could help me with splitting a large file > 1 million lines with sed or awk. below is the text in the file input file.txt scaffold1 928 929 C/T + scaffold1 942 943 G/C + scaffold1 959 960 C/T +... (6 Replies)
Discussion started by: kapr0001
6 Replies

3. Shell Programming and Scripting

Sed: Splitting A large File into smaller files based on recursive Regular Expression match

I will simplify the explaination a bit, I need to parse through a 87m file - I have a single text file in the form of : <NAME>house........ SOMETEXT SOMETEXT SOMETEXT . . . . </script> MORETEXT MORETEXT . . . (6 Replies)
Discussion started by: sumguy
6 Replies

4. Shell Programming and Scripting

sed and awk not working on a large record file

Hi All, I have a very large single record file. abc;date||bcd;efg|......... pqr;stu||record_count;date when i do wc -l on this file it gives me "0" records, coz of missing line feed. my problem is there is an extra pipe that is coming at the end of this record like... (6 Replies)
Discussion started by: Gurkamal83
6 Replies

5. Shell Programming and Scripting

Need to delete large set of files (i.e) close to 100K from a directory based on the input file

Hi all, I need a script to delete a large set of files from a directory under / based on an input file and want to redirect errors into separate file. I have already prepared a list of files in the input file. Kndly help me. Thanks, Prash (36 Replies)
Discussion started by: prash358
36 Replies

6. Shell Programming and Scripting

How to delete 1 record in large file!

Hi All, I'm a newbie here, I'm just wondering on how to delete a single record in a large file in unix. ex. file1.txt is 1000 records nikki1 nikki2 nikki3 what i want to do is delete the nikki2 record in file1.txt. is it possible? Please advise, Thanks, (3 Replies)
Discussion started by: nikki1200
3 Replies

7. UNIX for Dummies Questions & Answers

Delete large number of columns rom file

Hi, I have a data file that contains 61 columns. I want to delete all the columns except columns, 3,6 and 8. The columns are tab de-limited. How would I achieve this on the terminal? Thanks (2 Replies)
Discussion started by: lost.identity
2 Replies

8. Shell Programming and Scripting

Updating a line in a large csv file, with sed/awk?

I have an extremely large csv file that I need to search the second field, and upon matches update the last field... I can pull the line with awk.. but apparently you cant use awk to directly update the file? So im curious if I can use sed to do this... The good news is the field I want to... (5 Replies)
Discussion started by: trey85stang
5 Replies

9. UNIX for Advanced & Expert Users

how to delete a large file in sco openserver 6

How does one delete a large file say file.txt of size 4331456 blocks (2217697280 bytes) in a directory in sco openserver 6 ... Upon executing the command # rm file.txt the error message is as below ... rm: file.txt non-existent: Value too large for defined data type (error 79) any help... (1 Reply)
Discussion started by: jksah
1 Replies

10. Shell Programming and Scripting

How to delete rows by RowNumber from a Large text file

Friends, I have text file with 700,000 rows. Once I load this file to our database via our cutom process, it logs the row number for rejected rows. How do I delete rows from a Large text file based on the Row Number? Thanks, Prashant (8 Replies)
Discussion started by: ppat7046
8 Replies
Login or Register to Ask a Question