Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

remove spaces and lines that start with --

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting remove spaces and lines that start with --
# 1  
Old 01-15-2012
remove spaces and lines that start with --
Is it possible to remove empty lines between >humid-sets (bold) and also humidset that start with -- (for ex: > humid3 | () : | (+) )
Thanx in advance

Note: The humid sets will be in thousands and lines will be more than 100 thousand.

input
Code:
> humid1 | () : | (+)
ababshdjbshjbjhsbfsfbksfbs----------------nb sbdnf sdbf ------
>sdfsgffsgs  | () : | ||
ababbabafgsfzuyhjkvsmzbcv hjszfmcd----------fvxcv cx
>hubjbmj  | () : | (+)  | () : | (+)
ajdfgcbshjdcgv rsghrjcfvn rhjsgfcv hjs-------afdcbhjsdbc sjhvc sd

> humid2 | () : | (+)
ababshdjbshjbjhsbfsfbksfbs----------------nb sbdnf sdbf ------
>sdfsgffsgs  | () : | ||
ababbabafgsfzuyhjkvsmzbcv hjszfmcd----------fvxcv cx
>hubjbmj  | () : | (+)  | () : | (+)
ajdfgcbshjdcgv rsghrjcfvn rhjsgfcv hjs-------afdcbhjsdbc sjhvc sd

> humid3 | () : | (+)
---------------ababshdjbshjbjhsbfsfbksfbs----------------nb sbdnf sdbf ------
>sdfsgffsgs  | () : | ||
ababbabafgsfzuyhjkvsmzbcv hjszfmcd----------fvxcv cx
>hubjbmj  | () : | (+)  | () : | (+)
ajdfgcbshjdcgv rsghrjcfvn rhjsgfcv hjs-------afdcbhjsdbc sjhvc sd

output
Code:
> humid1 | () : | (+)
ababshdjbshjbjhsbfsfbksfbs----------------nb sbdnf sdbf ------
>sdfsgffsgs  | () : | ||
ababbabafgsfzuyhjkvsmzbcv hjszfmcd----------fvxcv cx
>hubjbmj  | () : | (+)  | () : | (+)
ajdfgcbshjdcgv rsghrjcfvn rhjsgfcv hjs-------afdcbhjsdbc sjhvc sd
> humid2 | () : | (+)
ababshdjbshjbjhsbfsfbksfbs----------------nb sbdnf sdbf ------
>sdfsgffsgs  | () : | ||
ababbabafgsfzuyhjkvsmzbcv hjszfmcd----------fvxcv cx
>hubjbmj  | () : | (+)  | () : | (+)
ajdfgcbshjdcgv rsghrjcfvn rhjsgfcv hjs-------afdcbhjsdbc sjhvc sd

# 2  
Old 01-15-2012
grep -v -E '^$|^--' yourfile.txt > /tmp/output.txt
# 3  
Old 01-15-2012
your output
Code:
> humid1 | () : | (+)
ababshdjbshjbjhsbfsfbksfbs----------------nb sbdnf sdbf ------
>sdfsgffsgs  | () : | ||
ababbabafgsfzuyhjkvsmzbcv hjszfmcd----------fvxcv cx
>hubjbmj  | () : | (+)  | () : | (+)
ajdfgcbshjdcgv rsghrjcfvn rhjsgfcv hjs-------afdcbhjsdbc sjhvc sd
> humid2 | () : | (+)
ababshdjbshjbjhsbfsfbksfbs----------------nb sbdnf sdbf ------
>sdfsgffsgs  | () : | ||
ababbabafgsfzuyhjkvsmzbcv hjszfmcd----------fvxcv cx
>hubjbmj  | () : | (+)  | () : | (+)
ajdfgcbshjdcgv rsghrjcfvn rhjsgfcv hjs-------afdcbhjsdbc sjhvc sd
> humid3 | () : | (+)
>sdfsgffsgs  | () : | ||
ababbabafgsfzuyhjkvsmzbcv hjszfmcd----------fvxcv cx
>hubjbmj  | () : | (+)  | () : | (+)
ajdfgcbshjdcgv rsghrjcfvn rhjsgfcv hjs-------afdcbhjsdbc sjhvc sd

Thanx but it should remove the whole humdiset3 until it start with other humidset (it starts with >humid). So out should be like this
Code:
> humid1 | () : | (+) 
ababshdjbshjbjhsbfsfbksfbs----------------nb sbdnf sdbf ------ 
>sdfsgffsgs  | () : | || 
ababbabafgsfzuyhjkvsmzbcv hjszfmcd----------fvxcv cx
>hubjbmj  | () : | (+)  | () : | (+) 
ajdfgcbshjdcgv rsghrjcfvn rhjsgfcv hjs-------afdcbhjsdbc sjhvc sd 
> humid2 | () : | (+) 
ababshdjbshjbjhsbfsfbksfbs----------------nb sbdnf sdbf ------ 
>sdfsgffsgs  | () : | || 
ababbabafgsfzuyhjkvsmzbcv hjszfmcd----------fvxcv cx 
>hubjbmj  | () : | (+)  | () : | (+) 
ajdfgcbshjdcgv rsghrjcfvn rhjsgfcv hjs-------afdcbhjsdbc sjhvc sd

# 4  
Old 01-15-2012
Something like this?
Code:
awk '!/\n--/' ORS='\n' RS='\n\n' humid.txt

This User Gave Thanks to mirni For This Post:
quincyjones
# 5  
Old 01-16-2012
This should leave out the entire third record with the sample given as well as remove newlines..
Code:
awk '$8!~/^--/' RS= infile

or
Code:
awk '!/\n--/' RS= infile

Last edited by Scrutinizer; 01-18-2012 at 07:02 AM..
# 6  
Old 01-16-2012
Something like this?

Code:
perl -ne '!(/^\s+$/||/^--/)&&print' inputfile

# 7  
Old 01-16-2012
Quote:
Originally Posted by mirni
Something like this?
Code:
awk '!/\n--/' ORS='\n' RS='\n\n' humid.txt

Hi, AFAIK only gawk and mawk allow RS to have more than one character. The POSIX specification states that only the first character of the string value of RS should be used as input record separator, so it will break with the other awks.
This User Gave Thanks to Scrutinizer For This Post:
mirni
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

How to print lines from a files with specific start and end patterns and pick only the last lines?

Hi, I need to print lines which are matching with start pattern "SELECT" and END PATTERN ";" and only select the last "select" statement including the ";" . I have attached sample input file and the desired input should be as: INPUT FORMAT: SELECT ABCD, DEFGH, DFGHJ, JKLMN, AXCVB,... (5 Replies)
Discussion started by: nani2019
5 Replies

2. Shell Programming and Scripting

awk to remove lines that do not start with digit and combine line or lines

I have been searching and trying to come up with an awk that will perform the following on a converted text file (original is a pdf). 1. Since the first two lines are (begin with) text they are removed 2. if $1 is a number then all text is merged (combined) into one line until the next... (3 Replies)
Discussion started by: cmccabe
3 Replies

3. Shell Programming and Scripting

Remove duplicate lines after ignoring case and spaces between

Oracle Linux 6.5 $ cat someStrings.txt GRANT select on MANHPRD.S_PROD_INT TO OR_PHIL; GRANT select on MANHPRD.S_PROD_INT TO OR_PHIL; GRANT select on SCOTT.emp to JOHN; grant select on scott.emp to john; grant select on scott.dept to hr;If you ignore the case and the empty space between the... (6 Replies)
Discussion started by: kraljic
6 Replies

4. Shell Programming and Scripting

Remove all lines which start with #

Oracle Linux 6.4/Bash I have a file like below. I want to remove all lines which start with # character. Can I do this vi editor ? If not , which other utility can I use for this ? # This is a test script CUSER=`id |cut -d"(" -f2 | cut -d ")" -f1` # Some text CDATE=`date +%y%m%d` ## get... (3 Replies)
Discussion started by: kraljic
3 Replies

5. Shell Programming and Scripting

Remove certain lines from file based on start of line except beginning and ending

Hi, I have multiple large files which consist of the below format: I am trying to write an awk or sed script to remove all occurrences of the 00 record except the first and remove all of the 80 records except the last one. Any help would be greatly appreciated. (10 Replies)
Discussion started by: nwalsh88
10 Replies

6. Shell Programming and Scripting

Remove lines between the start string and end string including start and end string Python

Hi, I am trying to remove lines once a string is found till another string is found including the start string and end string. I want to basically grab all the lines starting with color (closing bracket). PS: The line after the closing bracket for color could be anything (currently 'more').... (1 Reply)
Discussion started by: Dabheeruz
1 Replies

7. Shell Programming and Scripting

Remove spaces from start of file names

Hi, I have a directory with the following file names 01 - abc hyn 02-def 03-ghi.dir 04 - jhu.dir abc1 kil def bil The last two file names abc1 starts with one space and def starts with double space. I want these files in my directory to be renamed as ABC HYN DEF GHI.dir... (6 Replies)
Discussion started by: jacobs.smith
6 Replies

8. Shell Programming and Scripting

Reform Lines in File without blank lines and spaces

Hello All, I have a file with data as below. Each line consists of 21 fields. I am not able to load them back to the database. 50733339,"834","834 ","005010X279A1","N","Y","007977163","0001 ",30,"2110D ","EB ","EB007 ","2 ","Conditional Required Data Element Miss ing... (3 Replies)
Discussion started by: Praveenkulkarni
3 Replies

9. Shell Programming and Scripting

Need to remove lines that start with an IP address

Hi, I keep having to remove lines have an IP address as the second field from my awstats logs, as it makes the processing fail. Rather than do it individually each time (once or twice a week) it fails, I'd like to remove any lines from the file that have 3 digits and then a dot as the start... (1 Reply)
Discussion started by: sfisk
1 Replies

10. Shell Programming and Scripting

How to remove plank spaces at the end of lines

Hello friends, I want to remove blank spaces at the end of lines. I use sed command to do this but it is not working correctly. sed ‘s/ $//’ file_name Can some body tell me what is the proper way to remove blank spaces at the end of a limes. Thanks, Mahesh Fernando. (3 Replies)
Discussion started by: maheshsri
3 Replies
Login or Register to Ask a Question