remove a specific line in a LARGE file


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting remove a specific line in a LARGE file
# 1  
Old 09-06-2009
remove a specific line in a LARGE file

Hi guys,
i have a really big file, and i want to remove a specific line.
Code:
sed -i '5d' file

This doesn't really work, it takes a lot of time...
The whole script is supposed to remove every word containing less than 5 characters and currently looks like this:
Code:
#!/bin/bash

line="1"
wholelines="38867934"
while [ $line -lt $wholelines ]; do
length=$(expr length `sed $line'q;d' list`)
if [ $length -lt "5" ]; then
sed -i $line' d' list
wholelines=$((wholelines-1))
else
line=$(($line+1))
fi
done

so is there a way to remove a specific line in less time than sed needs on large files?


Uli
# 2  
Old 09-06-2009
Quote:
Originally Posted by blubbiblubbkekz
The whole script is supposed to remove every word containing less than 5 characters ...
Hi Uli,

I presume you mean every line containing less than 5 characters (i.e. 4 or fewer), since that is what you script does.
If that is the case then this would be more efficient:

Code:
awk 'length>4' list > list2

The shell equivalent would be:
Code:
while read line; do
   if (( ${#line}>4 )); then
     echo $line
   fi
done<list>list2


Last edited by Scrutinizer; 09-06-2009 at 09:44 AM..
# 3  
Old 09-06-2009
awesome, thanks, this is what i wanted =)
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Remove dupes in a large file

I have a large file 1.5 gb and want to sort the file. I used the following AWK script to do the job !x++ The script works but it is very slow and takes over an hour to do the job. I suspect this is because the file is not sorted. Any solution to speed up the AWk script or a Perl script would... (4 Replies)
Discussion started by: gimley
4 Replies

2. Shell Programming and Scripting

Bash : Checking Large file for specific lines

Morning .. I have a file with approximately 1000 lines. I want to check that the file contains, for example, 100 lines. Something like whats given below is ugly. And even if I create a function I have to call it 100 times. I may need to look through multiple files at times. Is there a... (4 Replies)
Discussion started by: sumguy
4 Replies

3. Shell Programming and Scripting

Remove every line with specific string, and also the one above and below it

I would like to identify every line with a specific string, in this case: "Mamma". I would like to remove that line, and also the line above it and below it. So the below Where are all amazing Flats Look At The Great Big White Hey There Hot Mamma You Are So hot Baby I wish You were Mine... (5 Replies)
Discussion started by: phpchick
5 Replies

4. Shell Programming and Scripting

Remove line with specific character

HI Input :- Aog:0rt__dev_8 LAAXU24 vs.3 LAA40l0 ** LAAXU241 ** Output :- Aog:0rt__dev_8 LAAXU24 vs.3 Delete the line with ** (3 Replies)
Discussion started by: pareshkp
3 Replies

5. Shell Programming and Scripting

How to remove a subset of data from a large dataset based on values on one line

Hello. I was wondering if anyone could help. I have a file containing a large table in the format: marker1 marker2 marker3 marker4 position1 position2 position3 position4 genotype1 genotype2 genotype3 genotype4 with marker being a name, position a numeric... (2 Replies)
Discussion started by: davegen
2 Replies

6. Shell Programming and Scripting

remove unwanted specific line range

Hello everyone...I have large txt file and I would like to remove unwanted specific line. My data is like this: So I would like to remove from line below No. until line reassambled like this: Thanks... (4 Replies)
Discussion started by: taxi
4 Replies

7. Shell Programming and Scripting

Searching a specific line in a large file

Hey All Can any one please suggest the procedure to search a part of line in a very large file in which log entries are entered with very high speed. i have trued with grep and egrep grep 'text text text' <file-name> egrep 'text text text' <file-name> here 'text text text' is... (4 Replies)
Discussion started by: NIMISH AGARWAL
4 Replies

8. Shell Programming and Scripting

Count specific character(s) very large file

I'm trying to count the number of 2 specific characters in a very large file. I'd like to avoid using gsub because its taking too long. I was thinking something like: awk '-F' { t += NF - 1 } END {print t}' infile > outfile which isn't working Any ideas would be great. (3 Replies)
Discussion started by: dcfargo
3 Replies

9. UNIX for Dummies Questions & Answers

Help with selecting specific lines in a large file

Hello, I need to select the 3 lines above as well as below a search string, including the search string. I have been trying various combinations using sed command without any success. Can anuone help please. Thanking (2 Replies)
Discussion started by: tansha
2 Replies

10. Shell Programming and Scripting

Remove Line that contains specific string

I am looking for a way to remove any line in a text file that contains the string "Mac address". I guess you would grep and sed, but I am not sure how to do this. Thanks for you help. (3 Replies)
Discussion started by: CBarraford
3 Replies
Login or Register to Ask a Question