Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

How to delete rows by RowNumber from a Large text file

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting How to delete rows by RowNumber from a Large text file
# 8  
Old 05-15-2009
sorry - fat fingers:
Code:
awk 'NR==FNR{a[$1]++;next} !(FNR in a)' deletelistfile largetextfile > resultfile

# 9  
Old 05-15-2009
Quote:
Originally Posted by vgersh99
Actually.... I read somewhere (c.l.a most likely) that the 'lookup' is generally slower than the direct 'value fetch and comparison', but I don't quite remember for what version of awk.
Made some tests with an array of 9999 numbers.

GNU Awk 3.1.5, Copyright (C) 1989, 1991-2005 Free Software Foundation.
When k is in a then (k in a) is about as quick as a[k].
When k is not in a then (k in a) is quicker, it takes about 85% of the time of a[k].

mawk 1.3.3 Nov 1996, Copyright (C) Michael D. Brennan
a[k] is quicker, it takes about 88% of the time of (k in a) regardless of whether k is in a or not.


Overall, mawk takes only about 50% of the time of GNU awk.
On the other hand I have been able to consistently crash mawk with a program easily handled by GNU awk.
Last edited by colemar; 05-15-2009 at 06:20 PM..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Dedup a large file(30M rows)

Hi, I have a large file with number of records in there. I need some help to find only first row based on a key and ignore other rows with the same key. I tried few things but file is huge(30 million rows). So need some solution that is very efficient. e.g Junk|Apple|7|Random|data|here...... (2 Replies)
Discussion started by: ran123
2 Replies

2. Shell Programming and Scripting

How to delete 1 record in large file!

Hi All, I'm a newbie here, I'm just wondering on how to delete a single record in a large file in unix. ex. file1.txt is 1000 records nikki1 nikki2 nikki3 what i want to do is delete the nikki2 record in file1.txt. is it possible? Please advise, Thanks, (3 Replies)
Discussion started by: nikki1200
3 Replies

3. UNIX for Dummies Questions & Answers

Delete large number of columns rom file

Hi, I have a data file that contains 61 columns. I want to delete all the columns except columns, 3,6 and 8. The columns are tab de-limited. How would I achieve this on the terminal? Thanks (2 Replies)
Discussion started by: lost.identity
2 Replies

4. UNIX for Dummies Questions & Answers

Delete all rows that contain a specific string (text)

Hi, I have a text file and I want to delete all rows that contain a particular string of characters. How do I go about doing that? Thanks! (9 Replies)
Discussion started by: evelibertine
9 Replies

5. Shell Programming and Scripting

Delete rows in text file

Hi I do have a text file with 1000's of lines with 1 row and column with a specific pattern. 1102 1 1 1 1 1234 1 1 1 1 1009 1 1 1 1 1056 1 (3 Replies)
Discussion started by: Lucky Ali
3 Replies

6. Shell Programming and Scripting

Large file - columns into rows etc

I have done a couple of searches on this and have found many threads but I don't think I've found one that is useful to me - probably because I have very basic comprehension of perl and beginners shell so trying to manipulate a script already posted maybe beyond my capabilities.... Anyway - I... (26 Replies)
Discussion started by: Myrona
26 Replies

7. Shell Programming and Scripting

how to delete \n in a large file with sed

Hello i have a big file with a specific format and delimiter is "§" : §field1§$field2§$field3§$field4§$field5§$field6§$field§ in this file we have a field which are very long (more than 20000 chars !!!!) so through vi i cant manipulate them. despite this i managed to suppress lines that... (11 Replies)
Discussion started by: ade05fr
11 Replies

8. Shell Programming and Scripting

delete rows in a file based on the rows of another file

I need to delete rows based on the number of lines in a different file, I have a piece of code with me working but when I merge with my C application, it doesnt work. sed '1,'\"`wc -l < /tmp/fileyyyy`\"'d' /tmp/fileA > /tmp/filexxxx Can anyone give me an alternate solution for the above (2 Replies)
Discussion started by: Muthuraj K
2 Replies

9. Shell Programming and Scripting

Deleting specific rows in large files having rows greater than 100000

Hi Guys, I need help in modifying a large text file containing more than 1-2 lakh rows of data using unix commands. I am quite new to the unix language the text file contains data in a pipe delimited format sdfsdfs sdfsdfsd START_ROW sdfsd|sdfsdfsd|sdfsdfasdf|sdfsadf|sdfasdf... (9 Replies)
Discussion started by: manish2009
9 Replies

10. UNIX for Advanced & Expert Users

how to delete a large file in sco openserver 6

How does one delete a large file say file.txt of size 4331456 blocks (2217697280 bytes) in a directory in sco openserver 6 ... Upon executing the command # rm file.txt the error message is as below ... rm: file.txt non-existent: Value too large for defined data type (error 79) any help... (1 Reply)
Discussion started by: jksah
1 Replies
Login or Register to Ask a Question