Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

How to delete rows by RowNumber from a Large text file

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting How to delete rows by RowNumber from a Large text file
# 1  
Old 05-14-2009
How to delete rows by RowNumber from a Large text file
Friends,

I have text file with 700,000 rows.
Once I load this file to our database via our cutom process, it logs the row number for rejected rows.

How do I delete rows from a Large text file based on the Row Number?

Thanks,
Prashant
# 2  
Old 05-14-2009
Code:
sed '10d' filename > newfilename

delete line # 10.
# 3  
Old 05-14-2009
Quote:
Originally Posted by ppat7046
How do I delete rows from a Large text file based on the Row Number?
Assuming that deletelistfile is a file listing the row numbers to delete:

Code:
awk 'NR==FNR{a[$1]=1;next}{if(!a[FNR])print$0}' deletelistfile largetextfile > resultfile

# 4  
Old 05-14-2009
Quote:
Originally Posted by colemar
Assuming that deletelistfile is a file listing the row numbers to delete:

Code:
awk 'NR==FNR{a[$1]=1;next}{if(!a[FNR])print$0}' deletelistfile largetextfile > resultfile

or better yet:
Code:
awk 'NR==FNR{a[$1]++;next} !(FNR in a)}' deletelistfile largetextfile > resultfile

# 5  
Old 05-14-2009
Quote:
Originally Posted by vgersh99
Code:
awk 'NR==FNR{a[$1]++;next} !(FNR in a)}' deletelistfile largetextfile > resultfile

Yes, I forgot that print$0 is the default action.
Can you tell whether !(FNR in a) is any better than !a[FNR] performancewise?
# 6  
Old 05-14-2009
Quote:
Originally Posted by colemar
Yes, I forgot that print$0 is the default action.
Can you tell whether !(FNR in a) is any better than !a[FNR] performancewise?
Actually.... I read somewhere (c.l.a most likely) that the 'lookup' is generally slower than the direct 'value fetch and comparison', but I don't quite remember for what version of awk.
Me personally I'm just more accustomed to think of 'array membership' rather than 'value fetch' - just a matter of taste I reckon.
# 7  
Old 05-15-2009
I am geeting syntax error when I run
awk 'NR==FNR{a[$1]++;next} !(FNR in a)}' deletelistfile largetextfile > resultfile

Thanks,
Prashant
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Dedup a large file(30M rows)

Hi, I have a large file with number of records in there. I need some help to find only first row based on a key and ignore other rows with the same key. I tried few things but file is huge(30 million rows). So need some solution that is very efficient. e.g Junk|Apple|7|Random|data|here...... (2 Replies)
Discussion started by: ran123
2 Replies

2. Shell Programming and Scripting

How to delete 1 record in large file!

Hi All, I'm a newbie here, I'm just wondering on how to delete a single record in a large file in unix. ex. file1.txt is 1000 records nikki1 nikki2 nikki3 what i want to do is delete the nikki2 record in file1.txt. is it possible? Please advise, Thanks, (3 Replies)
Discussion started by: nikki1200
3 Replies

3. UNIX for Dummies Questions & Answers

Delete large number of columns rom file

Hi, I have a data file that contains 61 columns. I want to delete all the columns except columns, 3,6 and 8. The columns are tab de-limited. How would I achieve this on the terminal? Thanks (2 Replies)
Discussion started by: lost.identity
2 Replies

4. UNIX for Dummies Questions & Answers

Delete all rows that contain a specific string (text)

Hi, I have a text file and I want to delete all rows that contain a particular string of characters. How do I go about doing that? Thanks! (9 Replies)
Discussion started by: evelibertine
9 Replies

5. Shell Programming and Scripting

Delete rows in text file

Hi I do have a text file with 1000's of lines with 1 row and column with a specific pattern. 1102 1 1 1 1 1234 1 1 1 1 1009 1 1 1 1 1056 1 (3 Replies)
Discussion started by: Lucky Ali
3 Replies

6. Shell Programming and Scripting

Large file - columns into rows etc

I have done a couple of searches on this and have found many threads but I don't think I've found one that is useful to me - probably because I have very basic comprehension of perl and beginners shell so trying to manipulate a script already posted maybe beyond my capabilities.... Anyway - I... (26 Replies)
Discussion started by: Myrona
26 Replies

7. Shell Programming and Scripting

how to delete \n in a large file with sed

Hello i have a big file with a specific format and delimiter is "§" : §field1§$field2§$field3§$field4§$field5§$field6§$field§ in this file we have a field which are very long (more than 20000 chars !!!!) so through vi i cant manipulate them. despite this i managed to suppress lines that... (11 Replies)
Discussion started by: ade05fr
11 Replies

8. Shell Programming and Scripting

delete rows in a file based on the rows of another file

I need to delete rows based on the number of lines in a different file, I have a piece of code with me working but when I merge with my C application, it doesnt work. sed '1,'\"`wc -l < /tmp/fileyyyy`\"'d' /tmp/fileA > /tmp/filexxxx Can anyone give me an alternate solution for the above (2 Replies)
Discussion started by: Muthuraj K
2 Replies

9. Shell Programming and Scripting

Deleting specific rows in large files having rows greater than 100000

Hi Guys, I need help in modifying a large text file containing more than 1-2 lakh rows of data using unix commands. I am quite new to the unix language the text file contains data in a pipe delimited format sdfsdfs sdfsdfsd START_ROW sdfsd|sdfsdfsd|sdfsdfasdf|sdfsadf|sdfasdf... (9 Replies)
Discussion started by: manish2009
9 Replies

10. UNIX for Advanced & Expert Users

how to delete a large file in sco openserver 6

How does one delete a large file say file.txt of size 4331456 blocks (2217697280 bytes) in a directory in sco openserver 6 ... Upon executing the command # rm file.txt the error message is as below ... rm: file.txt non-existent: Value too large for defined data type (error 79) any help... (1 Reply)
Discussion started by: jksah
1 Replies
Login or Register to Ask a Question