Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Delete rows from big file
Top Forums Shell Programming and Scripting Delete rows from big file Post 302645173 by Tibbeche on Wednesday 23rd of May 2012 09:03:08 AM
Old 05-23-2012
Delete rows from big file
Hi all,
I have a big file (about 6 millions rows) and I have to delete same occurrences, stored in a small file (about 9000 rews). I have tried this:

Code:
while read line
do
  grep -v $line big_file > ok_file.tmp
  mv ok_file.tmp big_file 
done < small_file

It works, but is very slow.
How can I do the same thing with less time?

PS I try sed -i but on AIX dosen't work.

Thanks in advance

---------- Post updated at 03:03 PM ---------- Previous update was at 11:44 AM ----------

Just for information,
I resolved my problem with perl script......very fast (2 minutes instead of 2 hours).
Last edited by zaxxon; 05-23-2012 at 10:10 AM.. Reason: code tags
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Delete repeated rows from a file

Hi everybody: Could anybody tell me how I can delete repeated rows from a file?, this is, for exemple I have a file like this: 0.490 958.73 281.85 6.67985 0.002481 0.490 954.833 283.991 8.73019 0.002471 0.590 950.504 286.241 6.61451 0.002461 0.690 939.323 286.112 6.16451 0.00246 0.790... (8 Replies)
Discussion started by: tonet
8 Replies

2. Shell Programming and Scripting

How to delete particular rows from a file

Hi I have a file having 1000 rows. Now I would like to remove 10 rows from it. Plz give me the script. Eg: input file like 4 1 4500.0 1 5 1 1.0 30 6 1 1.0 4500 7 1 4.0 730 7 2 500000.0 730 8 1 785460.0 45 8 7 94255.0 30 9 1 31800.0 30 9 4 36000.0 30 10 1 15000.0 30... (5 Replies)
Discussion started by: suresh3566
5 Replies

3. Shell Programming and Scripting

how to delete duplicate rows in a file

I have a file content like below. "0000000","ABLNCYI","BOTH",1049,2058,"XYZ","5711002","","Y","","","","","","","","" "0000000","ABLNCYI","BOTH",1049,2058,"XYZ","5711002","","Y","","","","","","","","" "0000000","ABLNCYI","BOTH",1049,2058,"XYZ","5711002","","Y","","","","","","","",""... (5 Replies)
Discussion started by: vamshikrishnab
5 Replies

4. Shell Programming and Scripting

[HELP] - Delete rows on a CSV file

Hello to all members, I am very new in unix stuff (shell scripting), but a want to learn a lot. I am a ex windows user but now i am absolutely Linux super user... :D So i am tryng to made a function to do this: I have two csv files only with numbers, the first one a have: 1 2 3 4 5... (6 Replies)
Discussion started by: Sadarrab
6 Replies

5. Shell Programming and Scripting

delete rows in a file based on the rows of another file

I need to delete rows based on the number of lines in a different file, I have a piece of code with me working but when I merge with my C application, it doesnt work. sed '1,'\"`wc -l < /tmp/fileyyyy`\"'d' /tmp/fileA > /tmp/filexxxx Can anyone give me an alternate solution for the above (2 Replies)
Discussion started by: Muthuraj K
2 Replies

6. UNIX for Advanced & Expert Users

Delete rows from a file...!!

Say i have a file with X rows and Y columns....i see that in some of the rows,some columns are blank (no value set)...i wish to delete such rows....how can it be done? e.g 181766 100 2009-06-04 184443 2009-06-04 10962 151 2009-06-04 161 2009-06-04... (7 Replies)
Discussion started by: ak835
7 Replies

7. Shell Programming and Scripting

Delete rows in text file

Hi I do have a text file with 1000's of lines with 1 row and column with a specific pattern. 1102 1 1 1 1 1234 1 1 1 1 1009 1 1 1 1 1056 1 (3 Replies)
Discussion started by: Lucky Ali
3 Replies

8. UNIX for Advanced & Expert Users

Delete first 100 lines from a BIG File

Hi, I need a unix command to delete first n (say 100) lines from a log file. I need to delete some lines from the file without using any temporary file. I found sed -i is an useful command for this but its not supported in my environment( AIX 6.1 ). File size is approx 100MB. Thanks in... (18 Replies)
Discussion started by: unohu
18 Replies

9. UNIX for Dummies Questions & Answers

Delete records from a big file based on some condition

Hi, To load a big file in a table,I have a make sure that all rows in the file has same number of the columns . So in my file if I am getting any rows which have columns not equal to 6 , I need to delete it . Delimiter is space and columns are optionally enclosed by "". This can be ... (1 Reply)
Discussion started by: hemantraijain
1 Replies

10. Shell Programming and Scripting

Script to delete rows in a file

Hi All, I am new to UNIX . Please help me in writing code to delete all records from the file where all columns after cloumn 5 in file is either 0, #MI or NULL. Initial 5 columns are string e.g. "alsod" "1FEV2" "wjwroe" " wsse" "hd3" 1 2 34 #Mi "malasl" "wses" "trwwwe" " wsse" "hd3" 1 2 0... (4 Replies)
Discussion started by: alok2082
4 Replies

LEARN ABOUT DEBIAN

gchem3d

GCHEM3D(1)						       gnome-chemistry-utils							GCHEM3D(1)

NAME

       gchem3d - a small chemical viewer application

SYNOPSIS

       gchem3d [OPTION(S)...] [FILE...]

DESCRIPTION

       gchem3d is a small chemical viewer application, which can show several chemical file formats.

OPTIONS

       The following options are accepted:

       -b COLOR, --bgcolor=COLOR
	   Use the given color as background color.  COLOR can be one of "black" (default), "white", "#rrggbb" (don't forget to escape the "#"
	   character in the shell).

       -d MODEL, --display3d=MODEL
	   Choose how molecules are displayed.	MODEL can be one of "BallnStick" (default), "SpaceFill".

       -?, --help
	   Show application help options.

       --help-all, --help-*
	   Print all or just a group of help options. These options are not documented here. Instead see gtk-options(7) and gnome-options(7).

       -v, --version
	   Print gchem3d version information.

SEE ALSO

       gnome-options(7), gtk-options(7)

AUTHORS

       Jean Brefort <jean.brefort@normalesup.org>
	   Program author.

       Daniel Leidert <daniel.leidert@wgdd.de>
	   Manpage author.

COPYRIGHT

       Copyright (C) 2002-2007 Jean Brefort
       Copyright (C) 2004-2007 Daniel Leidert

       Permission is granted to copy, distribute and/or modify this document under the terms of the GNU General Public License, Version 2 or any
       later version published by the Free Software Foundation.

gcu 0.12				      $Date: 2009-03-19 10:53:47 +0100 (jeu. 19 mars 2009) $					GCHEM3D(1)
All times are GMT -4. The time now is 02:29 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy