Sponsored Content
Full Discussion: Delete rows from big file
Top Forums Shell Programming and Scripting Delete rows from big file Post 302645173 by Tibbeche on Wednesday 23rd of May 2012 09:03:08 AM
Old 05-23-2012
Delete rows from big file

Hi all,
I have a big file (about 6 millions rows) and I have to delete same occurrences, stored in a small file (about 9000 rews). I have tried this:

Code:
while read line
do
  grep -v $line big_file > ok_file.tmp
  mv ok_file.tmp big_file 
done < small_file

It works, but is very slow.
How can I do the same thing with less time?

PS I try sed -i but on AIX dosen't work.

Thanks in advance

---------- Post updated at 03:03 PM ---------- Previous update was at 11:44 AM ----------

Just for information,
I resolved my problem with perl script......very fast (2 minutes instead of 2 hours).

Last edited by zaxxon; 05-23-2012 at 10:10 AM.. Reason: code tags
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Delete repeated rows from a file

Hi everybody: Could anybody tell me how I can delete repeated rows from a file?, this is, for exemple I have a file like this: 0.490 958.73 281.85 6.67985 0.002481 0.490 954.833 283.991 8.73019 0.002471 0.590 950.504 286.241 6.61451 0.002461 0.690 939.323 286.112 6.16451 0.00246 0.790... (8 Replies)
Discussion started by: tonet
8 Replies

2. Shell Programming and Scripting

How to delete particular rows from a file

Hi I have a file having 1000 rows. Now I would like to remove 10 rows from it. Plz give me the script. Eg: input file like 4 1 4500.0 1 5 1 1.0 30 6 1 1.0 4500 7 1 4.0 730 7 2 500000.0 730 8 1 785460.0 45 8 7 94255.0 30 9 1 31800.0 30 9 4 36000.0 30 10 1 15000.0 30... (5 Replies)
Discussion started by: suresh3566
5 Replies

3. Shell Programming and Scripting

how to delete duplicate rows in a file

I have a file content like below. "0000000","ABLNCYI","BOTH",1049,2058,"XYZ","5711002","","Y","","","","","","","","" "0000000","ABLNCYI","BOTH",1049,2058,"XYZ","5711002","","Y","","","","","","","","" "0000000","ABLNCYI","BOTH",1049,2058,"XYZ","5711002","","Y","","","","","","","",""... (5 Replies)
Discussion started by: vamshikrishnab
5 Replies

4. Shell Programming and Scripting

[HELP] - Delete rows on a CSV file

Hello to all members, I am very new in unix stuff (shell scripting), but a want to learn a lot. I am a ex windows user but now i am absolutely Linux super user... :D So i am tryng to made a function to do this: I have two csv files only with numbers, the first one a have: 1 2 3 4 5... (6 Replies)
Discussion started by: Sadarrab
6 Replies

5. Shell Programming and Scripting

delete rows in a file based on the rows of another file

I need to delete rows based on the number of lines in a different file, I have a piece of code with me working but when I merge with my C application, it doesnt work. sed '1,'\"`wc -l < /tmp/fileyyyy`\"'d' /tmp/fileA > /tmp/filexxxx Can anyone give me an alternate solution for the above (2 Replies)
Discussion started by: Muthuraj K
2 Replies

6. UNIX for Advanced & Expert Users

Delete rows from a file...!!

Say i have a file with X rows and Y columns....i see that in some of the rows,some columns are blank (no value set)...i wish to delete such rows....how can it be done? e.g 181766 100 2009-06-04 184443 2009-06-04 10962 151 2009-06-04 161 2009-06-04... (7 Replies)
Discussion started by: ak835
7 Replies

7. Shell Programming and Scripting

Delete rows in text file

Hi I do have a text file with 1000's of lines with 1 row and column with a specific pattern. 1102 1 1 1 1 1234 1 1 1 1 1009 1 1 1 1 1056 1 (3 Replies)
Discussion started by: Lucky Ali
3 Replies

8. UNIX for Advanced & Expert Users

Delete first 100 lines from a BIG File

Hi, I need a unix command to delete first n (say 100) lines from a log file. I need to delete some lines from the file without using any temporary file. I found sed -i is an useful command for this but its not supported in my environment( AIX 6.1 ). File size is approx 100MB. Thanks in... (18 Replies)
Discussion started by: unohu
18 Replies

9. UNIX for Dummies Questions & Answers

Delete records from a big file based on some condition

Hi, To load a big file in a table,I have a make sure that all rows in the file has same number of the columns . So in my file if I am getting any rows which have columns not equal to 6 , I need to delete it . Delimiter is space and columns are optionally enclosed by "". This can be ... (1 Reply)
Discussion started by: hemantraijain
1 Replies

10. Shell Programming and Scripting

Script to delete rows in a file

Hi All, I am new to UNIX . Please help me in writing code to delete all records from the file where all columns after cloumn 5 in file is either 0, #MI or NULL. Initial 5 columns are string e.g. "alsod" "1FEV2" "wjwroe" " wsse" "hd3" 1 2 34 #Mi "malasl" "wses" "trwwwe" " wsse" "hd3" 1 2 0... (4 Replies)
Discussion started by: alok2082
4 Replies
PHOTON(1)																 PHOTON(1)

NAME
photon - generates photo albums SYNOPSIS
photon [OPTIONS] [PATH] DESCRIPTION
Photon generates clean, yet configurable, static HTML photo galleries. All of the work is done from the command line, and the resulting output can easily be configured by changing the template files found in /usr/share/photon/templates. To create a new template (skin), first create a new directory: mkdir -p ${HOME}/.photon/templates/my_theme/ Then copy the contents of /usr/share/photon/templates/photonv1/ to this new directory. Photon only needs three files to build pages: index.html used to display the content of a folder image.html used to display the page for an image movie.html used to display the page for a movie If a file is missing, then photon will throw an error and will not run. HOW TO BUILD A GALLERY
Quick start $ photon -o myoutputdir /photos Use the gimp plugin $ photon --resize-plugin=gimp -v -o myoutputdir /photos Change the size for all images and the thumbnail and keep the original photo $ photon --sizelist=0,1600x1200,800x600 --thumbsize=320x240 -o myoutputdir /photos Create a gallery from your custom template (skin) $ photon -k my_theme -o myoutputdir /photos See photon help $ photon --help OPTIONS
-a | --awstats Insert Javascript for Awstats --awstats-url=URL Change default value for Awstats script url -c NAME | --comment Name of the comment file (default .comments) -d NUM | --display-columns=NUM Number of columns in index (default 3) -E | --no-exif Don't include EXIF information in HTML file -f | --force Overwrite non-image generated files (html, javascript, etc) (default no) -ff | --force --force Overwrite image files (default no) -h | --help Print this help -I | --no-index Do not generate the high level index.html -J | --no-javascript Do not use javascript (no shortcuts, ... ) -k | --skin=THEME Skin selector (default photonv1) --print-skins Print out available skins (templates) -l NUM | --display-lines=NUM Number of lines in index (default 5) -m | --movie Movie mode, include link to movie (*.mov) -o NAME | --output-directory=NAME Name of the output directory -s LIST | --sizelist=LIST Image sizes (default 0,1024x768,800x600,640x480). Zero, which is the default, uses the original resolution. W x H specifies the width and height of the image in pixels. -t SIZE | --thumbsize=SIZE Size of thumbnails (default 160x120) -v | --verbose Verbosely print information -V | --version Print program version number -z | --compress Compress the original image if selected (default copy the image) --exif-bordercolor=COLOR Exif window border color (default #008000) --exif-bgcolor=COLOR Exif window background color (default #f0fff0) --exif-fgcolor=COLOR Exif window text color (default black) --body-bgcolor=COLOR Body background color (default #ccccff) --img-bgcolor=COLOR Image background color (default white) --exclude=PATTERN Exclude files matching PATTERN --resize-plugin=PLUG Where PLUG is the program used to create thumbnails internal: fastest method (default) gimp: use Gimp>1.x (better quality) --resize-quality-low=NUM Quality for small image. 0 (bad) and 100 (good) --resize-quality-high=NUM Quality for big image 0 (bad) and 100 (good) --gimp-program=PROG use PROG for gimp SHORTCUTS WHILE VIEWING AN IMAGE
n/SPACE Go to the next image (with the same resolution) p/BACKSPACE Go to the previous image (with the same resolution) s Start/Stop the slideshow +/- Increase/Decrease by one second the slideshow's period h Show shortcut and help i Show Exif information z Change to the higher resolution [NOT YET IMPLEMENTED] SEE ALSO
gimp(1) AUTHOR
Written by Luc Saillard, <luc@saillard.org> Main web site: http://www.saillard.org/programs_and_patches/photon/ This manual page was written by Kevin Coyner <kcoyner@debian.org> for the Debian project but may be used by others. 12/16/2006 PHOTON(1)
All times are GMT -4. The time now is 09:29 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy