Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Shell script for searching a record,copy to a file and then delete it

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Shell script for searching a record,copy to a file and then delete it
# 1  
Old 04-19-2010
Shell script for searching a record,copy to a file and then delete it
Hi,

I have a requirement in hand:

I have a file with millions of records say file 1.I have another file, say file 2 which has 2000 records in it. The requirement is to read file2 , and remove the read record from file 1 and move i to a seperate file, file 3.
For eg: Read file 2, get the first record (eg: record id as 200) , read file 1, search for record id 200 and copy it to file 3, remove 200 from file 1.

Can some one help me with a flexible shell script which acheives the same faster. I tried using compare command but its taking a lot of time.Smilie

Any help would be much appreciated..Smilie
# 2  
Old 04-19-2010
What you have tried for your requirement?

Give me your input files and output file with some sample data.
# 3  
Old 04-19-2010
I tried with basic grep -x,-f,-v and appended to file 3. Just wanted to know if there is any better approach. I am attaching a sample of the data below:

file 1:
Code:
"T25","Insufficient Consumption Data to Calculate AQ","BUS",242900,139588342,2148,
"T25","Insufficient Consumption Data to Calculate AQ","SCP",1089204310,159690484,10813,
"T25","Insufficient Consumption Data to Calculate AQ","BUS",269707,36302598,20716,
"T25","Insufficient Consumption Data to Calculate AQ","BUS",1089211307,36203698,593,
"T25","Insufficient Consumption Data to Calculate AQ","BUS",1089218400,36203538,8737,
"T25","Insufficient Consumption Data to Calculate AQ","SOP",1089224204,178221566,551,
"T25","Insufficient Consumption Data to Calculate AQ","BUS",272909,173561749,7875,
"T25","Insufficient Consumption Data to Calculate AQ","BUS",1089227300,36205098,6863,
"T25","Insufficient Consumption Data to Calculate AQ","BUS",298108,36650298,311,
"T25","Insufficient Consumption Data to Calculate AQ","TCG",1089229708,106656758,4970,

file 2:
Code:
152708
218005
242900
243610
269707
272909
298108

Hope this helps!!
Last edited by Franklin52; 04-19-2010 at 09:34 AM.. Reason: Please use code tags!
# 4  
Old 04-19-2010
If you have given your expected output, it would be good.
Ok, anyway, check whether the following helps.


Code:
grep -f file2 file1
"T25","Insufficient Consumption Data to Calculate AQ","BUS",242900,139588342,2148,
"T25","Insufficient Consumption Data to Calculate AQ","BUS",269707,36302598,20716,
"T25","Insufficient Consumption Data to Calculate AQ","BUS",272909,173561749,7875,
"T25","Insufficient Consumption Data to Calculate AQ","BUS",298108,36650298,311,

Move the matching to file3 as you asked for.

Code:
grep -vf file2 file1
"T25","Insufficient Consumption Data to Calculate AQ","SCP",1089204310,159690484,10813,
"T25","Insufficient Consumption Data to Calculate AQ","BUS",1089211307,36203698,593,
"T25","Insufficient Consumption Data to Calculate AQ","BUS",1089218400,36203538,8737,
"T25","Insufficient Consumption Data to Calculate AQ","SOP",1089224204,178221566,551,
"T25","Insufficient Consumption Data to Calculate AQ","BUS",1089227300,36205098,6863,
"T25","Insufficient Consumption Data to Calculate AQ","TCG",1089229708,106656758,4970,

It displays only the record which is not there in the file2. So redirect it to file4, which you asked for.
# 5  
Old 04-19-2010
Thts excellent mate..Looks like it will work. I tried with a sample file.Only doubt is on the turn around time to deal with millions of records..lemme chk it out.. Thnx for this again!!
# 6  
Old 06-16-2010
Hi,I have been using the resolution you had given last time but it doesnt seem to work with huge files. I wil explain.As in similar lines given in example above,in the real time scenario that i havefile 1 has 30K recordsfile 2 has 25 lac recordsRequirement is: Read file 1 and copy all occurances of the record in file2 to a new file,say file3.When i use grep -f file1 file2 > file 3,file 3 is having the records required along with many unwanted records in file2 and i have to delete the unwanted records in file3. Considering the huge count,its becoming tedious. Any work around to ensure that only desired records in file1 are copied to file3 from file 2??
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Delete last 2 fields from every record in a file

Sample file record : "20130617003","2013-06-18T07:00:03","OUTWARD","01001011","TEST PLC","","HFX834346364364","20130617","10","DUM87534758","","1.28","826","020201","65879278","","","","","","010101","56789","DUMMY... (3 Replies)
Discussion started by: bigbuk
3 Replies

2. Shell Programming and Scripting

Print record count of a file using shell script

HI, I need to print the record count of a file using shell script. If the record count of a file excluding header and trailer record if greater than zero then print 'Record count of a file is xxxx records'. If the record count is zero print 'zero records' Thanks Mahendra (1 Reply)
Discussion started by: mmeda
1 Replies

3. UNIX for Dummies Questions & Answers

Delete a record in a xml file using shell scripting

find pattern, delete line with pattern and 3 lines above and 8 lines below the pattern. The pattern is "isup". The entire record with starting tag <record> and ending tag </record> containing the pattern is to be deleted and the rest to be retained. <record> ... (4 Replies)
Discussion started by: sdesstp
4 Replies

4. UNIX for Dummies Questions & Answers

how to copy files and record original file location?

:EDIT: I think my post name should have been labeled: how to copy files and record original file location. not "retain". Hello, this is my first post! I searched the forums a lot before posting, but was unable to answer my question. Here's my problem: There are several hundred text files... (4 Replies)
Discussion started by: willie8605
4 Replies

5. Shell Programming and Scripting

How to delete 1 record in large file!

Hi All, I'm a newbie here, I'm just wondering on how to delete a single record in a large file in unix. ex. file1.txt is 1000 records nikki1 nikki2 nikki3 what i want to do is delete the nikki2 record in file1.txt. is it possible? Please advise, Thanks, (3 Replies)
Discussion started by: nikki1200
3 Replies

6. Shell Programming and Scripting

Shell script to copy file

Dear all, I have a database with thousands of files with the structure of name is: Filename_hour_year.abc Filename_hour_year_1.abc .............. So what I need is how to write a script that all file with contain the character "_1" will copy to "_2" For example: file name:... (7 Replies)
Discussion started by: hainguyen1402
7 Replies

7. Shell Programming and Scripting

How to delete first record from all the file?

hi All, need help...!! I want to delete header record from all the files in current directory. using sed command i can delete first record from a file but i want to delete first record from all the files so can anybosy help me how can i do this? I will appreciate your help. (3 Replies)
Discussion started by: NirajThakar
3 Replies

8. Shell Programming and Scripting

Copy file after searching in a directory

Hi, I am looking for an answer for following senario: I have a text file (base.txt) which consist list of files to be searched like: base.txt abc.txt def.txt fgh.txt Now i am going to search all the listed files in another directory after reading them one by one, once i found the... (10 Replies)
Discussion started by: apjneeraj
10 Replies

9. Shell Programming and Scripting

Shell Script - Copy File at intervals

Hi, I want to copy some files from a Folder say, /usr/X at random intervals to another location. Basically, new files will be dumped at random intervals to location /usr/X and I have to copy those new files to some other location (after copying, I cannot delete those files from source... (2 Replies)
Discussion started by: angshuman_ag
2 Replies

10. UNIX for Dummies Questions & Answers

How to delete a record from a csv file

Hi Guys I have downloaded a table from oracle database in .csv format. it has many fields as Title, First Name, Last Name etc. I have to download distinct titles from database and now i have to check all those titles from data of First Name one by one. and then i have to delete matched record.... (1 Reply)
Discussion started by: Rajeev Agrawal
1 Replies
Login or Register to Ask a Question