Advice on Script


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Advice on Script
# 1  
Old 02-10-2007
Advice on Script

I would like some advice on how to logically put together a script to handle a daily task of data gathering for the following problem.

I have two files, file1 has 125,000 records that I cut and remove unwanted fields through scripts and cron. In file2, I have 25000 records that has the same data as file1 plus one extra field of data. I would like the records from file2 (25,000) to replace entries in file1 with the extra data which would then be file3.

I hope I can express this clearly and appreicate any help.

Eample:
file1
00:01:A6:A9:3A:7E|20070103_120040|1|8|0|0|
00:0A:73:1F:13:54|20070103_120041|1|8|0|0|
00:40:7B:D6:74:E8|20070103_120041|1|8|0|0|

file2
00:01:A6:A9:3A:7E|20070103_120040|1|8|0|0|6 days
00:0A:73:1F:13:54|20070103_120041|1|8|0|0|37 day
00:40:7B:D6:74:E8|20070103_120041|1|8|0|0|24 days

file3
00:01:A6:A9:3A:7E|20070103_120040|1|8|0|0|6 days
00:0A:73:1F:13:54|20070103_120041|1|8|0|0|37 day
00:40:7B:D6:74:E8|20070103_120041|1|8|0|0|24 days
00:0A:73:25:B9:EC|20070103_120039|1|7|0|0|
00:01:A6:96:48:60|20070103_120040| | | | |
00:01:A6:A9:3A:7E|20070103_120040|1|8|0|0|
00:0A:73:1F:13:54|20070103_120041|1|8|0|0|
00:40:7B:D6:74:E8|20070103_120041|1|8|0|0|
00:0A:73:23:BA:A8|20070103_120042| | | | |

Last edited by Perderabo; 02-10-2007 at 04:26 PM.. Reason: Disable smilies for readability
# 2  
Old 02-10-2007
I did not get it clearly enough.


PHP Code:
I would like the records from file2 (25,000to replace entries in file1 with the extra data 
records which has to be replaced in file1 with records from file2, do they have any specific criteria or is that just file2 records and replace any record from file1 to create file3.

Please clarify.

Thanks
Sumeet
# 3  
Old 02-10-2007
sumeet,

File2 data (25,000) records will replace the 25,000 records in file1. The records can be in any order as long as the entire line is replaced. The only net gain is the data in the last column, field making total records remaining at 125,000.

I hope this helps.
# 4  
Old 02-10-2007
#!/usr/bin/ksh

> file3
while read LINE
do
LINE2=`grep "^${LINE}|" file2 2> /dev/null`
if [ ${?} -eq 0 ]
then
echo "${LINE2} >> file3
else
echo "${LINE} >> file3
fi
done < file1
# 5  
Old 02-11-2007
That would fire off a lot of grep processes. This might be faster:
sort -t'|' -k1,6 -k7r file1 file2| sort -m -t'|' -k1,6 -u > file3
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Advice on a backup script, maybe one is out there already?

Hi, Not sure whether this is the right place to post it. I decided to post it here 'coz Advanced and Expert users will most likely have the answer to what I am looking for. I want to backup scripts that I have access to to a tar file file and zip it. At the moment I am creating a directory... (4 Replies)
Discussion started by: newbie_01
4 Replies

2. Shell Programming and Scripting

Script advice

All I have 2 parent directories - input and output. Each parent has multiple sub-directories...each sub-directory has multiple files. Each parent directory structure is a mirror image of itself I need to poll the imput directory and if a new file is found, encrypt the file, move the file to... (2 Replies)
Discussion started by: davidra
2 Replies

3. Shell Programming and Scripting

Need advice on approach for script

Greetings all. I have a repository server which receives, without exhaggeration, several million files a week. The majority of these files are in .csv format, which means they're highly compressable. They are spread throughout numerous directories where there are configured monitoring utilities... (4 Replies)
Discussion started by: msarro
4 Replies

4. Shell Programming and Scripting

Advice on script

Hi folks, I use following script:- #!/bin/sh # cd Linbread TODAY=`date +"%m%d"` DATA=`grep $TODAY linbread.dat` HOUR=`date +"%H"` if then TOD="Morning" elif then TOD="Afternoon" else TOD="Evening" fi echo $DATA | gawk -F"|" '{printf("%s\n\n%s",$2,$3)}' > $$tmp fold -s -w60... (0 Replies)
Discussion started by: satimis
0 Replies

5. Shell Programming and Scripting

Looking for optimization advice on a short script

I already have a solution to my problem, but I'm looking to see if it can be made more succinct and faster. The problem: given a list, as shown below, extract the pathname for any file in a directory named 'ample' and return it's index into the list. The index is also in the data itself. Note that... (1 Reply)
Discussion started by: prigo
1 Replies

6. Shell Programming and Scripting

advice on shell script

Hello, I have this script running on cron every 20 minutes. By 12pm daily, our system is expecting all input files to be uploaded by the script. After this cutoff time, the script would still be running though, but i need some kind of alerts/logs to know which input files weren't received for... (1 Reply)
Discussion started by: gholdbhurg
1 Replies

7. Shell Programming and Scripting

Script Help/Advice

Alright, I feel like I have a pretty good basic knowledge of shell scripting, but this one is throwing me for a loop. I know I've seen something similar done with awk, but I couldn't find it with the search function. I've grepped through my log file and get results like this: --... (14 Replies)
Discussion started by: earnstaf
14 Replies

8. Shell Programming and Scripting

Script Advice please?

Ok. I want to parse a log file and search only for denied traffic for the previous hour. The log looks like this: Jun 18 17:47:56 routername 36806: Jun 18 17:53:01.088: %SEC-6-IPACCESSLOG: list ingress-filter denied tcp 1.2.3.4(1234) -> 6.7.8.9(53), 4 packets I only really care about the... (12 Replies)
Discussion started by: earnstaf
12 Replies

9. Shell Programming and Scripting

first script. need help and advice.

Hello everyone, This is my first post here and this is the first time I am using UNIX OS (Slackware). I find it really useful and powerful and would like to master it but as you may guess I am expreicing quite a few problems. I've been reading a few documentations about it and bash this week... (17 Replies)
Discussion started by: sanchopansa
17 Replies

10. Shell Programming and Scripting

c-shell script advice please.

Hi, I have the following script running in my cron. -------------------------------------------------------------------- #!/bin/csh bnstat -p GPD_VSLinux | grep pg | grep varcon | awk '{print $1, $2, $3, $4, $5, $6, $7, $8, $9, $10}' > /tmp/LX_xbatch.log bnstat -p GPD_VSLinux_test | grep pg... (2 Replies)
Discussion started by: killerserv
2 Replies
Login or Register to Ask a Question