Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Creating DELTA file in UNIX

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Creating DELTA file in UNIX
# 1  
Old 04-15-2010
Creating DELTA file in UNIX
I have a fixed length file (854 characters file).

Our project will start getting this file soon. On the first day this file will have 100000 records. From the next day the file will have all the records from previous day + some new records (there will be few additions + few changes in day1 file records).

Now from day 2 onwards we want to create delta file from the incoming file. This means we only want records which are changed or are totally new.

We can keep the first day file as shadow file and day 2 onward we can perform a daily activity which compares records in day2 file with day1 file and only additions/modifications are written into new file.

Could someone please suggest how to achieve this in UNIX?

# 2  
Old 04-15-2010
man diff
# 3  
Old 04-15-2010
not so simple

I don't think simple comparison or diff tools will do the things for me. Every record in day 2 file should be checked with day 1 file and in case there is any change (in any of the 854 characters) or in case there is any new record it should be written to new file.(This new file would be the DELTA file)

Also it's not necessary that record which appeared in line 1 of day 1 file will appear in the same line in day 2 file. Even if it comes in some other file it should not be written to new file as it's exactly the same as in day 1 file.
# 4  
Old 04-15-2010
Depending upon your records, you can use "cat first_file second_file|grep -v 'pattern'" with the combination of "sort and uniq command".

Or else if you can explane a bit more about the type of your records...
I mean wether it is going to be numeric or alphabatic or something else..
# 5  
Old 04-15-2010
Code:
 
I don't think simple comparison or diff tools will do the things for me

y ?..

As suggested by Pludi , "diff" , or "comm" you can use to get the difference.
# 6  
Old 04-15-2010
OK, let's try to establish your situation here, so I can stop guessing around:
  • You are getting a file with 1..n records every day, once a day. The length of each record is fixed (not that it would matter)
  • On day 2, and each following day you want to extract those records that differ from the previous day, either because their content changed, or because they are new. Removed records do not concern you.
  • Assumption: the order in which the records appear in the difference file is not relevant.

Is this correct? Any more details to add?
# 7  
Old 04-15-2010
@Pludi
Yes , u have summarized the problem perfectly.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Script for creating Control file in UNIX

Delete ---- Original post, restored by mod after being deleted by abhilashnair ---- I have a requirement where, I need to create a control file which will have 3 columns in the header row as below: Filename Count Checksum This above control file has to contain metadata as above... (2 Replies)
Discussion started by: abhilashnair
2 Replies

2. Shell Programming and Scripting

Delta from the first digit

Thanks of your suggestions i was able to calculate the delta between some numbers in a column file with . awk 'BEGIN{last=0}{delta=$1-last; last=$1; print $0" "delta}' the file was like 499849120.00 500201312.00 500352416.00 500402784.00 500150944.00 499849120.00 500150944.00... (3 Replies)
Discussion started by: Board27
3 Replies

3. Shell Programming and Scripting

Comparing delta values of one xml file in other xml file

Hi All, I have two xml files. One is having below input <NameValuePair> <name>Daemon</name> <value>tcp:7474</value> </NameValuePair> <NameValuePair> <name>Network</name> <value></value> </NameValuePair> ... (2 Replies)
Discussion started by: sharsour
2 Replies

4. UNIX for Dummies Questions & Answers

Creating a csv file with header in UNIX

I have a flat file that contains dynamic list of variables like a=1 b=2 c=3 . .. z=26 I need to convert the above into a csv file having the format below: a,b,c,..,z 1,2,3,..,26 Please note, I do not want a comma separating the last variable. I tried to refer the post... (4 Replies)
Discussion started by: vkumbhakarna
4 Replies

5. Shell Programming and Scripting

File delta detection

Hello, I need to compare two flat files (ASCII format), say file OLD and file NEW. Both have similar structure. These files are | delimitted files and have around few million of records (lines) each. Each file has same set columns and same set of key columns (i.e. the 3rd and 5th column of the... (7 Replies)
Discussion started by: manubatham20
7 Replies

6. Shell Programming and Scripting

problem in creating execute profile file in unix

first i created a profile file(my_var.profile) which contains export my_var=20 after that i created shell scripts(my_var.sh) which contains #!/bin/bash . ./my_var.profile echo '$my_var='$my_var but when i am executing sh my_var.sh it is showing error that no such file/directory .profile.... (6 Replies)
Discussion started by: pratikjain998
6 Replies

7. Shell Programming and Scripting

problem in creating my own profile file in unix

I am new in shell scripting. currently i am using cygwin. My problem is i created a profile file in my own folder. file name is first.profile in which i gave following values to variable export a=10 now i am executing this profile file by below command ./.first.profile it executed... (4 Replies)
Discussion started by: pratikjain998
4 Replies

8. UNIX for Dummies Questions & Answers

UNIX script for reading a file and creating another file

Hi, I am a beginner in scripting...I have to do a script where I have to read a file which has list of job names, line by line and for every line execute a dsjob command to find the log details of the job and extract only the start time of the job, if it is greater than jan 01 2008 and create... (1 Reply)
Discussion started by: Vijay81
1 Replies

9. OS X (Apple)

What's The Easiest Route To Creating A Unix Executable File for Terminal?

I've seen the executable open in the application OmniOutliner, can I create an executable with this app? I'd like to be able to create the unix executable and insert it into terminal, but I'm not sure if the Omni app will allow me to create it. Any one have any ideas or possibly familiar with... (10 Replies)
Discussion started by: unimachead
10 Replies

10. Shell Programming and Scripting

Need help in creating a Unix Script to parse xml file

Hi All, My requirement is create an unix script to parse the xml file and display the values of the Elements/value between the tags on console. Like say, I would like to fetch the value of errorCode from the below xml which is 'U007' and display it. Can we use SED command for this? I have tried... (10 Replies)
Discussion started by: Anil.Wmg
10 Replies
Login or Register to Ask a Question