Sponsored Content
Top Forums Shell Programming and Scripting Compare two files and output difference, by first field using awk. Post 302570624 by charles33 on Friday 4th of November 2011 12:55:38 AM
Old 11-04-2011
Amazing

First off I want to thank you, ahamed101. Thanks!

Ok, I was skeptical at first. I tried the awk solution, on the sample above; it worked. Still skeptical, I tried it on a few large "test" data sets; it worked. I'm still skeptical, before trying on large "production" data sets. So I will keep testing and report back.

I am completely blown away, so far. Which means something has got to be missing; to good to be true!

If this works, I will make ahamed101, my COD WAW nickname Smilie
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Compare Files and Output Difference

I have to compare two files for any differences, then output the lab and question number for any differences. This is what I currently have: diff lab2.txt lab2answer.txt > lab2compare.txt Though the output doesn't have to be sent to a .txt (or any sort of log), I found that easier, at least... (2 Replies)
Discussion started by: Joesgrrrl
2 Replies

2. Shell Programming and Scripting

AWK to compare two files for each field value

I have "n" files in directory A and "n" files in directory B. The files are expected to be the same with same data. Each file has 14 columns and "x" rows. Of the 14 column, 2 columns are to be considered as key identifiers. Based on this unique combination, I need to compare each field value... (2 Replies)
Discussion started by: Sangtha
2 Replies

3. Shell Programming and Scripting

AWK Compare files, different fields, output

Hi All, Looking for a quick AWK script to output some differences between two files. FILE1 device1 1.1.1.1 PINGS device1 2.2.2.2 PINGS FILE2 2862 SITE1 device1-prod 1.1.1.1 icmp - 0 ... (4 Replies)
Discussion started by: stacky69
4 Replies

4. Shell Programming and Scripting

AWK: Pattern match between 2 files, then compare a field in file1 as > or < field in file2

First, thanks for the help in previous posts... couldn't have gotten where I am now without it! So here is what I have, I use AWK to match $1 and $2 as 1 string in file1 to $1 and $2 as 1 string in file2. Now I'm wondering if I can extend this AWK command to incorporate the following: If $1... (4 Replies)
Discussion started by: right_coaster
4 Replies

5. Shell Programming and Scripting

Compare two files Field by field and output the result in another file

Hi Friends, Need Help. I have file1.txt as File1.txt |123|A|7267|Hyder|Cross|Sell|7801 |995|A|7051|2008|Lunar|New|Year|Promotion|7801 |996|A|7022|Q108|Targ|Prospect|&|SSCC|Savings|Promo|7801 |997|A|7182|Q1|Feb-Apr|08|Credit|ITA|PA|SBA|Campaign|7801 File2.txt... (7 Replies)
Discussion started by: i150371485
7 Replies

6. Shell Programming and Scripting

Plz Help. Compare 2 files field by field and get the output in another file.

Hi Freinds, I have 2 files . one is source.txt and second one is target.txt. I want to keep source.txt as baseline and compare target.txt. please find the data in 2 files and Expected output. Source.txt 1|HYD|NAG|TRA|34.5|1234 2|CHE|ESW|DES|36.5|134 3|BAN|MEH|TRA|33.5|234... (5 Replies)
Discussion started by: i150371485
5 Replies

7. Shell Programming and Scripting

Compare two text files and output difference

Hi experts, I am trying to compare two text files and output the difference to another file. I'm not strictly looking for differences in text but additional text at the end of one file that isn't in another, so basically comparing the file 2 against file 1 and printing any additional text to... (9 Replies)
Discussion started by: martin0852
9 Replies

8. Shell Programming and Scripting

Script to compare 2 files and prints difference as output sidebyside

Hi All, Am trying script to compare 2 files and print the difference found from old file to new file on line by line basis on side by side display. Basically line by line comparision and files may contain blank line as well I know we have compare/diff commands but i don't how to make... (10 Replies)
Discussion started by: Optimus81
10 Replies

9. Shell Programming and Scripting

Compare two CSV files and put the difference in third file with line no,field no and diff value.

I am having two csv files i need to compare these files and the output file should have the information of the differences at the field level. For Example, File 1: A,B,C,D,E,F 1,2,3,4,5,6 File 2: A,C,B,D,E,F 1,2,4,5,5,6 out put file: (12 Replies)
Discussion started by: karingulanagara
12 Replies

10. Shell Programming and Scripting

Simple awk command to compare two files and print first difference

Hello, I have two text files, each with a single column, file 1: 124152970 123899868 123476854 54258288 123117283 file 2: 124152970 123899868 54258288 123117283 122108330 (5 Replies)
Discussion started by: LMHmedchem
5 Replies
fitsmd5(1)						      General Commands Manual							fitsmd5(1)

NAME
fitsmd5 - Compute/update the DATAMD5 keyword/value SYNOPSIS
fitsmd5 [-u] [-s] [-a] <FITS files...> DESCRIPTION
fitsmd5 computes the MD5 signature of all data sections in a FITS file, and prints out the results on stdout. This command can optionally update the main FITS header in modifying the value of the DATAMD5 key. This command is useful to give a unique ID to a FITS file. The algorithm simply browses through all data sections in the input file and passes the data blocks to an MD5 hash function. The final result is a 128-bit signature that can be used to uniquely identify the file. This approach is meant to provide a tool to tag FITS files with unique IDs, it is not meant to be used as a checksum for file integrity (the CKSUM key is the solution for that), although it could be used in that spirit. The main point is that only data sections are taken into account, leaving the possibility of changing the headers without affecting the data signature. MD5 hashing is cryptographically strong, which means the probability of having two different FITS files getting the same ID is almost zero. It should be good enough to assign a unique ID to several tens of thousands of frames. Since there is still a tiny but non-zero possibility that two different files will get an identical key, this approach is not recommended to tag very large numbers of files (typically: mil- lions of them). If you do have a large database of FITS files, using a timestamp is usually a better approach. The MD5 signature is a good solution to tag a list of FITS files which might have originated from various sources on which the database maintainer has no control. Typically, calibration databases holding calibration frames for a given instrument, receive data from different actors who might not be in sync with unique file naming conventions. This command makes sure it is always possible to assign a unique ID to each frame. Notice that if the input FITS file has no data section, the returned MD5 key will be non-zero (it is exactly d41d8cd98f00b204e9800998ecf8427e). This signature also offers the interesting property that if two files have exactly the same pixels (bit- wise comparisons) they will get the same ID, this is useful e.g. for regression tests. If you want to produce files containing the DATAMD5 key in their main headers, you should use the qfits library, which always inserts this key. If you are working with other FITS-processing software, you should allocate an empty DATAMD5 placeholder and apply this command with the -u option to update the value. Notice that this command can also compute the MD5 sum of a complete file, not just its data sections (see -a option). In this mode, the command is completely identical to the GNU md5sum command, which is used to compute checksums on files. Input files in that case need not be FITS, though they still need to be regular files. OPTIONS
-u Try to update the DATAMD5 keyword in the main header if present. -s Silent mode: run without printing any message. -a Compute the MD5 sum on all bits in the file. In this mode, the command behaves like the GNU md5sum command, to be used e.g. as a checksum. This option excludes all others. FILES
Input files to fitsmd5 shall comply with the FITS format, except when used with -a option. 01 Aug 2001 fitsmd5(1)
All times are GMT -4. The time now is 05:28 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy