Finding difference in 1st field for rows of data


 
Thread Tools Search this Thread
# 1  
Finding difference in 1st field for rows of data

I have a file that has multiple lines, of grouped data, that typically all have the same values in the 1st field, however, I would like to search the 1st field for any differences and set a flag to use in an "if" statement to run some other routine.

An example of the typical file is below, which is a comma separated file:

Code:
1st field     2nd field     3rd field
   "20"  ,       "file1"     ,   "xxxx"
   "20"  ,       "file1"     ,   "xxxx"
   "20"  ,       "file1"     ,   "xxxx"
   "20"  ,       "file1"     ,   "xxxx"
   "20"  ,       "file2"     ,   "yyyy"
   "20"  ,       "file2"     ,   "yyyy"
   "20"  ,       "file2"     ,   "yyyy"
   "20"  ,       "file2"     ,   "yyyy"
   "20"  ,       "file3"     ,   "zzzz"
   "20"  ,       "file3"     ,   "zzzz"
   "20"  ,       "file3"     ,   "zzzz"
   "20"  ,       "file3"     ,   "zzzz"

I want to set a flag for each data set of the 3rd field, if the file has differences in the 1st field like below:
Code:
1st field     2nd field     3rd field
   "20"  ,       "file1"     ,   "xxxx"
   "1"   ,       "file1"     ,   "xxxx"
   "20"  ,       "file1"     ,   "xxxx"
   "20"  ,       "file1"     ,   "xxxx"
   "1"  ,       "file2"     ,   "yyyy"
   "20"  ,       "file2"     ,   "yyyy"
   "20"  ,       "file2"     ,   "yyyy"
   "20"  ,       "file2"     ,   "yyyy"
   "20"  ,       "file3"     ,   "zzzz"
   "1"  ,       "file3"     ,   "zzzz"
   "20"  ,       "file3"     ,   "zzzz"
   "20"  ,       "file3"     ,   "zzzz"

One method I've tried so far is to strip out the lines where the 3rd field is all the same, and then try to run through each line and check the 1st field for differences.

I've tried the following, but can't get it to work.
Code:
 awk -F ',' '{f = $1} {for (i=1;i<=NR;i++) if ($1 != $f) {printf "%s\n", "FAIL"} }' tmpfile > fail.log


Last edited by co21ss; 02-07-2013 at 02:29 PM.. Reason: Add awk method used
# 2  
Not clear. The file seems to be comma AND multiple space delimited. AND, first four lines (and next four ... etc.) seem to contain identical values. It's easy to set a flag on a field changing, but to what avail? Where does $1 == "1" into play? I don't see a connection between your input sample and the desired output you post. Pls be more specific, post more detailed input / output samples.
And, btw, don't use $f in your awk script. That's a shell'ism; use f to reference the variable itself. In awk, $f would address the f'th field of the input line...
# 3  
Quote:
Originally Posted by RudiC
Not clear. The file seems to be comma AND multiple space delimited. AND, first four lines (and next four ... etc.) seem to contain identical values. It's easy to set a flag on a field changing, but to what avail? Where does $1 == "1" into play? I don't see a connection between your input sample and the desired output you post. Pls be more specific, post more detailed input / output samples.
And, btw, don't use $f in your awk script. That's a shell'ism; use f to reference the variable itself. In awk, $f would address the f'th field of the input line...
Sorry for not being clear. The code is not space separated, but just shown as such to clarify the input. Here is a better example:

Typical input -
Code:
"20","file1","xxxx"
"20","file1","xxxx"
"20","file1","xxxx"
"20","file1","xxxx"
"20","file2","yyyy"
"20","file2","yyyy"
"20","file2","yyyy"
"20","file2","yyyy"
"20","file3","zzzz"
"20","file3","zzzz"
"20","file3","zzzz"
"20","file3","zzzz"

Non-typical input -
Code:
"1","file1","xxxx"
"20","file1","xxxx"
"20","file1","xxxx"
"20","file1","xxxx"
"20","file2","yyyy"
"20","file2","yyyy"
"5","file2","yyyy"
"20","file2","yyyy"
"20","file3","zzzz"
"20","file3","zzzz"
"20","file3","zzzz"
"20","file3","zzzz"

In the non-typical input case, I would like to create an output, such as "FAIL", to use in a later routine if the 1st field is different from the rest for any of the individual files.

The 1st field will not always be known, but however is a value input from another file entirely. The goal here is run a script to automate having to visually look through the file for differences in the 1st field for any "file1" or "file2" lines of the file.

Last edited by co21ss; 02-07-2013 at 05:08 PM.. Reason: clarification
 

Previous Thread | Next Thread
Thread Tools Search this Thread
Search this Thread:
Advanced Search

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Finding difference in 2 different timestamps

Legends, I have a requirement to run the script exactly after one hour of completion of dependent script. Eg: Script B should run after one hour on the completion of Script A. I got the time stamps using following variables. these scripts runs in autosys > DATE=`date +%H:%M` >... (4 Replies)
Discussion started by: sdosanjh
4 Replies

2. Shell Programming and Scripting

Finding first difference between two files

Hi! I'd like to know if it is possible for a command to find the first difference between two large files, output that line from both file and stop, so no need to continue after that to save some computation time. I don't think looping through it will be efficient enough but that's the only... (6 Replies)
Discussion started by: Mojing
6 Replies

3. Shell Programming and Scripting

Summing up rows data regarding 1st column

Dear all, I have one file like LABEL A B C D E F G H I J K L M N G02100 64651.3 25630.7 8225.21 51238 267324 268005 234001 52410.9 18598.2 10611 10754.7 122535 267170 36631.4 G02100 12030.3 8260.15 8569.91 ... (4 Replies)
Discussion started by: AAWT
4 Replies

4. Shell Programming and Scripting

finding difference between two files

Hi, I have two files one with 12486 lines second one with 13116 As per the comparsion between two files the count have 630 difference I used diff command to find the difference between two files but it's not understandable could any one suggest any command to get 630 records in a new... (4 Replies)
Discussion started by: thelakbe
4 Replies

5. UNIX for Dummies Questions & Answers

Suggestion to convert data in rows to data in columns

Hello everyone! I have a huge dataset looking like this: nameX nameX 0 1 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 1 2 2 2 ............... nameY nameY 2 2 2 2 2 2 2 2 2 2 1 2 2 2 1 2 2 2 ..... nameB nameB 0 1 2 2 2 2 2 2 2 2 1 2 2 2 1 2 2 2 ..... (can be several thousands of codes) and I need... (8 Replies)
Discussion started by: kush
8 Replies

6. Shell Programming and Scripting

Difference between corresponding elements of successive rows

Hi, I have a file in the following format a1 b1 c1 d1 a2 b2 c2 d2 a3 b3 c3 d3 a4 b4 c4 d4 I need a script to find the difference between corresponding values of successive rows. So the output would have one less row than the input file and should look like: a2-a1 b2-b1 c2-c1 d2-d1... (4 Replies)
Discussion started by: sajal.bhatia
4 Replies

7. Shell Programming and Scripting

Extract difference of two columns from different rows

Hello guys, Please help me to solve this problem. I have tried some awk commands but couldn't succeed. I have a tab delimited file where each record is separated by ------ and 4th column of each record is same. <INPUT FILE> ------ peon 53931587 53931821 ... (12 Replies)
Discussion started by: sam_2921
12 Replies

8. Shell Programming and Scripting

Finding the time difference

Hi, I have two files A.txt and B.txt. And i have the following attributes in both the files. <date and time> <a unique id> For eg: <2007 May 30 20:29:36:034 GMT> <ID1> in A.txt <2007 May 30 20:42:36:038 GMT> <ID1> in B.txt Now, i need to find the time difference... (0 Replies)
Discussion started by: padma.raajesh
0 Replies

9. Shell Programming and Scripting

Finding different rows

I have two files. They are prety much the same, except a few lines. I would like to get the differences between the two file, but only those differences and nothing more. "diff" and "sdiff" cannot seem to do it. "sdiff -s" gets close, but I still get "less than" or "greater than" signs in the... (2 Replies)
Discussion started by: dombi
2 Replies

10. Shell Programming and Scripting

Difference between two rows

Dears, I have a list as follows, 2 4 8 If I want to find the difference between two consecutive rows. Then I have to store the specific rows in two variables and then find the difference. Could someone tell how this can be done. Regards, (7 Replies)
Discussion started by: JimJim
7 Replies

Featured Tech Videos