Sponsored Content
Top Forums Shell Programming and Scripting Identifying entries based on 2 fields in a string. Post 302601964 by wabbit02 on Saturday 25th of February 2012 08:46:02 AM
Old 02-25-2012
Identifying entries based on 2 fields in a string.

Hi Guys,

I’m struggling to use two fields to do a duplicate/ unique by output.

I want to look IP addresses assigned to more than one account during a given period in the logs. So duplicate IP and account > 1 then print all the logs for that IP. I have been Using AWK (just as its installed on the system in question). Any help would be greatly appreciated.

log:
Code:
2012/01/01:01:30:35  type: 1, ip-assigned: 10.10.10.236, account: E8C200511C63, 
2012/01/01:01:30:36  type: 2, ip-assigned: 10.10.10.236, account: E8C200511C63, 
2012/01/01:01:30:37  type: 1, ip-assigned: 10.10.10.37, account: E8C3004BF14E,
2012/01/01:01:30:38  type: 2, ip-assigned: 10.10.10.11, account: E8C201111C63,
2012/01/01:01:30:39  type: 1, ip-assigned: 10.10.10.228, account: E8C300314D4A, 
2012/01/01:01:30:40  type: 2, ip-assigned: 10.10.10.147, account: E8C30031407A, 
2012/01/01:01:30:41  type: 1, ip-assigned: 10.10.10.236, account: E8C3003149CA, 
2012/01/01:01:30:42  type: 2, ip-assigned: 10.10.10.37, account: E8C3004BF14E, 
2012/01/01:01:30:43  type: 1, ip-assigned: 10.10.10.37, account: E8C3007069AD,
2012/01/01:01:30:44  type: 1, ip-assigned: 10.10.10.11, account: E8C201DB1C63,
2012/01/01:01:30:45  type: 2, ip-assigned: 10.10.10.228, account: E8C300314D4A, 
2012/01/01:01:30:46  type: 1, ip-assigned: 10.10.10.230, account: E8C300314D4A, 
2012/01/01:01:30:47  type: 2, ip-assigned: 10.10.10.230, account: E8C300314D4A,
2012/01/01:01:30:48  type: 1, ip-assigned: 10.10.10.101, account: E8C200511C63,

Desired output;
Code:
2012/01/01:01:30:35  type: 1, ip-assigned: 10.10.10.236, account: E8C200511C63, 
2012/01/01:01:30:36  type: 2, ip-assigned: 10.10.10.236, account: E8C200511C63, 
2012/01/01:01:30:41  type: 1, ip-assigned: 10.10.10.236, account: E8C3003149CA, 
2012/01/01:01:30:37  type: 1, ip-assigned: 10.10.10.37, account: E8C3004BF14E,
2012/01/01:01:30:42  type: 2, ip-assigned: 10.10.10.37, account: E8C3004BF14E, 
2012/01/01:01:30:43  type: 1, ip-assigned: 10.10.10.37, account: E8C3007069AD,
2012/01/01:01:30:38  type: 2, ip-assigned: 10.10.10.11, account: E8C201111C63,
2012/01/01:01:30:44  type: 1, ip-assigned: 10.10.10.11, account: E8C201DB1C63,


Last edited by Franklin52; 02-26-2012 at 05:13 AM.. Reason: Please use code tags for data and code samples, thank you
 

9 More Discussions You Might Find Interesting

1. Solaris

Identifying new fields of data

i have hundreds of lines of formatted data with 10 different fields per line. the data is refreshed every few minutes and some fields in some lines may reflect new data. i'm looking for a sample of code that help me to identify those new fields so that i can write them to a file to indicate that... (0 Replies)
Discussion started by: davels
0 Replies

2. Shell Programming and Scripting

Identifying a string from a set of files and printing to a new file

Dear All, I'm an amateur to writing scripts and need to do the following Need to read all files with a .log extension in a directory and identify the value for username i.e. all files have something like username = John. Once this is read, I need to print this value to a new file. The new file... (2 Replies)
Discussion started by: Kelly_B
2 Replies

3. Shell Programming and Scripting

Identifying specific fields in a Row

Hi, I am new to UNIX. Can some one help me to solve the below. I have a requirement to to identify the specific fields in row and also some part of the field. In my file I have a record as sundra;10.44.48.65;10thstreet TCP packet out of state: First packet isn't SYN;telno:... (3 Replies)
Discussion started by: suneel.mekala
3 Replies

4. Shell Programming and Scripting

Help identifying the first word in a string

Hi all, I'd like to know how to identify the first word in a string (in bash) for e.g. echo "enter your name" read name (user enters 'Joe Bloggs' for e.g.) echo "hello $name" (output says "hello Joe") Thanks for any help (5 Replies)
Discussion started by: kazazza
5 Replies

5. Shell Programming and Scripting

Awk - Script assistance on identifying non matching fields

Hoping for some assistance. my source file consists of: os, ip, username win7, 123.56.78, john win7, 123.56.78, paul win7, 10.1.1.1, john win7, 10.2.2.3, joe I've been trying to run a script that will only return ip and username where the IP address is the same and the username is... (3 Replies)
Discussion started by: tekvaio
3 Replies

6. Shell Programming and Scripting

sed or awk command to replace a string pattern with another string based on position of this string

here is what i want to achieve... consider a file contains below contents. the file size is large about 60mb cat dump.sql INSERT INTO `table1` (`id`, `action`, `date`, `descrip`, `lastModified`) VALUES (1,'Change','2011-05-05 00:00:00','Account Updated','2012-02-10... (10 Replies)
Discussion started by: vivek d r
10 Replies

7. Shell Programming and Scripting

Identifying columns and their values based on schema file

I have 3 files, data file,schema file and a threshold file. Data file contains data in which columns are distributed according to schema file. This data file doesn't contain any headers. Three continuous columns in the data file represent single variable in schema file. first column represent... (1 Reply)
Discussion started by: bharathbangalor
1 Replies

8. Shell Programming and Scripting

Find and replace string based on entries on another file

I have a file1 with different with multiple fields and records File2 has 2 fields. I want to find and replace strings in file1 based on file2 values (I Want an exact match i.e. for example: when searching for DT:3, Substr of DT:34 should not be matched) File2: DT:3 foo_err DT:34 bar_frr... (8 Replies)
Discussion started by: aydj
8 Replies

9. Shell Programming and Scripting

awk sort based on difference of fields and print all fields

Hi I have a file as below <field1> <field2> <field3> ... <field_num1> <field_num2> Trying to sort based on difference of <field_num1> and <field_num2> in desceding order and print all fields. I tried this and it doesn't sort on the difference field .. Appreciate your help. cat... (9 Replies)
Discussion started by: newstart
9 Replies
uniq(1) 							   User Commands							   uniq(1)

NAME
uniq - report or filter out repeated lines in a file SYNOPSIS
/usr/bin/uniq /usr/bin/uniq [-c | -d | -u] [-f fields] [-s char] [input_file [output_file]] /usr/bin/uniq [-c | -d | -u] [-n] [+ m] [input_file [output_file]] ksh93 uniq [-cdiu] [-D[delimit]] [-f fields] [-s chars] [-w chars] [input_file [output_file]] uniq [-cdiu] [-D[delimit]] [-n] [+m] [-w chars] [input_file [output_file]] DESCRIPTION
/usr/bin/uniq The uniq utility reads an input file comparing adjacent lines and writes one copy of each input line on the output. The second and succeed- ing copies of repeated adjacent input lines are not written. Repeated lines in the input are not detected if they are not adjacent. ksh93 The uniq built-in in ksh93 is associated with the /bin or /usr/bin path. It is invoked when uniq is executed without a pathname prefix and the pathname search finds a /bin/uniq or /usr/bin/uniq executable. uniq reads an input, comparing adjacent lines, and writing one copy of each input line on the output. The second and succeeding copies of the repeated adjacent lines are not written. If output_file is not specified, uniq writes to standard output. If input_file is not specified, or if input_file is -, uniq reads from standard input, and the start of the file is defined as the current offset. OPTIONS
/usr/bin/uniq The following options are supported by /usr/bin/uniq: -c Precedes each output line with a count of the number of times the line occurred in the input. -d Suppresses the writing of lines that are not repeated in the input. -f fields Ignores the first fields fields on each input line when doing comparisons, where fields is a positive decimal integer. A field is the maximal string matched by the basic regular expression: [[:blank:]]*[^[:blank:]]* If fields specifies more fields than appear on an input line, a null string is used for comparison. +m Equivalent to -s chars with chars set to m. -n Equivalent to -f fields with fields set to n. -s chars Ignores the first chars characters when doing comparisons, where chars is a positive decimal integer. If specified in conjunc- tion with the -f option, the first chars characters after the first fields fields is ignored. If chars specifies more charac- ters than remain on an input line, a null string is used for comparison. -u Suppresses the writing of lines that are repeated in the input. ksh93 The following options are supported by the uniq built-in command is ksh93: -c Outputs the number of times each line occurred along with the line. --count -d Outputs only duplicate lines. --repeated | duplicates -D Outputs all duplicate lines as a group with an empty line delimiter specified by delimit. --all-repeated[=delimit] Specify delimit as one of the following: none Do not delimit duplicate groups. prepend Prepend an empty line before each group. separate Separate each group with an empty line. The value for delimit can be omitted. The default value is none. -f Skips over fields number of fields before checking for uniqueness. A field is the minimal string matching the --skip-fields=fields BRE [[:blank:]]*[^[:blank:]]*. -i Ignore case in comparisons. --ignore-case +m Equivalent to the -s chars option, with chars set to m. -n Equivalent to the -f fields option, with fields set to n. -s Skips over chars number of characters before checking for uniqueness. --skip-chars=chars If specified with the -f option, the first chars after the first fields are ignored. If the chars specifies more characters than are on the line, an empty string is used for comparison. -u Outputs unique lines. --uniq -w Skips over any specified fields and characters, then compares chars number of characters. --check-chars=chars OPERANDS
The following operands are supported: input_file A path name of the input file. If input_file is not specified, or if the input_file is -, the standard input is used. output_file A path name of the output file. If output_file is not specified, the standard output is used. The results are unspecified if the file named by output_file is the file named by input_file. EXAMPLES
Example 1 Using the uniq Command The following example lists the contents of the uniq.test file and outputs a copy of the repeated lines. example% cat uniq.test This is a test. This is a test. TEST. Computer. TEST. TEST. Software. example% uniq -d uniq.test This is a test. TEST. example% The next example outputs just those lines that are not repeated in the uniq.test file. example% uniq -u uniq.test TEST. Computer. Software. example% The last example outputs a report with each line preceded by a count of the number of times each line occurred in the file: example% uniq -c uniq.test 2 This is a test. 1 TEST. 1 Computer. 2 TEST. 1 Software. example% ENVIRONMENT VARIABLES
See environ(5) for descriptions of the following environment variables that affect the execution of uniq: LANG, LC_ALL, LC_CTYPE, LC_MES- SAGES, and NLSPATH. EXIT STATUS
The following exit values are returned: 0 Successful completion. >0 An error occurred. ATTRIBUTES
See attributes(5) for descriptions of the following attributes: /usr/bin/uniq +-----------------------------+-----------------------------+ | ATTRIBUTE TYPE | ATTRIBUTE VALUE | +-----------------------------+-----------------------------+ |Availability |SUNWesu | +-----------------------------+-----------------------------+ |CSI |Enabled | +-----------------------------+-----------------------------+ |Interface Stability |Committed | +-----------------------------+-----------------------------+ |Standard |See standards(5). | +-----------------------------+-----------------------------+ ksh93 +-----------------------------+-----------------------------+ | ATTRIBUTE TYPE | ATTRIBUTE VALUE | +-----------------------------+-----------------------------+ |Availability |SUNWcsu | +-----------------------------+-----------------------------+ |Interface Stability |See below. | +-----------------------------+-----------------------------+ The ksh93 built-in binding to /bin and /usr/bin is Volatile. The built-in interfaces are Uncommitted. SEE ALSO
comm(1), ksh93(1), , pcat(1), sort(1), uncompress(1), attributes(5), environ(5), standards(5) SunOS 5.11 13 Mar 2008 uniq(1)
All times are GMT -4. The time now is 06:43 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy