Sponsored Content
Top Forums Shell Programming and Scripting Count duplicate lines ignoring certain columns Post 302887135 by Yoda on Thursday 6th of February 2014 12:39:22 AM
Old 02-06-2014
Using awk; reading input file twice:
Code:
awk '
        NR == 1 {
                print
        }
        NR == FNR && NR > 1 {
                i = $2 FS $3 FS $4 FS $5
                A[i] += $1
                next
        }
        {
                i = $2 FS $3 FS $4 FS $5
                if ( ( i in A ) && !( i in R ) )
                        print A[i], i
                R[i]
        }
' file file

This User Gave Thanks to Yoda For This Post:
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

How to count lines - ignoring blank lines and commented lines

What is the command to count lines in a files, but ignore blank lines and commented lines? I have a file with 4 sections in it, and I want each section to be counted, not including the blank lines and comments... and then totalled at the end. Here is an example of what I would like my... (6 Replies)
Discussion started by: kthatch
6 Replies

2. Shell Programming and Scripting

Ignoring several lines at once in cshell

Hi We use # sign to ignore any line (i.e. comment ). But is it possible to ignore group of line at once or i have to use # in front of each line. Thanks Sarbjit (3 Replies)
Discussion started by: sarbjit
3 Replies

3. UNIX for Dummies Questions & Answers

Duplicate columns and lines

Hi all, I have a tab-delimited file and want to remove identical lines, i.e. all of line 1,2,4 because the columns are the same as the columns in other lines. Any input is appreciated. abc gi4597 9997 cgcgtgcg $%^&*()()* abc gi4597 9997 cgcgtgcg $%^&*()()* ttt ... (1 Reply)
Discussion started by: dr_sabz
1 Replies

4. UNIX for Dummies Questions & Answers

deleteing duplicate lines sing uniq while ignoring a column

I have a data set that has 4 columns, I want to know if I can delete duplicate lines while ignoring one of the columns, for example 10 chr1 ASF 30 15 chr1 ASF 20 5 chr1 ASF 30 6 chr2 EBC 15 4 chr2 EBC 30 ... I want to know if I can delete duplicate lines while ignoring column 1, so the... (5 Replies)
Discussion started by: japaneseguitars
5 Replies

5. Shell Programming and Scripting

ignoring lines in a file

HI, command to cat a readable file by ignoring the first line and last line or command to cat a readable file by ignoring the lines with delimiter Please advise on this. (2 Replies)
Discussion started by: thelakbe
2 Replies

6. UNIX for Advanced & Expert Users

In a huge file, Delete duplicate lines leaving unique lines

Hi All, I have a very huge file (4GB) which has duplicate lines. I want to delete duplicate lines leaving unique lines. Sort, uniq, awk '!x++' are not working as its running out of buffer space. I dont know if this works : I want to read each line of the File in a For Loop, and want to... (16 Replies)
Discussion started by: krishnix
16 Replies

7. UNIX for Dummies Questions & Answers

remove duplicate lines based on two columns and judging from a third one

hello all, I have an input file with four columns like this with a lot of lines and for example, line 1 and line 5 match because the first 4 characters match and the fourth column matches too. I want to keep the line that has the lowest number in the third column. So I discard line 5.... (5 Replies)
Discussion started by: TheTransporter
5 Replies

8. Shell Programming and Scripting

Ignoring lines and create new file

Hello, I have a requirement to ignore few lines in a file before keyword FILEHEADER . As soon as there is keyword FILEHEADER is identified in file , it will form another file with data from FILEHEADER to whatever in file after FILEHEADER. I wrote filename=$1 awk... (4 Replies)
Discussion started by: callmatkarna
4 Replies

9. Shell Programming and Scripting

Replace a column with a value by ignoring the header lines

i have a file in the gz format , the content of the file is as follow. gzcat f1.gz # 1.name # 2.location # 3.age # 4.dob . . . . . . . . . # 43.hobbies < Aravind,33,chennai,09091980, , , , , , , surfing> (5 Replies)
Discussion started by: aravindj80
5 Replies

10. Shell Programming and Scripting

Remove duplicate lines after ignoring case and spaces between

Oracle Linux 6.5 $ cat someStrings.txt GRANT select on MANHPRD.S_PROD_INT TO OR_PHIL; GRANT select on MANHPRD.S_PROD_INT TO OR_PHIL; GRANT select on SCOTT.emp to JOHN; grant select on scott.emp to john; grant select on scott.dept to hr;If you ignore the case and the empty space between the... (6 Replies)
Discussion started by: kraljic
6 Replies
STRTOK(3)						     Linux Programmer's Manual							 STRTOK(3)

NAME
strtok, strtok_r - extract tokens from strings SYNOPSIS
#include <string.h> char *strtok(char *str, const char *delim); char *strtok_r(char *str, const char *delim, char **saveptr); Feature Test Macro Requirements for glibc (see feature_test_macros(7)): strtok_r(): _POSIX_C_SOURCE || /* Glibc versions <= 2.19: */ _BSD_SOURCE || _SVID_SOURCE DESCRIPTION
The strtok() function breaks a string into a sequence of zero or more nonempty tokens. On the first call to strtok(), the string to be parsed should be specified in str. In each subsequent call that should parse the same string, str must be NULL. The delim argument specifies a set of bytes that delimit the tokens in the parsed string. The caller may specify different strings in delim in successive calls that parse the same string. Each call to strtok() returns a pointer to a null-terminated string containing the next token. This string does not include the delimiting byte. If no more tokens are found, strtok() returns NULL. A sequence of calls to strtok() that operate on the same string maintains a pointer that determines the point from which to start searching for the next token. The first call to strtok() sets this pointer to point to the first byte of the string. The start of the next token is determined by scanning forward for the next nondelimiter byte in str. If such a byte is found, it is taken as the start of the next token. If no such byte is found, then there are no more tokens, and strtok() returns NULL. (A string that is empty or that contains only delim- iters will thus cause strtok() to return NULL on the first call.) The end of each token is found by scanning forward until either the next delimiter byte is found or until the terminating null byte ('') is encountered. If a delimiter byte is found, it is overwritten with a null byte to terminate the current token, and strtok() saves a pointer to the following byte; that pointer will be used as the starting point when searching for the next token. In this case, strtok() returns a pointer to the start of the found token. From the above description, it follows that a sequence of two or more contiguous delimiter bytes in the parsed string is considered to be a single delimiter, and that delimiter bytes at the start or end of the string are ignored. Put another way: the tokens returned by strtok() are always nonempty strings. Thus, for example, given the string "aaa;;bbb,", successive calls to strtok() that specify the delimiter string ";," would return the strings "aaa" and "bbb", and then a null pointer. The strtok_r() function is a reentrant version strtok(). The saveptr argument is a pointer to a char * variable that is used internally by strtok_r() in order to maintain context between successive calls that parse the same string. On the first call to strtok_r(), str should point to the string to be parsed, and the value of saveptr is ignored. In subsequent calls, str should be NULL, and saveptr should be unchanged since the previous call. Different strings may be parsed concurrently using sequences of calls to strtok_r() that specify different saveptr arguments. RETURN VALUE
The strtok() and strtok_r() functions return a pointer to the next token, or NULL if there are no more tokens. ATTRIBUTES
For an explanation of the terms used in this section, see attributes(7). +-----------+---------------+-----------------------+ |Interface | Attribute | Value | +-----------+---------------+-----------------------+ |strtok() | Thread safety | MT-Unsafe race:strtok | +-----------+---------------+-----------------------+ |strtok_r() | Thread safety | MT-Safe | +-----------+---------------+-----------------------+ CONFORMING TO
strtok() POSIX.1-2001, POSIX.1-2008, C89, C99, SVr4, 4.3BSD. strtok_r() POSIX.1-2001, POSIX.1-2008. BUGS
Be cautious when using these functions. If you do use them, note that: * These functions modify their first argument. * These functions cannot be used on constant strings. * The identity of the delimiting byte is lost. * The strtok() function uses a static buffer while parsing, so it's not thread safe. Use strtok_r() if this matters to you. EXAMPLE
The program below uses nested loops that employ strtok_r() to break a string into a two-level hierarchy of tokens. The first command-line argument specifies the string to be parsed. The second argument specifies the delimiter byte(s) to be used to separate that string into "major" tokens. The third argument specifies the delimiter byte(s) to be used to separate the "major" tokens into subtokens. An example of the output produced by this program is the following: $ ./a.out 'a/bbb///cc;xxx:yyy:' ':;' '/' 1: a/bbb///cc --> a --> bbb --> cc 2: xxx --> xxx 3: yyy --> yyy Program source #include <stdio.h> #include <stdlib.h> #include <string.h> int main(int argc, char *argv[]) { char *str1, *str2, *token, *subtoken; char *saveptr1, *saveptr2; int j; if (argc != 4) { fprintf(stderr, "Usage: %s string delim subdelim ", argv[0]); exit(EXIT_FAILURE); } for (j = 1, str1 = argv[1]; ; j++, str1 = NULL) { token = strtok_r(str1, argv[2], &saveptr1); if (token == NULL) break; printf("%d: %s ", j, token); for (str2 = token; ; str2 = NULL) { subtoken = strtok_r(str2, argv[3], &saveptr2); if (subtoken == NULL) break; printf(" --> %s ", subtoken); } } exit(EXIT_SUCCESS); } Another example program using strtok() can be found in getaddrinfo_a(3). SEE ALSO
index(3), memchr(3), rindex(3), strchr(3), string(3), strpbrk(3), strsep(3), strspn(3), strstr(3), wcstok(3) COLOPHON
This page is part of release 4.15 of the Linux man-pages project. A description of the project, information about reporting bugs, and the latest version of this page, can be found at https://www.kernel.org/doc/man-pages/. GNU
2017-09-15 STRTOK(3)
All times are GMT -4. The time now is 02:07 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy