Sponsored Content
Top Forums Shell Programming and Scripting Remove Duplicates on multiple Key Columns and get the Latest Record from Date/Time Column Post 302799151 by Don Cragun on Thursday 25th of April 2013 07:55:09 PM
Old 04-25-2013
Can we assume that CDC_FLAG will always be the 3rd field, SRC_PMTN_I will always be the 5th field, and CDC_PRCS_TS will always be the 1st field; or do we have to match the strings against the header line to determine which fields to use?

Do the output records need to be in the same order as they appeared in the input file or can the output be in random order except that the 1st output line must be the 1st input line (the headings)?
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Remove duplicates based on the two key columns

Hi All, I needs to fetch unique records based on a keycolumn(ie., first column1) and also I needs to get the records which are having max value on column2 in sorted manner... and duplicates have to store in another output file. Input : Input.txt 1234,0,x 1234,1,y 5678,10,z 9999,10,k... (7 Replies)
Discussion started by: kmsekhar
7 Replies

2. Shell Programming and Scripting

Search based on 1,2,4,5 columns and remove duplicates in the same file.

Hi, I am unable to search the duplicates in a file based on the 1st,2nd,4th,5th columns in a file and also remove the duplicates in the same file. Source filename: Filename.csv "1","ccc","information","5000","temp","concept","new" "1","ddd","information","6000","temp","concept","new"... (2 Replies)
Discussion started by: onesuri
2 Replies

3. Shell Programming and Scripting

need to remove duplicates based on key in first column and pattern in last column

Given a file such as this I need to remove the duplicates. 00060011 PAUL BOWSTEIN ad_waq3_921_20100826_010517.txt 00060011 PAUL BOWSTEIN ad_waq3_921_20100827_010528.txt 0624-01 RUT CORPORATION ad_sade3_10_20100827_010528.txt 0624-01 RUT CORPORATION ... (13 Replies)
Discussion started by: script_op2a
13 Replies

4. Shell Programming and Scripting

finding duplicates in csv based on key columns

Hi team, I have 20 columns csv files. i want to find the duplicates in that file based on the column1 column10 column4 column6 coulnn8 coulunm2 . if those columns have same values . then it should be a duplicate record. can one help me on finding the duplicates, Thanks in advance. ... (2 Replies)
Discussion started by: baskivs
2 Replies

5. Shell Programming and Scripting

Removing duplicates in fixed width file which has multiple key columns

Hi All , I have a requirement where I need to remove duplicates from a fixed width file which has multiple key columns .Also , need to capture the duplicate records into another file . File has 8 columns. Key columns are col1 and col2. Col1 has the length of 8 col 2 has the length of 3. ... (5 Replies)
Discussion started by: saj
5 Replies

6. Shell Programming and Scripting

Remove the time from the date column

Hi, I have file named file1.txt with below contents cat file1.txt 1/29/2014 0:00,706886 1/30/2014 0:00,791265 1/31/2014 0:00,987087 2/1/2014 0:00,1098572 2/2/2014 0:00,572477 2/3/2014 0:00,701715 I want to display as below 1/29/2014,706886 1/30/2014,791265 1/31/2014,987087... (5 Replies)
Discussion started by: villain41
5 Replies

7. UNIX for Dummies Questions & Answers

Display latest record from file based on multiple columns combination

I have requirement to print latest record from file based on multiple columns combination. EWAPE EW1SLE0000 EW1SOMU01 ABORTED 03/16/2015 100004 03/16/2015 100005 001 EWAPE EW1SLE0000 EW1SOMU01 ABORTED 03/18/2015 140003 03/18/2015 140004 001 EWAPE EW1SLE0000 EW1SOMU01 ABORTED 03/18/2015 220006... (1 Reply)
Discussion started by: tmalik79
1 Replies

8. UNIX for Beginners Questions & Answers

Sort and remove duplicates in directory based on first 5 columns:

I have /tmp dir with filename as: 010020001_S-FOR-Sort-SYEXC_20160229_2212101.marker 010020001_S-FOR-Sort-SYEXC_20160229_2212102.marker 010020001-S-XOR-Sort-SYEXC_20160229_2212104.marker 010020001-S-XOR-Sort-SYEXC_20160229_2212105.marker 010020001_S-ZOR-Sort-SYEXC_20160229_2212106.marker... (4 Replies)
Discussion started by: gnnsprapa
4 Replies

9. Shell Programming and Scripting

awk to Sum columns when other column has duplicates and append one column value to another with Care

Hi Experts, Please bear with me, i need help I am learning AWk and stuck up in one issue. First point : I want to sum up column value for column 7, 9, 11,13 and column15 if rows in column 5 are duplicates.No action to be taken for rows where value in column 5 is unique. Second point : For... (1 Reply)
Discussion started by: as7951
1 Replies

10. UNIX for Beginners Questions & Answers

Remove duplicates in a dataframe (table) keeping all the different cells of just one of the columns

Hello all, I need to filter a dataframe composed of several columns of data to remove the duplicates according to one of the columns. I did it with pandas. In the main time, I need that the last column that contains all different data ( not redundant) is conserved in the output like this: A ... (5 Replies)
Discussion started by: pedro88
5 Replies
STRFILE(8)						    BSD System Manager's Manual 						STRFILE(8)

NAME
strfile, unstr -- create a random access file for storing strings SYNOPSIS
strfile [-Ciorsx] [-c char] source_file [output_file] unstr source_file DESCRIPTION
The strfile utility reads a file containing groups of lines separated by a line containing a single percent '%' sign and creates a data file which contains a header structure and a table of file offsets for each group of lines. This allows random access of the strings. The output file, if not specified on the command line, is named source_file.dat. The options are as follows: -C Flag the file as containing comments. This option cases the STR_COMMENTS bit in the header str_flags field to be set. Comments are designated by two delimiter characters at the beginning of the line, though strfile does not give any special treatment to comment lines. -c char Change the delimiting character from the percent sign to char. -i Ignore case when ordering the strings. -o Order the strings in alphabetical order. The offset table will be sorted in the alphabetical order of the groups of lines refer- enced. Any initial non-alphanumeric characters are ignored. This option causes the STR_ORDERED bit in the header str_flags field to be set. -r Randomize access to the strings. Entries in the offset table will be randomly ordered. This option causes the STR_RANDOM bit in the header str_flags field to be set. -s Run silently; do not give a summary message when finished. -x Note that each alphabetic character in the groups of lines is rotated 13 positions in a simple caesar cypher. This option causes the STR_ROTATED bit in the header str_flags field to be set. The format of the header is: #define VERSION 1 uint32_t str_version; /* version number */ uint32_t str_numstr; /* # of strings in the file */ uint32_t str_longlen; /* length of longest string */ uint32_t str_shortlen; /* length of shortest string */ #define STR_RANDOM 0x1 /* randomized pointers */ #define STR_ORDERED 0x2 /* ordered pointers */ #define STR_ROTATED 0x4 /* rot-13'd text */ #define STR_COMMENTS 0x8 /* embedded comments */ uint32_t str_flags; /* bit field for flags */ char str_delim; /* delimiting character */ All fields are written in network byte order. The purpose of unstr is to undo the work of strfile. It prints out the strings contained in the file source_file in the order that they are listed in the header file source_file.dat to standard output. It is possible to create sorted versions of input files by using -o when strfile is run and then using unstr to dump them out in the table order. FILES
strfile.dat default output file. SEE ALSO
byteorder(3), fortune(6) HISTORY
The strfile utility first appeared in 4.4BSD. BSD
February 17, 2005 BSD
All times are GMT -4. The time now is 04:07 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy