Sponsored Content
Top Forums Shell Programming and Scripting Duplicate line removal matching some columns only Post 302783161 by RudiC on Wednesday 20th of March 2013 05:19:41 AM
Old 03-20-2013
Above proposal was thought to be a starting point to be adapted and refined. For a nearly three year member the reaction is highly disappointing.
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Sort, duplicate removal - Query

Hi All, I have a problem with the sort and duplicate filter command I am using in one of my scripts. I have a '|' delimited file and want to sort and remove duplicates on the 1,2,15 fields. These fields constitute the primary key of the table I will be loading the data into. But I see that some... (4 Replies)
Discussion started by: novice1324
4 Replies

2. UNIX for Dummies Questions & Answers

exclude columns with a matching line pattern

Hi , I have 5 columns total and am wanting to search lines in columns 3-5 and basically grep -v patterns that match 'BBB_0123' 'BVG_0895' 'BSD_0987' Does anyone know how to do this? I tried combining grep -v with grep -e but, it didn't work. Thanks! (5 Replies)
Discussion started by: greptastic
5 Replies

3. Shell Programming and Scripting

Removal of Duplicate Entries from the file

I have a file which consists of 1000 entries. Out of 1000 entries i have 500 Duplicate Entires. I want to remove the first Duplicate Entry (i,e entire Line) in the File. The example of the File is shown below: 8244100010143276|MARISOL CARO||MORALES|HSD768|CARR 430 KM 1.7 ... (1 Reply)
Discussion started by: ravi_rn
1 Replies

4. Shell Programming and Scripting

using command line arguments as columns for pattern matching using awk

Hi, I wish to use a column, as inputted by a user from command line, for pattern matching. awk file: { if($1 ~ /^8/) { print $0> "temp2.csv" } } something like this, but i want '$1' to be any column as selected by the user from command line. ... (1 Reply)
Discussion started by: invinclible0009
1 Replies

5. Shell Programming and Scripting

Column Search and Line Removal

Hello Gurus, I need to remove lines within a file if it contains specific criteria. Here is what I am trying to resolve: Users of AppRuntime: (Total of 10 licenses issued; Total of 6 licenses in use) buih02 dsktp501 AppGui 1 (compute_lic/27006 3122), start Mon 2/22 7:58 dingj1... (3 Replies)
Discussion started by: leepet01
3 Replies

6. Shell Programming and Scripting

Remove duplicate lines (the first matching line by field criteria)

Hello to all, I have this file 2002 1 23 0 0 2435.60 131.70 5.60 20.99 0.89 0.00 285.80 2303.90 2002 1 23 15 0 2436.60 132.90 6.45 21.19 1.03 0.00 285.80 2303.70 2002 1 23 ... (6 Replies)
Discussion started by: joggdial3000
6 Replies

7. UNIX for Advanced & Expert Users

Duplicate removal

I have an input file of 5GB which contains duplicate records and have to remove duplicate records by retaing first instance of that record . Based on 5 fields the duplicates has to be removed . Kindly request to help me in writing a Unix Script. Thanks Asim (11 Replies)
Discussion started by: duplicate
11 Replies

8. Shell Programming and Scripting

awk to copy previous line matching a particular columns

Hello Help, 2356798 7689867 999 000 123678 20385907 9797 666 17978975 87468976 968978 98798 I am trying to have out put which actually look for the third column value of 9797 and then it insert line there after with first, second column value exactly as the previous line and replace the third... (3 Replies)
Discussion started by: Indra2011
3 Replies

9. Shell Programming and Scripting

Honey, I broke awk! (duplicate line removal in 30M line 3.7GB csv file)

I have a script that builds a database ~30 million lines, ~3.7 GB .cvs file. After multiple optimzations It takes about 62 min to bring in and parse all the files and used to take 10 min to remove duplicates until I was requested to add another column. I am using the highly optimized awk code: awk... (34 Replies)
Discussion started by: Michael Stora
34 Replies

10. UNIX for Dummies Questions & Answers

Print only the duplicate line only with matching columns

Hi There, I have an I/P which looks like -- 1 2 3 4 5 1 2 3 4 6 4 7 8 9 9 5 6 7 8 9 I would like O/P to be --- 1 2 3 4 5 1 2 3 4 6 So, printing only the consecutive lines where $1,$2,$3,$4 are matching. Is there any command to do this or small awk script? Thanks, (12 Replies)
Discussion started by: Indra2011
12 Replies
CALENDAR(3)						   BSD Library Functions Manual 					       CALENDAR(3)

NAME
easterg, easterog, easteroj, gdate, jdate, ndaysg, ndaysj, week, weekday -- Calendar arithmetic for the Christian era LIBRARY
Calendar Arithmetic Library (libcalendar, -lcalendar) SYNOPSIS
#include <calendar.h> struct date * easterg(int year, struct date *dt); struct date * easterog(int year, struct date *dt); struct date * easteroj(int year, struct date *dt); struct date * gdate(int nd, struct date *dt); struct date * jdate(int nd, struct date *dt); int ndaysg(struct date *dt); int ndaysj(struct date *dt); int week(int nd, int *year); int weekday(int nd); DESCRIPTION
These functions provide calendar arithmetic for a large range of years, starting at March 1st, year zero (i.e., 1 B.C.) and ending way beyond year 100000. Programs should be linked with -lcalendar. The functions easterg(), easterog() and easteroj() store the date of Easter Sunday into the structure pointed at by dt and return a pointer to this structure. The function easterg() assumes Gregorian Calendar (adopted by most western churches after 1582) and the functions easterog() and easteroj() compute the date of Easter Sunday according to the orthodox rules (Western churches before 1582, Greek and Russian Orthodox Church until today). The result returned by easterog() is the date in Gregorian Calendar, whereas easteroj() returns the date in Julian Calendar. The functions gdate(), jdate(), ndaysg() and ndaysj() provide conversions between the common "year, month, day" notation of a date and the "number of days" representation, which is better suited for calculations. The days are numbered from March 1st year 1 B.C., starting with zero, so the number of a day gives the number of days since March 1st, year 1 B.C. The conversions work for nonnegative day numbers only. The gdate() and jdate() functions store the date corresponding to the day number nd into the structure pointed at by dt and return a pointer to this structure. The ndaysg() and ndaysj() functions return the day number of the date pointed at by dt. The gdate() and ndaysg() functions assume Gregorian Calendar after October 4, 1582 and Julian Calendar before, whereas jdate() and ndaysj() assume Julian Calendar throughout. The two calendars differ by the definition of the leap year. The Julian Calendar says every year that is a multiple of four is a leap year. The Gregorian Calendar excludes years that are multiples of 100 and not multiples of 400. This means the years 1700, 1800, 1900, 2100 are not leap years and the year 2000 is a leap year. The new rules were inaugurated on October 4, 1582 by deleting ten days following this date. Most catholic countries adopted the new calendar by the end of the 16th century, whereas others stayed with the Julian Calendar until the 20th century. The United Kingdom and their colonies switched on September 2, 1752. They already had to delete 11 days. The function week() returns the number of the week which contains the day numbered nd. The argument *year is set with the year that contains (the greater part of) the week. The weeks are numbered per year starting with week 1, which is the first week in a year that includes more than three days of the year. Weeks start on Monday. This function is defined for Gregorian Calendar only. The function weekday() returns the weekday (Mo = 0 .. Su = 6) of the day numbered nd. The structure date is defined in <calendar.h>. It contains these fields: int y; /* year (0000 - ????) */ int m; /* month (1 - 12) */ int d; /* day of month (1 - 31) */ The year zero is written as "1 B.C." by historians and "0" by astronomers and in this library. SEE ALSO
ncal(1), strftime(3) STANDARDS
The week number conforms to ISO 8601: 1988. HISTORY
The calendar library first appeared in FreeBSD 3.0. AUTHORS
This manual page and the library was written by Wolfgang Helbig <helbig@FreeBSD.org>. BUGS
The library was coded with great care so there are no bugs left. BSD
November 29, 1997 BSD
All times are GMT -4. The time now is 06:51 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy