Sponsored Content
Top Forums Shell Programming and Scripting How to put the command to remove duplicate lines in my awk script? Post 303037748 by Don Cragun on Monday 12th of August 2019 05:06:10 PM
Old 08-12-2019
Please show us the exact output you hope to produce from the three sample files that we can assume were used to produce the output shown in the image you showed us in post #1 in this thread (preferably as text in CODE tags rather than as an image).

If you're trying to do what I think you are trying to do, you cannot use !a[$0]++ to filter your input because it will throw away input before you know whether or not you will want to print it. I think you need to read all of your input files and then compare the data for each LPARS value. If and only if all of the entries are identical, then you can decide not to print that row of output.

Note that in your sample input file shown in post #1 in this thread, you showed us two lines that seem to be in completely different formats. Please explain what the real format is for your input files.

It is also unclear as to whether or not all of the input files will contain an entry for each LPARS value. If a record for a specific LPARS value is not included in a file, should that be treated as a "different" value causing a line to be printed? Or should a file be ignored when determining whether or not to print an LPARS value line if there is no entry for that LPARS value in that file?
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

how to remove duplicate lines

I have following file content (3 fields each line): 23 888 10.0.0.1 dfh 787 10.0.0.2 dssf dgfas 10.0.0.3 dsgas dg 10.0.0.4 df dasa 10.0.0.5 df dag 10.0.0.5 dfd dfdas 10.0.0.5 dfd dfd 10.0.0.6 daf nfd 10.0.0.6 ... as can be seen, that the third field is ip address and sorted. but... (3 Replies)
Discussion started by: fredao
3 Replies

2. Shell Programming and Scripting

Command/Script to remove duplicate lines from the file?

Hello, Can anyone tell Command/Script to remove duplicate lines from the file? (2 Replies)
Discussion started by: Rahulpict
2 Replies

3. Shell Programming and Scripting

awk script to remove duplicate rows in line

i have the long file more than one ns and www and mx in the line like . i need the first ns record and first www and first mx from line . the records are seperated with tthe ; i am try ing in awk scripting not getiing the solution. ... (4 Replies)
Discussion started by: kiranmosarla
4 Replies

4. Shell Programming and Scripting

Command to remove duplicate lines with perl,sed,awk

Input: hello hello hello hello monkey donkey hello hello drink dance drink Output should be: hello hello monkey donkey drink dance (9 Replies)
Discussion started by: cola
9 Replies

5. Shell Programming and Scripting

remove duplicate lines using awk

Hi, I came to know that using awk '!x++' removes the duplicate lines. Can anyone please explain the above syntax. I want to understand how the above awk syntax removes the duplicates. Thanks in advance, sudvishw :confused: (7 Replies)
Discussion started by: sudvishw
7 Replies

6. Shell Programming and Scripting

Remove duplicate lines

Hi, I have a huge file which is about 50GB. There are many lines. The file format likes 21 rs885550 0 9887804 C C T C C C C C C C 21 rs210498 0 9928860 0 0 C C 0 0 0 0 0 0 21 rs303304 0 9941889 A A A A A A A A A A 22 rs303304 0 9941890 0 A A A A A A A A A The question is that there are a few... (4 Replies)
Discussion started by: zhshqzyc
4 Replies

7. Shell Programming and Scripting

[uniq + awk?] How to remove duplicate blocks of lines in files?

Hello again, I am wanting to remove all duplicate blocks of XML code in a file. This is an example: input: <string-array name="threeItems"> <item>item1</item> <item>item2</item> <item>item3</item> </string-array> <string-array name="twoItems"> <item>item1</item> <item>item2</item>... (19 Replies)
Discussion started by: raidzero
19 Replies

8. Shell Programming and Scripting

AWK Command to duplicate lines in a file?

Hi, I have a file with date in it like: UserString1 UserString2 UserString3 UserString4 UserString5 I need two entries for each line so it reads like UserString1 UserString1 UserString2 UserString2 etc. Can someone help me with the awk command please? Thanks (4 Replies)
Discussion started by: Grueben
4 Replies

9. Shell Programming and Scripting

Cant get awk 1liner to remove duplicate lines from Delimited file, get "event not found" error..help

Hi, I am on a Solaris8 machine If someone can help me with adjusting this awk 1 liner (turning it into a real awkscript) to get by this "event not found error" ...or Present Perl solution code that works for Perl5.8 in the csh shell ...that would be great. ****************** ... (3 Replies)
Discussion started by: andy b
3 Replies

10. Shell Programming and Scripting

How to remove duplicate lines?

Hi All, I am storing the result in the variable result_text using the below code. result_text=$(printf "$result_text\t\n$name") The result_text is having the below text. Which is having duplicate lines. file and time for the interval 03:30 - 03:45 file and time for the interval 03:30 - 03:45 ... (4 Replies)
Discussion started by: nalu
4 Replies
COMPFACE(1)						      General Commands Manual						       COMPFACE(1)

NAME
compface, uncompface - compress and expand 48x48x1 face image files SYNOPSIS
compface [ infile [ outfile ] ] ... uncompface [-X] [ infile [ outfile ] ] ... DESCRIPTION
compface is a filter for generating highly compressed representations of 48x48x1 face image files. uncompface is an inverse filter which performs an inverse transformation with no loss of data. The algorithm used is highly tuned for its purpose and achieves better than a five to one compression ratio on average. Arguments operate identically for both programmes. The first argument, if present, causes input to be taken from the named file instead of from standard input unless it is the string ``-''. A second argument, if present, causes output to go to the named file instead of to standard output unless it is the string ``-''. Subsequent pairs of arguments may be used to specify further pairs of input and output files. The input format for compface (and the output format for uncompface) is 48 lines each of 3 sixteen bit hexadecimal integers, comma termi- nated in C initialiser style. The output format of compface (and the input format for uncompface) is some number of lines made up of a space followed by printable characters (in the range ``!'' to ``~'' inclusive). The first line contains 72 characters and following lines contain 79 characters except that the last line may be short. This version of compface has been patched to also be able to handle normal XBM images. uncompface will produce XBM output only if the -X switch is applied. The amount of compression obtained varies between face image files but the output of compface averages less than 200 characters. The aver- age number of output lines is three. DIAGNOSTICS
Exit status is normally 0. Possible errors include IO errors when opening, reading or writing files, format errors in compface input files and invalid arguments. Each results in an explanatory message on standard error and an exit status of 1. A warning will be produced if compface detects extra input data. BUGS
Nominating standard input as input more than once does not work. SEE ALSO
Compface(3) 25 January 1990 COMPFACE(1)
All times are GMT -4. The time now is 09:38 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy