Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: How to put the command to remove duplicate lines in my awk script?
Top Forums Shell Programming and Scripting How to put the command to remove duplicate lines in my awk script? Post 303037666 by Tim2424 on Friday 9th of August 2019 06:05:56 AM
Old 08-09-2019
Hello !

Thank you for your answer !

Quote:
Note that the awk code '!a[$0]++' will discard duplicated lines within all of the files processed by a single invocation of awk. But, since you are processing each of your 276 CSV input files in a separate invocation of awk there is no way for any of those invocations of awk to compare any input values from one input file to any input values from any other input file.
That is why I decide to change my script.

For the moment, I have this :


Code:
awk -F",|;" ' {$0=$1","$2","$5","$6","$7 } /'$test'/ { if (!a[$0]++)
{
print ""
printf "LPARS : %s\n", $2
printf "RAM : %s\n", $3
printf "CPU1 : %s\n", $4
printf "CPU2 : %s\n", $5
print ""
}
}' /var/www/cgi-bin/LPAR_MAP/*.csv ;

And it works.

But I don't know how to put :

Code:
NR==1 { 
        split(FILENAME ,a,"[-.]");
     }


and

Code:
$0 ~ test {
          if(!header++){
              print "DATE ========================== : " a[4] 
          }


Any ideas ?
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

how to remove duplicate lines

I have following file content (3 fields each line): 23 888 10.0.0.1 dfh 787 10.0.0.2 dssf dgfas 10.0.0.3 dsgas dg 10.0.0.4 df dasa 10.0.0.5 df dag 10.0.0.5 dfd dfdas 10.0.0.5 dfd dfd 10.0.0.6 daf nfd 10.0.0.6 ... as can be seen, that the third field is ip address and sorted. but... (3 Replies)
Discussion started by: fredao
3 Replies

2. Shell Programming and Scripting

Command/Script to remove duplicate lines from the file?

Hello, Can anyone tell Command/Script to remove duplicate lines from the file? (2 Replies)
Discussion started by: Rahulpict
2 Replies

3. Shell Programming and Scripting

awk script to remove duplicate rows in line

i have the long file more than one ns and www and mx in the line like . i need the first ns record and first www and first mx from line . the records are seperated with tthe ; i am try ing in awk scripting not getiing the solution. ... (4 Replies)
Discussion started by: kiranmosarla
4 Replies

4. Shell Programming and Scripting

Command to remove duplicate lines with perl,sed,awk

Input: hello hello hello hello monkey donkey hello hello drink dance drink Output should be: hello hello monkey donkey drink dance (9 Replies)
Discussion started by: cola
9 Replies

5. Shell Programming and Scripting

remove duplicate lines using awk

Hi, I came to know that using awk '!x++' removes the duplicate lines. Can anyone please explain the above syntax. I want to understand how the above awk syntax removes the duplicates. Thanks in advance, sudvishw :confused: (7 Replies)
Discussion started by: sudvishw
7 Replies

6. Shell Programming and Scripting

Remove duplicate lines

Hi, I have a huge file which is about 50GB. There are many lines. The file format likes 21 rs885550 0 9887804 C C T C C C C C C C 21 rs210498 0 9928860 0 0 C C 0 0 0 0 0 0 21 rs303304 0 9941889 A A A A A A A A A A 22 rs303304 0 9941890 0 A A A A A A A A A The question is that there are a few... (4 Replies)
Discussion started by: zhshqzyc
4 Replies

7. Shell Programming and Scripting

[uniq + awk?] How to remove duplicate blocks of lines in files?

Hello again, I am wanting to remove all duplicate blocks of XML code in a file. This is an example: input: <string-array name="threeItems"> <item>item1</item> <item>item2</item> <item>item3</item> </string-array> <string-array name="twoItems"> <item>item1</item> <item>item2</item>... (19 Replies)
Discussion started by: raidzero
19 Replies

8. Shell Programming and Scripting

AWK Command to duplicate lines in a file?

Hi, I have a file with date in it like: UserString1 UserString2 UserString3 UserString4 UserString5 I need two entries for each line so it reads like UserString1 UserString1 UserString2 UserString2 etc. Can someone help me with the awk command please? Thanks (4 Replies)
Discussion started by: Grueben
4 Replies

9. Shell Programming and Scripting

Cant get awk 1liner to remove duplicate lines from Delimited file, get "event not found" error..help

Hi, I am on a Solaris8 machine If someone can help me with adjusting this awk 1 liner (turning it into a real awkscript) to get by this "event not found error" ...or Present Perl solution code that works for Perl5.8 in the csh shell ...that would be great. ****************** ... (3 Replies)
Discussion started by: andy b
3 Replies

10. Shell Programming and Scripting

How to remove duplicate lines?

Hi All, I am storing the result in the variable result_text using the below code. result_text=$(printf "$result_text\t\n$name") The result_text is having the below text. Which is having duplicate lines. file and time for the interval 03:30 - 03:45 file and time for the interval 03:30 - 03:45 ... (4 Replies)
Discussion started by: nalu
4 Replies

LEARN ABOUT DEBIAN

fntsample

fntsample(1)						      General Commands Manual						      fntsample(1)

NAME

       fntsample - PDF and PostScript font samples generator

SYNOPSIS

       fntsample [ OPTIONS ] -f FONT-FILE -o OUTPUT-FILE
       fntsample -h

DESCRIPTION

       fntsample  program  can	be  used  to generate font samples that show Unicode coverage of the font and are similar in appearance to Unicode
       charts. Samples can be saved into PDF (default) or PostScript file.

OPTIONS

       fntsample supports the following options.

       --font-file, -f FONT-FILE
	      Make samples of FONT-FILE.

       --font-index, -n IDX
	      Font index for FONT-FILE specified using --font-file option.  Useful for files that contain multiple fonts,  like  TrueType  Collec-
	      tions (.ttc).  By default font with index 0 is used.

       --output-file, -o OUTPUT-FILE
	      Write output to OUTPUT-FILE.

       --other-font-file, -d OTHER-FONT
	      Compare FONT-FILE with OTHER-FONT.  Glyphs added to FONT-FILE will be highlighted.

       --other-index, -m IDX
	      Font index for OTHER-FONT specified using --other-font-file option.

       --postscript-output, -s
	      Use PostScript format for output instead of PDF.

       --svg, -g
	      Use SVG format for output.  The generated document contains one page.  Use range selection options to specify which.

       --print-outline, -l
	      Print  document  outlines data to standard output.  This data can be used to add outlines (aka bookmarks) to resulting PDF file with
	      pdfoutline program.

       --include-range, -i RANGE
	      Show characters in RANGE.

       --exclude-range, -x RANGE
	      Do not show characters in RANGE.

       --style, -t "STYLE: VAL"
	      Set STYLE to value VAL.  Run fntsample with option --help to see list of styles and default values.

       --help, -h
	      Display help text and exit.

       Parameter RANGE for -i and -x can be given as one integer or a pair of integers delimited by minus sign (-).  Integers can be specified	in
       decimal,  hexadecimal  (0x...) or octal (0...) format.  One integer of a pair can be missing (-N can be used to specify all characters with
       codes less or equal to N, and N- for all characters with codes greather or equal to N).	Multiple -i and -x options can be used.

EXAMPLES

       Make PDF samples for font.ttf and write them to file samples.pdf:
	      fntsample -f font.ttf -o samples.pdf

       Make PDF samples for font.ttf, compare it with oldfont.ttf and highlight new glyphs. Write output to file samples.pdf:
	      fntsample -f font.ttf -d oldfont.ttf -o samples.pdf

       Make PostScript samples for font.ttf and write output to file samples.ps.  Show only glyphs for characters with	codes  less  or  equal	to
       U+04FF but exclude U+0370-U+03FF:
	      fntsample -f font.ttf -s -o samples.ps -i -0x04FF -x 0x0370-0x03FF

       Make PDF samples for font.ttf and save output to file samples.pdf adding outlines to it:
	      fntsample -f font.ttf -o temp.pdf -l > outlines.txt
	      pdfoutline temp.pdf outlines.txt samples.pdf

AUTHOR

       Copyright (C) 2007 Eugeniy Meshcheryakov <eugen@debian.org>
       Homepage: <http://fntsample.sourceforge.net/>

SEE ALSO

       pdfoutline(1)

								    2010-10-14							      fntsample(1)
All times are GMT -4. The time now is 11:13 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy