Sponsored Content
Top Forums Shell Programming and Scripting awk - Remove duplicates during array build Post 302976805 by RudiC on Wednesday 6th of July 2016 11:45:58 AM
Old 07-06-2016
There shouldn't be any NR == FNR nor NR != FNR; I simply put in FNR == 1 to exclude the header line(s). The scriptlet should work on any number of files supplied to it as one single stream of data (unless you terribly messed up something).
This User Gave Thanks to RudiC For This Post:
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Arranging an array so that duplicates will turn up first

Hi All, I have an array that contains duplicates as well unique numbers. ex- (21, 33, 35, 21, 33, 70, 33, 35, 50) I need to arrange it in such a way that all the duplicates will come up first followed by unique numbers. Result for the given example should be: (21, 21, 33, 33, 35, 35, 70,... (4 Replies)
Discussion started by: ashim
4 Replies

2. Shell Programming and Scripting

Remove duplicates

Hello Experts, I have two files named old and new. Below are my example files. I need to compare and print the records that only exist in my new file. I tried the below awk script, this script works perfectly well if the records have exact match, the issue I have is my old file has got extra... (4 Replies)
Discussion started by: forumthreads
4 Replies

3. Shell Programming and Scripting

bash - remove duplicates

I need to use a bash script to remove duplicate files from a download list, but I cannot use uniq because the urls are different. I need to go from this: http://***/fae78fe/file1.wmv http://***/39du7si/file1.wmv http://***/d8el2hd/file2.wmv http://***/h893js3/file2.wmv to this: ... (2 Replies)
Discussion started by: locoroco
2 Replies

4. Shell Programming and Scripting

Awk: Remove Duplicates

I have the following code for removing duplicate records based on fields in inputfile file & moves the duplicate records in duplicates file(1st Awk) & in 2nd awk i fetch the non duplicate entries in inputfile to tmp file and use move to update the original file. Requirement: Can both the awk... (4 Replies)
Discussion started by: siramitsharma
4 Replies

5. Shell Programming and Scripting

awk remove first duplicates

Hi All, I have searched many threads for possible close solution. But I was unable to get simlar scenario. I would like to print all duplicate based on 3rd column except the first occurance. Also would like to print if it is single entry(non-duplicate). i/P file 12 NIL ABD LON 11 NIL ABC... (6 Replies)
Discussion started by: sybadm
6 Replies

6. Shell Programming and Scripting

Help with merge and remove duplicates

Hi all, I need some help to remove duplicates from a file before merging. I have got 2 files: file1 has data in format 4300 23456 4301 2357 the 4 byte values on the right hand side is uniq, and are not repeated anywhere in the file file 2 has data in same format but is not in... (10 Replies)
Discussion started by: roy121
10 Replies

7. Shell Programming and Scripting

Remove duplicates

I have a file with the following format: fields seperated by "|" title1|something class|long...content1|keys title2|somhing class|log...content1|kes title1|sothing class|lon...content1|kes title3|shing cls|log...content1|ks I want to remove all duplicates with the same "title field"(the... (3 Replies)
Discussion started by: dtdt
3 Replies

8. Shell Programming and Scripting

Remove top 3 duplicates

hello , I have a requirement with input in below format abc 123 xyz bcd 365 kii abc 987 876 cdf 987 uii abc 456 yuu bcd 654 rrr Expecting Output abc 456 yuu bcd 654 rrr cdf 987 uii (1 Reply)
Discussion started by: Tomlight
1 Replies

9. Shell Programming and Scripting

Remove duplicates

Hi I have a below file structure. 200,1245,E1,1,E1,,7611068,KWH,30, ,,,,,,,, 200,1245,E1,1,E1,,7611070,KWH,30, ,,,,,,,, 300,20140223,0.001,0.001,0.001,0.001,0.001 300,20140224,0.001,0.001,0.001,0.001,0.001 300,20140225,0.001,0.001,0.001,0.001,0.001 300,20140226,0.001,0.001,0.001,0.001,0.001... (1 Reply)
Discussion started by: tejashavele
1 Replies

10. Shell Programming and Scripting

How to remove duplicates in C shell Array?

Please help me on this My script name is uniqueArray.csh #!/bin/csh set ARRAY = ( one teo three one three ) set ARRAY = ( $ARRAY one five three five ) How to remove the duplicates in this array ,sort and save those in the same variable or different variable. Thanks in the advance ... (3 Replies)
Discussion started by: SA_Palani
3 Replies
halibut(1)							   Simon Tatham 							halibut(1)

NAME
halibut - multi-format documentation formatting tool SYNOPSIS
halibut [options] file1.but [file2.but ...] DESCRIPTION
halibut reads the given set of input files, assembles them into a document, and outputs that document in one or more formats. The available command-line options can configure what formats Halibut should output in, and can also configure other things about the way Halibut works. OPTIONS
The command-line options supported by halibut are: --text[=filename] Makes Halibut generate an output file in plain text format. If the optional filename parameter is supplied, the output text file will be given that name. Otherwise, the name of the output text file will be as specified in the input files, or output.txt if none is specified at all. --html[=filename] Makes Halibut generate one or more output files in HTML format. If the optional filename parameter is supplied, there will be pre- cisely one HTML output file with that name, containing the whole document. Otherwise, there may be one or more than one HTML file produced as output; this, and the file names, will be as specified in the input files, or given a set of default names starting with Contents.html if none is specified at all. --winhelp[=filename] Makes Halibut generate an output file in Windows Help format. If the optional filename parameter is supplied, the output help file will be given that name. Otherwise, the name of the output help file will be as specified in the input files, or output.hlp if none is specified at all. The output help file must have a name ending in .hlp; if it does not, .hlp will be added. A secondary contents file will be created alongside the main help file, with the same name except that it will end in .cnt (for example output.cnt, if the main file is out- put.hlp). --man[=filename] Makes Halibut generate an output file in Unix man page format. If the optional filename parameter is supplied, the output man page will be given that name. Otherwise, the name of the output man page will be as specified in the input files, or output.1 if none is specified at all. --info[=filename] Makes Halibut generate an info file. If the optional filename parameter is supplied, the output info file will be given that name. Otherwise, the name of the output info file will be as specified in the input files, or output.info if none is specified at all. By default, unless configured otherwise using the cfg{info-max-file-size}{0} directive, the info output will be split into multiple files. The main file will have the name you specify; the subsidiary files will have suffixes -1, -2 etc. --pdf[=filename] Makes Halibut generate an output file in PDF format. If the optional filename parameter is supplied, the PDF output file will be given that name. Otherwise, the name of the output PDF file will be as specified in the input files, or output.pdf if none is speci- fied at all. --ps[=filename] Makes Halibut generate an output file in PostScript format. If the optional filename parameter is supplied, the PostScript output file will be given that name. Otherwise, the name of the output PostScript file will be as specified in the input files, or out- put.ps if none is specified at all. -Cword:word[:word...] Adds a configuration directive to the input processed by Halibut. Using this directive is exactly equivalent to appending an extra input file to the command line which contains the directive cfg{word}{word}{word...}. --input-charset=charset Changes the assumed character set for input files from the default of ASCII. --list-charsets Makes Halibut list character sets known to it. --list-fonts Makes Halibut list fonts known to it, either intrinsically or by being passed as input files. --precise Makes Halibut report the column number as well as the line number when it encounters an error in an input file. --help Makes Halibut display a brief summary of its command-line options. --version Makes Halibut report its version number. --licence Makes Halibut display its licence (MIT). MORE INFORMATION
For more information on Halibut, including full details of the input file format, look in the full manual. If this is not installed locally on your system, you can also find it at the Halibut web site: http://www.chiark.greenend.org.uk/~sgtatham/halibut/ BUGS
This man page isn't terribly complete. Halibut 2004-04-08 halibut(1)
All times are GMT -4. The time now is 09:06 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy