Sponsored Content
Top Forums Shell Programming and Scripting awk - Remove duplicates during array build Post 302976805 by RudiC on Wednesday 6th of July 2016 11:45:58 AM
Old 07-06-2016
There shouldn't be any NR == FNR nor NR != FNR; I simply put in FNR == 1 to exclude the header line(s). The scriptlet should work on any number of files supplied to it as one single stream of data (unless you terribly messed up something).
This User Gave Thanks to RudiC For This Post:
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Arranging an array so that duplicates will turn up first

Hi All, I have an array that contains duplicates as well unique numbers. ex- (21, 33, 35, 21, 33, 70, 33, 35, 50) I need to arrange it in such a way that all the duplicates will come up first followed by unique numbers. Result for the given example should be: (21, 21, 33, 33, 35, 35, 70,... (4 Replies)
Discussion started by: ashim
4 Replies

2. Shell Programming and Scripting

Remove duplicates

Hello Experts, I have two files named old and new. Below are my example files. I need to compare and print the records that only exist in my new file. I tried the below awk script, this script works perfectly well if the records have exact match, the issue I have is my old file has got extra... (4 Replies)
Discussion started by: forumthreads
4 Replies

3. Shell Programming and Scripting

bash - remove duplicates

I need to use a bash script to remove duplicate files from a download list, but I cannot use uniq because the urls are different. I need to go from this: http://***/fae78fe/file1.wmv http://***/39du7si/file1.wmv http://***/d8el2hd/file2.wmv http://***/h893js3/file2.wmv to this: ... (2 Replies)
Discussion started by: locoroco
2 Replies

4. Shell Programming and Scripting

Awk: Remove Duplicates

I have the following code for removing duplicate records based on fields in inputfile file & moves the duplicate records in duplicates file(1st Awk) & in 2nd awk i fetch the non duplicate entries in inputfile to tmp file and use move to update the original file. Requirement: Can both the awk... (4 Replies)
Discussion started by: siramitsharma
4 Replies

5. Shell Programming and Scripting

awk remove first duplicates

Hi All, I have searched many threads for possible close solution. But I was unable to get simlar scenario. I would like to print all duplicate based on 3rd column except the first occurance. Also would like to print if it is single entry(non-duplicate). i/P file 12 NIL ABD LON 11 NIL ABC... (6 Replies)
Discussion started by: sybadm
6 Replies

6. Shell Programming and Scripting

Help with merge and remove duplicates

Hi all, I need some help to remove duplicates from a file before merging. I have got 2 files: file1 has data in format 4300 23456 4301 2357 the 4 byte values on the right hand side is uniq, and are not repeated anywhere in the file file 2 has data in same format but is not in... (10 Replies)
Discussion started by: roy121
10 Replies

7. Shell Programming and Scripting

Remove duplicates

I have a file with the following format: fields seperated by "|" title1|something class|long...content1|keys title2|somhing class|log...content1|kes title1|sothing class|lon...content1|kes title3|shing cls|log...content1|ks I want to remove all duplicates with the same "title field"(the... (3 Replies)
Discussion started by: dtdt
3 Replies

8. Shell Programming and Scripting

Remove top 3 duplicates

hello , I have a requirement with input in below format abc 123 xyz bcd 365 kii abc 987 876 cdf 987 uii abc 456 yuu bcd 654 rrr Expecting Output abc 456 yuu bcd 654 rrr cdf 987 uii (1 Reply)
Discussion started by: Tomlight
1 Replies

9. Shell Programming and Scripting

Remove duplicates

Hi I have a below file structure. 200,1245,E1,1,E1,,7611068,KWH,30, ,,,,,,,, 200,1245,E1,1,E1,,7611070,KWH,30, ,,,,,,,, 300,20140223,0.001,0.001,0.001,0.001,0.001 300,20140224,0.001,0.001,0.001,0.001,0.001 300,20140225,0.001,0.001,0.001,0.001,0.001 300,20140226,0.001,0.001,0.001,0.001,0.001... (1 Reply)
Discussion started by: tejashavele
1 Replies

10. Shell Programming and Scripting

How to remove duplicates in C shell Array?

Please help me on this My script name is uniqueArray.csh #!/bin/csh set ARRAY = ( one teo three one three ) set ARRAY = ( $ARRAY one five three five ) How to remove the duplicates in this array ,sort and save those in the same variable or different variable. Thanks in the advance ... (3 Replies)
Discussion started by: SA_Palani
3 Replies
GIT-SH-SETUP(1) 						    Git Manual							   GIT-SH-SETUP(1)

NAME
git-sh-setup - Common Git shell script setup code SYNOPSIS
. "$(git --exec-path)/git-sh-setup" DESCRIPTION
This is not a command the end user would want to run. Ever. This documentation is meant for people who are studying the Porcelain-ish scripts and/or are writing new ones. The git sh-setup scriptlet is designed to be sourced (using .) by other shell scripts to set up some variables pointing at the normal Git directories and a few helper shell functions. Before sourcing it, your script should set up a few variables; USAGE (and LONG_USAGE, if any) is used to define message given by usage() shell function. SUBDIRECTORY_OK can be set if the script can run from a subdirectory of the working tree (some commands do not). The scriptlet sets GIT_DIR and GIT_OBJECT_DIRECTORY shell variables, but does not export them to the environment. FUNCTIONS
die exit after emitting the supplied error message to the standard error stream. usage die with the usage message. set_reflog_action set the message that will be recorded to describe the end-user action in the reflog, when the script updates a ref. git_editor runs an editor of user's choice (GIT_EDITOR, core.editor, VISUAL or EDITOR) on a given file, but error out if no editor is specified and the terminal is dumb. is_bare_repository outputs true or false to the standard output stream to indicate if the repository is a bare repository (i.e. without an associated working tree). cd_to_toplevel runs chdir to the toplevel of the working tree. require_work_tree checks if the current directory is within the working tree of the repository, and otherwise dies. require_work_tree_exists checks if the working tree associated with the repository exists, and otherwise dies. Often done before calling cd_to_toplevel, which is impossible to do if there is no working tree. require_clean_work_tree <action> [<hint>] checks that the working tree and index associated with the repository have no uncommitted changes to tracked files. Otherwise it emits an error message of the form Cannot <action>: <reason>. <hint>, and dies. Example: require_clean_work_tree rebase "Please commit or stash them." get_author_ident_from_commit outputs code for use with eval to set the GIT_AUTHOR_NAME, GIT_AUTHOR_EMAIL and GIT_AUTHOR_DATE variables for a given commit. create_virtual_base modifies the first file so only lines in common with the second file remain. If there is insufficient common material, then the first file is left empty. The result is suitable as a virtual base input for a 3-way merge. GIT
Part of the git(1) suite Git 1.8.3.1 06/10/2014 GIT-SH-SETUP(1)
All times are GMT -4. The time now is 06:45 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy