Sponsored Content
Top Forums Shell Programming and Scripting awk - Remove duplicates during array build Post 302976805 by RudiC on Wednesday 6th of July 2016 11:45:58 AM
Old 07-06-2016
There shouldn't be any NR == FNR nor NR != FNR; I simply put in FNR == 1 to exclude the header line(s). The scriptlet should work on any number of files supplied to it as one single stream of data (unless you terribly messed up something).
This User Gave Thanks to RudiC For This Post:
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Arranging an array so that duplicates will turn up first

Hi All, I have an array that contains duplicates as well unique numbers. ex- (21, 33, 35, 21, 33, 70, 33, 35, 50) I need to arrange it in such a way that all the duplicates will come up first followed by unique numbers. Result for the given example should be: (21, 21, 33, 33, 35, 35, 70,... (4 Replies)
Discussion started by: ashim
4 Replies

2. Shell Programming and Scripting

Remove duplicates

Hello Experts, I have two files named old and new. Below are my example files. I need to compare and print the records that only exist in my new file. I tried the below awk script, this script works perfectly well if the records have exact match, the issue I have is my old file has got extra... (4 Replies)
Discussion started by: forumthreads
4 Replies

3. Shell Programming and Scripting

bash - remove duplicates

I need to use a bash script to remove duplicate files from a download list, but I cannot use uniq because the urls are different. I need to go from this: http://***/fae78fe/file1.wmv http://***/39du7si/file1.wmv http://***/d8el2hd/file2.wmv http://***/h893js3/file2.wmv to this: ... (2 Replies)
Discussion started by: locoroco
2 Replies

4. Shell Programming and Scripting

Awk: Remove Duplicates

I have the following code for removing duplicate records based on fields in inputfile file & moves the duplicate records in duplicates file(1st Awk) & in 2nd awk i fetch the non duplicate entries in inputfile to tmp file and use move to update the original file. Requirement: Can both the awk... (4 Replies)
Discussion started by: siramitsharma
4 Replies

5. Shell Programming and Scripting

awk remove first duplicates

Hi All, I have searched many threads for possible close solution. But I was unable to get simlar scenario. I would like to print all duplicate based on 3rd column except the first occurance. Also would like to print if it is single entry(non-duplicate). i/P file 12 NIL ABD LON 11 NIL ABC... (6 Replies)
Discussion started by: sybadm
6 Replies

6. Shell Programming and Scripting

Help with merge and remove duplicates

Hi all, I need some help to remove duplicates from a file before merging. I have got 2 files: file1 has data in format 4300 23456 4301 2357 the 4 byte values on the right hand side is uniq, and are not repeated anywhere in the file file 2 has data in same format but is not in... (10 Replies)
Discussion started by: roy121
10 Replies

7. Shell Programming and Scripting

Remove duplicates

I have a file with the following format: fields seperated by "|" title1|something class|long...content1|keys title2|somhing class|log...content1|kes title1|sothing class|lon...content1|kes title3|shing cls|log...content1|ks I want to remove all duplicates with the same "title field"(the... (3 Replies)
Discussion started by: dtdt
3 Replies

8. Shell Programming and Scripting

Remove top 3 duplicates

hello , I have a requirement with input in below format abc 123 xyz bcd 365 kii abc 987 876 cdf 987 uii abc 456 yuu bcd 654 rrr Expecting Output abc 456 yuu bcd 654 rrr cdf 987 uii (1 Reply)
Discussion started by: Tomlight
1 Replies

9. Shell Programming and Scripting

Remove duplicates

Hi I have a below file structure. 200,1245,E1,1,E1,,7611068,KWH,30, ,,,,,,,, 200,1245,E1,1,E1,,7611070,KWH,30, ,,,,,,,, 300,20140223,0.001,0.001,0.001,0.001,0.001 300,20140224,0.001,0.001,0.001,0.001,0.001 300,20140225,0.001,0.001,0.001,0.001,0.001 300,20140226,0.001,0.001,0.001,0.001,0.001... (1 Reply)
Discussion started by: tejashavele
1 Replies

10. Shell Programming and Scripting

How to remove duplicates in C shell Array?

Please help me on this My script name is uniqueArray.csh #!/bin/csh set ARRAY = ( one teo three one three ) set ARRAY = ( $ARRAY one five three five ) How to remove the duplicates in this array ,sort and save those in the same variable or different variable. Thanks in the advance ... (3 Replies)
Discussion started by: SA_Palani
3 Replies
GIT-LS-FILES(1) 						    Git Manual							   GIT-LS-FILES(1)

NAME
git-ls-files - Show information about files in the index and the working tree SYNOPSIS
git ls-files [-z] [-t] [-v] (--[cached|deleted|others|ignored|stage|unmerged|killed|modified])* (-[c|d|o|i|s|u|k|m])* [-x <pattern>|--exclude=<pattern>] [-X <file>|--exclude-from=<file>] [--exclude-per-directory=<file>] [--exclude-standard] [--error-unmatch] [--with-tree=<tree-ish>] [--full-name] [--abbrev] [--] [<file>...] DESCRIPTION
This merges the file listing in the directory cache index with the actual working directory list, and shows different combinations of the two. One or more of the options below may be used to determine the files shown: OPTIONS
-c, --cached Show cached files in the output (default) -d, --deleted Show deleted files in the output -m, --modified Show modified files in the output -o, --others Show other (i.e. untracked) files in the output -i, --ignored Show only ignored files in the output. When showing files in the index, print only those matched by an exclude pattern. When showing "other" files, show only those matched by an exclude pattern. -s, --stage Show staged contents' object name, mode bits and stage number in the output. --directory If a whole directory is classified as "other", show just its name (with a trailing slash) and not its whole contents. --no-empty-directory Do not list empty directories. Has no effect without --directory. -u, --unmerged Show unmerged files in the output (forces --stage) -k, --killed Show files on the filesystem that need to be removed due to file/directory conflicts for checkout-index to succeed. -z line termination on output. -x <pattern>, --exclude=<pattern> Skip untracked files matching pattern. Note that pattern is a shell wildcard pattern. See EXCLUDE PATTERNS below for more information. -X <file>, --exclude-from=<file> Read exclude patterns from <file>; 1 per line. --exclude-per-directory=<file> Read additional exclude patterns that apply only to the directory and its subdirectories in <file>. --exclude-standard Add the standard Git exclusions: .git/info/exclude, .gitignore in each directory, and the user's global exclusion file. --error-unmatch If any <file> does not appear in the index, treat this as an error (return 1). --with-tree=<tree-ish> When using --error-unmatch to expand the user supplied <file> (i.e. path pattern) arguments to paths, pretend that paths which were removed in the index since the named <tree-ish> are still present. Using this option with -s or -u options does not make any sense. -t This feature is semi-deprecated. For scripting purpose, git-status(1)--porcelain and git-diff-files(1)--name-status are almost always superior alternatives, and users should look at git-status(1)--short or git-diff(1)--name-status for more user-friendly alternatives. This option identifies the file status with the following tags (followed by a space) at the start of each line: H cached S skip-worktree M unmerged R removed/deleted C modified/changed K to be killed ? other -v Similar to -t, but use lowercase letters for files that are marked as assume unchanged (see git-update-index(1)). --full-name When run from a subdirectory, the command usually outputs paths relative to the current directory. This option forces paths to be output relative to the project top directory. --abbrev[=<n>] Instead of showing the full 40-byte hexadecimal object lines, show only a partial prefix. Non default number of digits can be specified with --abbrev=<n>. --debug After each line that describes a file, add more data about its cache entry. This is intended to show as much information as possible for manual inspection; the exact format may change at any time. -- Do not interpret any more arguments as options. <file> Files to show. If no files are given all files which match the other specified criteria are shown. OUTPUT
git ls-files just outputs the filenames unless --stage is specified in which case it outputs: [<tag> ]<mode> <object> <stage> <file> git ls-files --unmerged and git ls-files --stage can be used to examine detailed information on unmerged paths. For an unmerged path, instead of recording a single mode/SHA-1 pair, the index records up to three such pairs; one from tree O in stage 1, A in stage 2, and B in stage 3. This information can be used by the user (or the porcelain) to see what should eventually be recorded at the path. (see git-read-tree(1) for more information on state) When -z option is not used, TAB, LF, and backslash characters in pathnames are represented as , , and \, respectively. EXCLUDE PATTERNS
git ls-files can use a list of "exclude patterns" when traversing the directory tree and finding files to show when the flags --others or --ignored are specified. gitignore(5) specifies the format of exclude patterns. These exclude patterns come from these places, in order: 1. The command line flag --exclude=<pattern> specifies a single pattern. Patterns are ordered in the same order they appear in the command line. 2. The command line flag --exclude-from=<file> specifies a file containing a list of patterns. Patterns are ordered in the same order they appear in the file. 3. The command line flag --exclude-per-directory=<name> specifies a name of the file in each directory git ls-files examines, normally .gitignore. Files in deeper directories take precedence. Patterns are ordered in the same order they appear in the files. A pattern specified on the command line with --exclude or read from the file specified with --exclude-from is relative to the top of the directory tree. A pattern read from a file specified by --exclude-per-directory is relative to the directory that the pattern file appears in. SEE ALSO
git-read-tree(1), gitignore(5) GIT
Part of the git(1) suite Git 1.8.3.1 06/10/2014 GIT-LS-FILES(1)
All times are GMT -4. The time now is 11:41 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy