Sponsored Content
Top Forums Shell Programming and Scripting How to : Find duplicate number from file? with bash Post 302268432 by joeyg on Monday 15th of December 2008 05:59:06 PM
Old 12-15-2008
Question

Those five lines are unique.
Or, are you referring to the number after the : delimiter?
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Counting The Number Of Duplicate Lines In a File

Hello. First time poster here. I have a huge file of IP numbers. I am trying to output only the class b of the IPs and rank them by most common and output the total # of duplicate class b's before the class b. An example is below: 12.107.1.1 12.107.9.54 12.108.3.89 12.109.109.4 12.109.6.3 ... (2 Replies)
Discussion started by: crunchtime
2 Replies

2. Shell Programming and Scripting

Bash Script duplicate file names

I am trying to write a housekeeping bash script. Part of it involves searching all of my attached storage media for photographs and moving them into a single directory. The problem occurs when files have duplicate names, obviously a file called 001.jpg will get overwritten with another file... (6 Replies)
Discussion started by: stumpyuk
6 Replies

3. Shell Programming and Scripting

find out duplicate records in file?

Dear All, I have one file which looks like : account1:passwd1 account2:passwd2 account3:passwd3 account1:passwd4 account5:passwd5 account6:passwd6 you can see there're two records for account1. and is there any shell command which can find out : account1 is the duplicate record in... (3 Replies)
Discussion started by: tiger2000
3 Replies

4. Shell Programming and Scripting

Find Duplicate records in first Column in File

Hi, Need to find a duplicate records on the first column, ANU4501710430989 0000000W20389390 ANU4501710430989 0000000W67065483 ANU4501130050520 0000000W80838713 ANU4501210170685 0000000W69246611... (3 Replies)
Discussion started by: Murugesh
3 Replies

5. Shell Programming and Scripting

Find the number of non-duplicate names recursively.

Hi, here comes another newbie question: How to find the number of non-duplicate names recursively? For example, my files are stored in the folders like: If I do find . -depth -name "*.txt" | wc -l This will gives out a result "4". One .txt file named "1.txt" in folder "1", and... (2 Replies)
Discussion started by: jiapei100
2 Replies

6. UNIX for Dummies Questions & Answers

CSV file:Find duplicates, save original and duplicate records in a new file

Hi Unix gurus, Maybe it is too much to ask for but please take a moment and help me out. A very humble request to you gurus. I'm new to Unix and I have started learning Unix. I have this project which is way to advanced for me. File format: CSV file File has four columns with no header... (8 Replies)
Discussion started by: arvindosu
8 Replies

7. Shell Programming and Scripting

Find duplicate based on 'n' fields and mark the duplicate as 'D'

Hi, In a file, I have to mark duplicate records as 'D' and the latest record alone as 'C'. In the below file, I have to identify if duplicate records are there or not based on Man_ID, Man_DT, Ship_ID and I have to mark the record with latest Ship_DT as "C" and other as "D" (I have to create... (7 Replies)
Discussion started by: machomaddy
7 Replies

8. Shell Programming and Scripting

bash keep only duplicate lines in file

hello all in my bash script I have a file and I only want to keep the lines that appear twice in the file.Is there a way to do this? thanks in advance! (4 Replies)
Discussion started by: vlm
4 Replies

9. Shell Programming and Scripting

Bash script to find the number of files and identify which ones are 0 bytes.

I am writing a bash script to find out all the files in a directory which are empty. I am running into multiple issues. I will really appreciate if someone can please help me. #!/bin/bash DATE=$(date +%m%d%y) TIME=$(date +%H%M) DIR="/home/statsetl/input/civil/test" ... (1 Reply)
Discussion started by: monasharma13
1 Replies

10. Shell Programming and Scripting

Bash Script to find/sort/move images/duplicate images from USB drive

Ultimately, I'm looking to create a script that allows me to plug in a usb drive with lots of jpegs on it & copy them over to a folder on my hard drive. So in the process of copying I am looking to hash check them, record dupes to a file, copy only 1 of the identical files (if it doesn't exsist... (1 Reply)
Discussion started by: JonaQuinn
1 Replies
fmlcut(1F)							   FMLI Commands							fmlcut(1F)

NAME
fmlcut - cut out selected fields of each line of a file SYNOPSIS
fmlcut -clist [filename...] fmlcut -flist [-dchar] [-s] [filename...] DESCRIPTION
The fmlcut function cuts out columns from a table or fields from each line in filename; in database parlance, it implements the projection of a relation. fmlcut can be used as a filter; if filename is not specified or is -, the standard input is read. list specifies the fields to be selected. Fields can be fixed length (character positions) or variable length (separated by a field delimiter character), depending on whether -c or -f is specified. Note: Either the -c or the -f option must be specified. OPTIONS
list A comma-separated list of integer field numbers (in increasing order), with optional - to indicate ranges. For example: 1,4,7; 1-3,8; -5,10 (short for 1-5,10); or 3- (short for third through last field). -clist If -c is specified, list specifies character positions (for instance, -c1-72 would pass the first 72 characters of each line). Note: No space intervenes between -c and list. -flist If -f is specified, list is a list of fields assumed to be separated in the file by the default delimiter character, TAB, or by char if the -d option is specified. For example, -f1,7 copies the first and seventh field only. Lines with no delimiter characters are passed through intact (useful for table subheadings), unless -s is specified. Note: No space intervenes between -f and list. The following options can be used if you have specified -f. -dchar If -d is specified, char is the field delimiter. Space or other characters with special meaning to FMLI must be quoted. Note: No space intervenes between -d and char . The default field delimiter is TAB. -s Suppresses lines with no delimiter characters. If -s is not specified, lines with no delimiters will be passed through untouched. EXAMPLES
Example 1: Getting login IDs and names The following example gets the login IDs and names. example% fmlcut -d: -f1,5 /etc/passwd Example 2: Getting the current login name The next example gets the current login name. example% `who am i | fmlcut -f1 -d" "` ATTRIBUTES
See attributes(5) for descriptions of the following attributes: +-----------------------------+-----------------------------+ | ATTRIBUTE TYPE | ATTRIBUTE VALUE | +-----------------------------+-----------------------------+ |Availability |SUNWcsu | +-----------------------------+-----------------------------+ SEE ALSO
fmlgrep(1F), attributes(5) DIAGNOSTICS
fmlcut returns the following exit values: 0 when the selected field is successfully cut out 2 on syntax errors The following error messages may be displayed on the FMLI message line: ERROR: line too long A line has more than 1023 characters or fields, or there is no new-line character. ERROR: bad list for c/f option Missing -c or -f option or incorrectly specified list. No error occurs if a line has fewer fields than the list calls for. ERROR: no fields The list is empty. ERROR: no delimiter Missing char on -d option. NOTES
fmlcut cannot correctly process lines longer than 1023 characters, or lines with no newline character. SunOS 5.10 5 Jul 1990 fmlcut(1F)
All times are GMT -4. The time now is 12:42 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy