Duplicates and unique segregation


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Duplicates and unique segregation
# 1  
Old 12-14-2016
Duplicates and unique segregation

Hi All,
I have multiple files and i need to segregate unique and duplicates into files.
Eg: /source/ -- path
abc_12092016.csv
abc_11092016.csv

abc_12092016.csv
Code:
ID,NAME,NUMBER
1,XYZ,1234
2,SDF,3456
1,XYZ,1234

abc_11092016.csv
Code:
4,RTY,7890
6,WER,5678
8,YUI,0987
6,WER,5678

in the source folder it may be 2 files today and tomorrow 3 files etc..
from each file the unique and duplictes hvae to be segregated and loaded

result--

abc_12092016.csv
Code:
ID,NAME,NUMBER
1,XYZ,1234
2,SDF,3456


abc_11092016.csv
Code:
ID,NAME,NUMBER
4,RTY,7890
6,WER,5678
8,YUI,0987

abc_12092016_dup.csv
Code:
ID,NAME,NUMBER
1,XYZ,1234

abc_11092016_dup.csv
Code:
ID,NAME,NUMBER
6,WER,5678

a script will do fine....

Last edited by rbatte1; 12-14-2016 at 10:18 AM..
# 2  
Old 12-14-2016
Welcome ajayr1982,

I have a few to questions pose in response first:-
  • Is this homework/assignment? There are specific forums for these.
  • What have you tried so far?
  • What output/errors do you get?
  • What OS and version are you using?
  • What are your preferred tools? (C, shell, perl, awk, etc.)
  • What logical process have you considered? (to help steer us to follow what you are trying to achieve)
Most importantly, What have you tried so far?

There are probably many ways to achieve most tasks, so giving us an idea of your style and thoughts will help us guide you to an answer most suitable to you so you can adjust it to suit your needs in future.


We're all here to learn and getting the relevant information will help us all.

Additionally, please wrap code, files, input & output/errors in CODE tags, like this:-
Quote:
[CODE]This is my code[/CODE]
to produce the following (fixed character width, space respected):-
Code:
This is my code

Not only does it make posts far easier to read, but CODE and ICODE sections respect multiple space and have fixed width characters, which is important for easily seeing input/output requirements. i have added some to your post. I hope I have guessed correctly.


Thanks, in advance,
Robin
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Count occurrence of column one unique value having unique second column value

Hello Team, I need your help on the following: My input file a.txt is as below: 3330690|373846|108471 3330690|373846|108471 0640829|459725|100001 0640829|459725|100001 3330690|373847|108471 Here row 1 and row 2 of column 1 are identical but corresponding column 2 value are... (4 Replies)
Discussion started by: angshuman
4 Replies

2. UNIX for Dummies Questions & Answers

Print unique lines without sort or unique

I would like to print unique lines without sort or unique. Unfortunately the server I am working on does not have sort or unique. I have not been able to contact the administrator of the server to ask him to add it for several weeks. (7 Replies)
Discussion started by: cokedude
7 Replies

3. Shell Programming and Scripting

Change unique file names into new unique filenames

I have 84 files with the following names splitseqs.1, spliseqs.2 etc. and I want to change the .number to a unique filename. E.g. change splitseqs.1 into splitseqs.7114_1#24 and change spliseqs.2 into splitseqs.7067_2#4 So all the current file names are unique, so are the new file names.... (1 Reply)
Discussion started by: avonm
1 Replies

4. Shell Programming and Scripting

How to ID duplicates in a string

Hi guys, I am trying to identify the number of duplicate entries in a string inputed by the user. Here is a command I use: $ user_input="M T T" $echo "${user_input}" | awk '{for(i=0;i<=NF;i++) print $i }'| sort | uniq -d The above works fine for string with multiple letters. The problem is... (2 Replies)
Discussion started by: aoussenko
2 Replies

5. Shell Programming and Scripting

select unique values from duplicates in linux

I have values in the variable as so the for loop just fetches one by one params=$'$HEW_SRC_DATABASE_LIB\nprmAttunityUser\nprmAttunityPwd\nprmODBCDataSource\nprmLoadInd\nprmSrc_Lib_ATM\nprmODBCDataSource_ATM' and i have a grep command like this ret=`grep \$y $pf` ... (0 Replies)
Discussion started by: vee_789
0 Replies

6. Shell Programming and Scripting

Removing duplicates

Hi, I have a file in the below format., test test (10) to to (25) see see (45) and i need the output in the format of test 10 to 25 see 45 Some one help me? (6 Replies)
Discussion started by: imdadulla
6 Replies

7. UNIX for Dummies Questions & Answers

Duplicates

Hi, How to eliminate the duplicate values in unix? I have a excel file which contains duplicate values. Need to use this in a script. Thanks in advance. (3 Replies)
Discussion started by: venkatesht
3 Replies

8. Shell Programming and Scripting

get part of file with unique & non-unique string

I have an archive file that holds a batch of statements. I would like to be able to extract a certain statement based on the unique customer # (ie. 123456). The end for each statement is noted by "ENDSTM". I can find the line number for the beginning of the statement section with sed. ... (5 Replies)
Discussion started by: andrewsc
5 Replies

9. Shell Programming and Scripting

Non Duplicates

I have input file like below. I00789524 0213 5212 D00789524 0213 5212 I00778787 2154 5412 The first two records are same(Duplicates) except I & D in the first character. I want non duplicates(ie. 3rd line) to be output. How can we get this . Can you help. Is there any single AWK or SED... (3 Replies)
Discussion started by: awk_beginner
3 Replies

10. HP-UX

getting duplicates

how to get duplicates in a file containing data in columns using command or scripting? (4 Replies)
Discussion started by: megh
4 Replies
Login or Register to Ask a Question

Featured Tech Videos