Shell Programming and Scripting

Hi All,
I have multiple files and i need to segregate unique and duplicates into files.
Eg: /source/ -- path
abc_12092016.csv
abc_11092016.csv

abc_12092016.csv

Code:

ID,NAME,NUMBER
1,XYZ,1234
2,SDF,3456
1,XYZ,1234

abc_11092016.csv

Code:

4,RTY,7890
6,WER,5678
8,YUI,0987
6,WER,5678

in the source folder it may be 2 files today and tomorrow 3 files etc..
from each file the unique and duplictes hvae to be segregated and loaded

result--

abc_12092016.csv

Code:

ID,NAME,NUMBER
1,XYZ,1234
2,SDF,3456

abc_11092016.csv

Code:

ID,NAME,NUMBER
4,RTY,7890
6,WER,5678
8,YUI,0987

abc_12092016_dup.csv

Code:

ID,NAME,NUMBER
1,XYZ,1234

abc_11092016_dup.csv

Code:

ID,NAME,NUMBER
6,WER,5678

a script will do fine....

Welcome ajayr1982,

I have a few to questions pose in response first:-Most importantly, What have you tried so far?

There are probably many ways to achieve most tasks, so giving us an idea of your style and thoughts will help us guide you to an answer most suitable to you so you can adjust it to suit your needs in future.


We're all here to learn and getting the relevant information will help us all.

Additionally, please wrap code, files, input & output/errors in CODE tags, like this:- to produce the following (fixed character width, space respected):-

Code:

This is my code

Not only does it make posts far easier to read, but CODE and ICODE sections respect multiple space and have fixed width characters, which is important for easily seeing input/output requirements. i have added some to your post. I hope I have guessed correctly.


Thanks, in advance,
Robin