Finding distinct characters from flat file


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Finding distinct characters from flat file
# 1  
Old 04-08-2013
Tools Finding distinct characters from flat file

Hi....I need one help....

I'm having a files which is having the data as follows...
a
b
c c
d d d
e
f

Now I need to find out distinct characters from this file and the output should be as follows -

a
b
c
d
e
f

Can you please help me on this? I'm using KSH script.

Thanks,
Krishanu
# 2  
Old 04-08-2013
Is this homework? If not, can you show us what have you tried so far?
# 3  
Old 04-08-2013
Tools

This is not a homework. Actually I'm working in one project where I have one similar kind of file which is having list of special characters (removed from xml file) exactly in same format. Now I need to find out the records from that xml files which is having those special characters. I have done everything and now the pending item is only this. That's why I need help. Any help will be highly appreciated.

Thanks.
# 4  
Old 04-08-2013
Using Associative Array in KSH:
Code:
#!/bin/ksh

typeset -A ARR

while read line
do
        for char in $line
        do
                (( ARR[$char] ))
        done
done < file

for k in ${!ARR[*]}
do
        print $k
done

Using awk program:
Code:
awk '{ for(i=1;i<=NF;i++) !A[$i]++ } END { for(i in A) print i } ' file

# 5  
Old 04-08-2013
Thanks Yoda. I've tried both options but did not get the expected result.

When I'm using the 1st option within a function, it is showing the error "typeset bad option" and when I'm using the 2nd option with nawk, it is working, but I'm getting the same list of characters in output, not distinct character.

Is there any options?
# 6  
Old 04-08-2013
I think awk needs the -F"" option otherwise it will split on whitespace, not characters.
# 7  
Old 04-08-2013
somehow the "typeset -A ARR" is not working in script, every time it is displaying the error message "typeset bad option(s)". Can anyone please help to find out the distinct character list?

Thanks,
Krishanu
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Finding total distinct count from multiple csv files through UNIX script

Hi All , I have multiple pipe delimited csv files are present in a directory.I need to find out distinct count on a column on those files and need the total distinct count on all files. We can't merge all the files here as file size are huge in millions.I have tried in below way for each... (9 Replies)
Discussion started by: STCET22
9 Replies

2. Shell Programming and Scripting

Finding first 6 characters

Hi , I'm using KSH88 I tried the following example to get the last 6 characters from a string echo 'abcdefghids' | sed 's/.*\(.\{6\}\)$/\1/' What chages i need to do to get the first 6 characters from the string my desired output should be abcdef Thank you (6 Replies)
Discussion started by: smile689
6 Replies

3. Shell Programming and Scripting

Finding Strings between 2 characters in a file

Hi All, Assuming i have got a file test.dat which has contains as follows: Unix = abc def fgt jug 111 2222 3333 Linux = gggg pppp qqq C# = ccc ffff llll I would like to traverse through the file, get the 1st occurance of "=" and then need to get the sting... (22 Replies)
Discussion started by: rtagarra
22 Replies

4. UNIX for Dummies Questions & Answers

How to remove numeric characters in the flat file

HI, can any one help me please .. i have flat file like qwer123rt ass3242ccf jjk654 kjh838ppp nhdg453ok hdkk34 i want remove numeric characters in the flat file i want output like this qwerrt assccf jjk kjhppp nhdgok hdkk help me... (4 Replies)
Discussion started by: rafimd1985
4 Replies

5. Shell Programming and Scripting

Finding File Names Ending In 3 Random Numerical Characters

Hi, I have a series of files (upwards of 500) the filename format is as follows CC10-1234P1999.WGS84.p190 each of this files is in a directory named for the file but excluding the extension. Now the last three numeric characters, in this case 999, can be anything from 001 to 999, I need to... (3 Replies)
Discussion started by: roche.j.mike
3 Replies

6. Shell Programming and Scripting

Select distinct values from a flat file

Hi , I have a similar problem. Please can anyone help me with a shell script or a perl. I have a flat file like this fruit country apple germany apple india banana pakistan banana saudi mango india I want to get a output like fruit country apple ... (7 Replies)
Discussion started by: smalya
7 Replies

7. AIX

How to cut a flat file according to a certain number of characters?

hello everybody i am looking for a shell to cut a flat file (with a long unique line) according to a certain number of characters and redirect every result to an output file. here is an example MyFile : 12 3 456 12 3 456 12 3 456 ..... and i took every 9-characters including BLANKS... (6 Replies)
Discussion started by: fastlane3000
6 Replies

8. Shell Programming and Scripting

Merge lines in Flat file based on first 5 characters

Hi I have the fixed width flat file having the following data 12345aaaaaaaaaabbbbbbbbbb 12365sssssssssscccccccccc 12365sssss 12367ddddddddddvvvvvvvvvv 12367 vvvvv Here the first column is length 5 second is length 10 third is length 10 if the second or third column exceeds... (3 Replies)
Discussion started by: Brado
3 Replies

9. UNIX for Advanced & Expert Users

foreign characters in flat file

Hey, Is there anyway I anks, Pocha (12 Replies)
Discussion started by: pochaman
12 Replies

10. Shell Programming and Scripting

Help Replacing Characters in Flat File

I was wondering if somebody could help me with something on UNIX. I have a file that looks like this - "nelson,bill","bill","123 Main St","Mpls","MN",55444,8877,william I want to replace all comma with pipes (|), except if the comma is within double quotes. (The first field is an example of... (8 Replies)
Discussion started by: nelson553011
8 Replies
Login or Register to Ask a Question