Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Execution problems with scripting

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Execution problems with scripting
# 1  
Old 01-31-2012
Execution problems with scripting
Hi,
I am new to scripting.I had one problem infront of me.I tried in many ways with minimal knowledge........Kindly help me.
Description:

I want a shell script where it has to read an input.txt file and need to remove duplicate lines and the result need to kept in output.txt file.

input file example:

Code:
1974929729,388004948949994,8849490049004,7894994590599
7240283747,778938934988489,87888948949894,998389499499
1974929729,778483889398899,99488495883994,783948859959
8575785999,849498599595050,90505096000660,885699606000

As above my input file looks ,the first 10 digits seperated by delimiter ,repeted 2 times are more.script need to delete such repeated lines keeping any one line.in above example 1974929729 is repeted 2 times.so the second nee to remove.my input file contains more than 25k lines.
Expected output.txt:
Code:
1974929729,388004948949994,8849490049004,7894994590599
7240283747,778938934988489,87888948949894,998389499499
8575785999,849498599595050,90505096000660,885699606000

Appreciated your help Smilie
Last edited by methyl; 01-31-2012 at 02:47 PM..
# 2  
Old 01-31-2012
Use sort with the -u option to show only unique lines

Code:
sort -u <filename> > outputfile.txt

Padow
# 3  
Old 01-31-2012
Quote:
Use sort with the -u option to show only unique lines
Does not match specification or the data samples in post #1. (Lines 1 and 3 are not identical. Only the key field is identical).
# 4  
Old 01-31-2012
Try this:

( was before edit )
Code:
cat input.txt | tr ',' ' ' | sort -u -k1,1

( after edit)
Code:
tr ',' ' ' < input.txt | sort -u -k1,1 | tr ' ' ','

As for your example:

Code:
$ cat input.txt 
1974929729,388004948949994,8849490049004,7894994590599
7240283747,778938934988489,87888948949894,998389499499
1974929729,778483889398899,99488495883994,783948859959
8575785999,849498599595050,90505096000660,885699606000


$ tr ',' ' ' < input.txt | sort -u -k1,1 | tr ' ' ','
1974929729,388004948949994,8849490049004,7894994590599
7240283747,778938934988489,87888948949894,998389499499
8575785999,849498599595050,90505096000660,885699606000

Last edited by frederik.nosi; 01-31-2012 at 08:55 PM.. Reason: Posted to quickly, forgot to add again the comas
# 5  
Old 02-02-2012
Code:
awk -F , '!a[$1]++' input.txt

# 6  
Old 02-03-2012
Problem solved.
Thanks much to all.

frederik,

Special thanks.Your code perfectly suited to my problem..Smilie
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Execution Problems with Solaris 8

i have problems with using solaries 8 . code and commands for extracting files from CD ? especially extracting file.jar from package and batch (1 Reply)
Discussion started by: moh_abaloo
1 Replies

2. UNIX for Beginners Questions & Answers

Execution problems

How to find a word in a directory which contains many files? i just want to count how many such words are present in all the files? This is the code which i tried for a single file echo "Enter the file name:" read file echo "Enter the word to search:" read word if then echo "The count... (4 Replies)
Discussion started by: Meeran Rizvi
4 Replies

3. Solaris

Execution problems with Mailx

Unable to send mail using mailx command. I am using solaris 5.9 I am trying to send notification for the scheduled jobs in crob but the mailx is not working. Checked the settings in submit.cf and sendmail.cf but unable to find the solution. Error message root@sshldb # nslookup mailhost... (8 Replies)
Discussion started by: Srinathkiru
8 Replies

4. Shell Programming and Scripting

Execution Problems

this my source file ************* fixed *************** Begin equipmentId : d9 processor : fox number : bhhhhhh Variable # 1: Id : 100 Type : 9 nType : s gType : 5f mType : 4 LField : England DataField : london Length ... (6 Replies)
Discussion started by: teefa
6 Replies

5. Linux

ssh and passwd scripting execution problems on linux

I'm having a problem here and I was wondering if anyone could help me? I'm putting together a password script. First off, I don't have root access. I have sudo access. Lets say the User ID is Trevor1, the password is H!rry23! and the server name is Linux1234 This is how the script begins ... (5 Replies)
Discussion started by: wdog17
5 Replies

6. Shell Programming and Scripting

Execution Problems with Cron

Hi, I have written a shell script to transfer files to a SFTP server passing the filername, source and dest directory as parameters and it runs well. :) I want to schedule this script to run periodically using a cron job. root@pingu # cat /etc/crontab SHELL=/bin/bash... (1 Reply)
Discussion started by: chetancrsp18
1 Replies

7. Shell Programming and Scripting

Execution Problems!!

i have been working on this for a about 12 hours today say's end of file un expected any idea's using the bourne shell and its driving me nuts worked fine in bash but prof says make it work in bourne and good luck worth 13% any help would be awesome #!/bin/sh trap "rm mnt2/source/tmp/* 2>... (1 Reply)
Discussion started by: mrhiab
1 Replies

8. Shell Programming and Scripting

Execution problems with grep command in scripting

Hi All, I was looking for grep command option which can exactly matches the word in a file, for examples you may be seeing one word that is also in another word, there might be lkk0lv23 and a lkk0lv234 in which case lkk0lv23 will put BOTH hosts from the grep in. I was using this in a bash... (2 Replies)
Discussion started by: bobby320
2 Replies

9. UNIX and Linux Applications

Execution Problems with Cron

Hi all!! I have a nerve-wracking concept (probably for me!!) which is not understood. My crontab entry looks this way. 33 09 22 3 * /home/myexp.sh "Bgp4 ALL" >/dev/null 2>&1 But cron gets started occasionally. Sometimes it does. Sometimes it does not. And sometimes it hangs in the middle (I... (1 Reply)
Discussion started by: dhivyasuresh
1 Replies

10. Programming

execution problems with cron

how to store a date into file? and how we can access date from the file? ---------- Post updated at 06:09 AM ---------- Previous update was at 06:08 AM ---------- how we can store date in file? (1 Reply)
Discussion started by: causalmodi777
1 Replies
Login or Register to Ask a Question