|
|
Sponsored Content
Top Forums
Shell Programming and Scripting
Help with removing duplicate entries with awk or Perl
Post 302723527 by Amit Pande on Tuesday 30th of October 2012 10:09:58 AM
10-30-2012
|
|
10 More Discussions You Might Find Interesting
1. Linux
I have huge txt file having millions of trade data.
For e.g
Trade.txt (first 8 lines in the file is header info)
COB_DATE,TRADE_ID,SOURCE_SYSTEM_TRADE_ID,TRADE_GROUP_ID,
TRADE_TYPE,DEALER_NAME,EXTERNAL_COUNTERPARTY_ID,
EXTERNAL_COUNTERPARTY_NAME,DB_COUNTERPARTY_ID,... (6 Replies)
Discussion started by: nmumbarkar
6 Replies
2. Shell Programming and Scripting
Can anyone help me to removing duplicate records from 2 separate files in UNIX?
Please find the sample records for both the files
cat Monday.dat
3FAHP0JA1AR319226MOHMED ATEK 966504453742 SAU2010DE
3LNHL2GC6AR636361HEA DEUK CHOI 821057314531 KOR2010LE
3MEHM0JG7AR652083MUTLAB NAL-NAFISAH... (4 Replies)
Discussion started by: zooby
4 Replies
3. Shell Programming and Scripting
Hi,
I have a very big (with around 1 million entries) txt file with IPv4 addresses in the standard format, i.e. a.b.c.d
The file looks like
10.1.1.1
10.1.1.1
10.1.1.1
10.1.2.4
10.1.2.4
12.1.5.6
.
.
.
.
and so on....
There are duplicate/multiple entries for some IP... (3 Replies)
Discussion started by: sajal.bhatia
3 Replies
4. Shell Programming and Scripting
Input:
hello hello
hello hello
monkey
donkey
hello hello
drink
dance
drink
Output should be:
hello hello
monkey
donkey
drink
dance (9 Replies)
Discussion started by: cola
9 Replies
5. Post Here to Contact Site Administrators and Moderators
I have made some threads that were identical and were about the same question :( I've made them in 3 forums , the moderator has moved and merged one of these threads. There is one thread left and it need to be merged or deleted. Is there any way I can delete it or merge it myself ? I have delete... (1 Reply)
Discussion started by: k.a.docpp
1 Replies
6. Linux
I have log file having Traffic line
2011-05-21 15:11:50.356599 TCP (6), length: 52) 10.10.10.1.3020 > 10.10.10.254.50404:
2011-05-21 15:11:50.652739 TCP (6), length: 52) 10.10.10.254.50404 > 10.10.10.1.3020:
2011-05-21 15:11:50.652558 TCP (6), length: 89) 10.10.10.1.3020 >... (1 Reply)
Discussion started by: Rastamed
1 Replies
7. Shell Programming and Scripting
Hi,
I have the following command in place
nawk -F, '!a++' file > file.uniq
It has been working perfectly as per requirements, by removing duplicates by taking into consideration only first 3 fields. Recently it has started giving below error:
bash-3.2$ nawk -F, '!a++'... (17 Replies)
Discussion started by: makn
17 Replies
8. Shell Programming and Scripting
Hi everybody
I have a .txt file that contains some assembly code for optimizing it i need to remove some replicated parts.
for example I have:e_li r0,-1
e_li r25,-1
e_lis r25,0000
add r31, r31 ,r0
e_li r28,-1
e_lis r28,0000
add r31, r31 ,r0
e_li r28,-1 ... (3 Replies)
Discussion started by: Behrouzx77
3 Replies
9. Shell Programming and Scripting
Hello..
I am trying to remove the duplicate entries in a log files and used the the below shell script to do the same.
awk '!x++' <filename>
Can I do without using the awk command and the regex? I do not want to start the search from the beginning of the line in the log file as it contains... (9 Replies)
Discussion started by: sandeepcm
9 Replies
10. Shell Programming and Scripting
I have a file which has connections given as:
A B 0.1
B C 5.8
C B 5.8
E F 0.67
B A 0.1
A B and B A are same, so I want to remove one of them. Same with BC and CB.
Desired output:
A B 0.1
B C 5.8
E F 0.67 (2 Replies)
Discussion started by: Sanchari
2 Replies
LEARN ABOUT OPENSOLARIS
tor-resolve
TOR-RESOLVE(1) Tor Manual TOR-RESOLVE(1) NAME
tor-resolve - resolve a hostname to an IP address via tor SYNOPSIS
tor-resolve [-4|-5] [-v] [-x] hostname [sockshost[:socksport]] DESCRIPTION
tor-resolve is a simple script to connect to a SOCKS proxy that knows about the SOCKS RESOLVE command, hand it a hostname, and return an IP address. By default, tor-resolve uses the Tor server running on 127.0.0.1 on SOCKS port 9050. If this isn't what you want, you should specify an explicit sockshost and/or socksport on the command line. OPTIONS
-v Display verbose output. -x Perform a reverse lookup: get the PTR record for an IPv4 address. -5 Use the SOCKS5 protocol. (Default) -4 Use the SOCKS4a protocol rather than the default SOCKS5 protocol. Doesn't support reverse DNS. SEE ALSO
tor(1), torify(1). See doc/socks-extensions.txt in the Tor package for protocol details. AUTHORS
Roger Dingledine <arma@mit.edu>, Nick Mathewson <nickm@alum.mit.edu>. AUTHOR
Peter Palfrader Author. Tor 09/26/2014 TOR-RESOLVE(1)