Sponsored Content
Full Discussion: sort and dedup problem
Operating Systems AIX sort and dedup problem Post 302249505 by jim mcnamara on Tuesday 21st of October 2008 01:13:54 PM
Old 10-21-2008
This is not fine tuned code
Code:
#!/bin/ksh
echo "Step 1"
sort -n -t'|'  -k1.1,1.4 -k 2.1,2.4 inputfile > outputfile
echo "Step 2: unique count \c"
awk -F'|' '!arr[$1]++' outputfile > tmp.tmp
cat tmp.tmp | wc -l             # not a UUOC done on purpose
set -A arr $( head -2 tmp.tmp | tr -s '\n' ' ')
echo "Step 3:"
grep -e ''^${arr[0]}"  -e "^$arr[1]}"  outputfile

 

9 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Problem with sort

I am attempting to sort a file using the following command: sort +0 -t"|" infilename > outfilename I am getting the following error: sort: 0653-657 A write error occurred while sorting. The file size is 15036274 bytes This is an AIX 5.2 version I believe this is a problem with the... (1 Reply)
Discussion started by: jyoung
1 Replies

2. UNIX for Dummies Questions & Answers

SORT problem on SUN

Hello, I tried to sort on column2 followed by column1 and notice how the "updated" value in column1 is not sorted correctly! Can you tell me if i have the sort statement setup correctly please, thanks much! sort -t "|" -k2 -k1 sortin > sortout ... (2 Replies)
Discussion started by: bobk544
2 Replies

3. Shell Programming and Scripting

how to use awk to sort this problem out

there has several numbers which are:1,2,3,45,6,7,8,9,0,10,34,34,54,122,6756,54,87,99,2,1,45; how to write a shell script orts the above numbers into descending order and puts them into and arrray and also find and prints the minimum and maximum of those numbers, and finds and prints the average... (4 Replies)
Discussion started by: sonicstage
4 Replies

4. Shell Programming and Scripting

sort problem

I have file (srv_lst) with the contents as ... 9.2 IRMD115 8.1 IRMD115 and I am using the sort as to get the bigger version as : sort -r -u +1 $srv_lst | sort -k 1,1r and the output is 9.2 which is good .. if I have the contents of file srv_lst as : 9.2 IRMD115 10.2 IRMD115 ... (4 Replies)
Discussion started by: talashil
4 Replies

5. Programming

sort problem

I am in need of some direction. First off I want to admit this is an assignment but I have hit a block. I need to sort, by the number of times a string occurs (count), and output the top 10. I have found what number gives me the top 10 so from there I need to know how to sort them. Any... (1 Reply)
Discussion started by: Cn00b
1 Replies

6. Shell Programming and Scripting

problem with sort

Hi all, i want to sort by the (1-8) columns and (9-7) columns: my file: MARTINEZ---PAUL --DUPOND---EDDY --DURANDJACQUES --DUPOND--ALAIN output: --DUPOND--ALAIN --DUPOND---EDDY --DURANDJACQUES MARTINEZ---PAUL (6 Replies)
Discussion started by: saw7
6 Replies

7. UNIX for Advanced & Expert Users

Problem with sort +4

Apologies if this should be in 'unix for dummies' thread.. I have a large file containing records like this: 16 Feb 02:49 s_A123_ctas_log.20100216024000.bin 26 Feb 02:55 s_B123_ctas_log.20100226024000.bin 05 Mar 05:22 s_A127_ctas_log.20100305024000.bin I want to sort it by column 4... (2 Replies)
Discussion started by: Grueben
2 Replies

8. Shell Programming and Scripting

Dedup a large file(30M rows)

Hi, I have a large file with number of records in there. I need some help to find only first row based on a key and ignore other rows with the same key. I tried few things but file is huge(30 million rows). So need some solution that is very efficient. e.g Junk|Apple|7|Random|data|here...... (2 Replies)
Discussion started by: ran123
2 Replies

9. UNIX for Dummies Questions & Answers

Sort problem!

Hi, I have a file having content: 123 123 1234 12131 121 23 1212 1212121 23421 1212 1213123 I want to remove the repeated lines from it, i.e. I just want the any number just one time without any sorting in it. The problem is that I am not getting result from 'uniq' command. as... (2 Replies)
Discussion started by: nixhead
2 Replies
JOIN(1) 						      General Commands Manual							   JOIN(1)

NAME
join - relational database operator SYNOPSIS
join [-an] [-e s] [-o list] [-tc] file1 file2 DESCRIPTION
Join forms, on the standard output, a join of the two relations specified by the lines of file1 and file2. If file1 is `-', the standard input is used. File1 and file2 must be sorted in increasing ASCII collating sequence on the fields on which they are to be joined, normally the first in each line. There is one line in the output for each pair of lines in file1 and file2 that have identical join fields. The output line normally con- sists of the common field, then the rest of the line from file1, then the rest of the line from file2. Fields are normally separated by blank, tab or newline. In this case, multiple separators count as one, and leading separators are dis- carded. These options are recognized: -an In addition to the normal output, produce a line for each unpairable line in file n, where n is 1 or 2. -e s Replace empty output fields by string s. -o list Each output line comprises the fields specified in list, each element of which has the form n.m, where n is a file number and m is a field number. -tc Use character c as a separator (tab character). Every appearance of c in a line is significant. SEE ALSO
sort(1), comm(1), awk(1). BUGS
With default field separation, the collating sequence is that of sort -b; with -t, the sequence is that of a plain sort. The conventions of join, sort, comm, uniq, look and awk(1) are wildly incongruous. 7th Edition April 29, 1985 JOIN(1)
All times are GMT -4. The time now is 01:09 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy