Shell script to check the unique numbers in huge data


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Shell script to check the unique numbers in huge data
# 1  
Old 08-10-2007
MySQL Shell script to check the unique numbers in huge data

Friends,
I have to write a shell script,the description is----

i Have to check the uniqueness of the numbers in a file.
A file is containing 200thousand tickets and a ticket have 15 numbers in asecending order.And there is a strip that is having 6 tickets that means 90 numbers.I have to write a shell script like this----- No two tickets are common in 200thousand tickets and no two numers are repeating in a strip.
I would appericiate your help.

If you want some more info i am pasting it here.
---------------------------------------------------
Each ticket strip will have six tickets.
A strip is a group of 6 tickets. Every possible number from 1 through to 90 will appear on each full strip.
Each ticket will have 9 columns, and 3 rows. Hence a ticket strip will effectively have 9 columns and 18 rows.
Column 1 will have numbers between 1-9, 2 will have 10-19, so on till 8th column having 70-79; the only exception is the last column (9th column) having 80-90.
In each row there will be only 5 columns generated, the other 4 will be blank.
If there are more than 1 value in a column within a ticket, then they should be in ascending order.
# 2  
Old 08-10-2007
If anyone can help me in writing this script,i will appericiate his help.
If anyone can guide me then also it will be good.I am trying but never wrote any script till now.
# 3  
Old 08-10-2007
Camel for the job

Hello namishtiwari,
I suggest you take a look at Perl if you have it on your system.
It's very powerful for parsing lots of data like this.
Can you please paste a couple of lines from the source data file so
that we can get a picture of how it looks like. When you use the word "common", you mean that every ticket is unique right?
# 4  
Old 08-10-2007
Here is the insert statement through which we are inserting data.

insert into Ticket (TicketID,TicketStripID,TicketData,SequenceNo) values (100,1,'071400004200007480000021364450660000001623390000007784',1);
insert into Ticket (TicketID,TicketStripID,TicketData,SequenceNo) values (101,1,'050000004055627100001028330000680087080000004959007988',2);
insert into Ticket (TicketID,TicketStripID,TicketData,SequenceNo) values (102,1,'001522300054600000060000004156007090091926310000610000',3);
insert into Ticket (TicketID,TicketStripID,TicketData,SequenceNo) values (103,1,'000000004652677685011329384700000000001700000057697886',4);
insert into Ticket (TicketID,TicketStripID,TicketData,SequenceNo) values (104,1,'020020340000007581001825004351640000030027370000650083',5);
insert into Ticket (TicketID,TicketStripID,TicketData,SequenceNo) values (105,1,'041100004553007200000024324800630082001200350058007389',6);
insert into Ticket (TicketID,TicketStripID,TicketData,SequenceNo) values (106,2,'000022340057007685071027004800620000000000360059687789',1);
insert into Ticket (TicketID,TicketStripID,TicketData,SequenceNo) values (107,2,'021221004600000090051500310050007100080024004700617800',2);
insert into Ticket (TicketID,TicketStripID,TicketData,SequenceNo) values (108,2,'001300334553000084040026000056657500001800394900660086',3);

Last edited by namishtiwari; 08-10-2007 at 06:20 AM..
# 5  
Old 08-10-2007
Quote:
Originally Posted by rexf
Hello namishtiwari,
I suggest you take a look at Perl if you have it on your system.
It's very powerful for parsing lots of data like this.
Can you please paste a couple of lines from the source data file so
that we can get a picture of how it looks like. When you use the word "common", you mean that every ticket is unique right?
Rexf,
Let me know the Perl script for this also.I will try to run that script in a system that have perl.
My requirement is---
1 strip has all the 90 numbers,numbers should be unique.
FYI one strip contains all the numbers from 1 to 90.

Last edited by namishtiwari; 08-10-2007 at 06:25 AM..
# 6  
Old 08-10-2007
Columns?

You have to clarify which columns you want to compare with what?
If we examine the first line;

insert into Ticket (TicketID,TicketStripID,TicketData,SequenceNo) values (100,1,'071400004200007480000021364450660000001623390000007784',1);

There are 4 values;

a) 100
b) 1
c) 071400004200007480000021364450660000001623390000007784
d) 1

Define the range limit for a, b, c and d, and explain _how_ you want
them to be compared, wheter the only requirement is that they should be unique within specific ranges, and compared to each other, or other comparison requirements?
# 7  
Old 08-10-2007
CPU & Memory

rexf thanks for your kind help.
That problem is solved now,i got a new one and pasted that as a new thread.I will require your help in that.
Thanks in advance.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Need Optimization shell/awk script to aggreagte (sum) for all the columns of Huge data file

Optimization shell/awk script to aggregate (sum) for all the columns of Huge data file File delimiter "|" Need to have Sum of all columns, with column number : aggregation (summation) for each column File not having the header Like below - Column 1 "Total Column 2 : "Total ... ...... (2 Replies)
Discussion started by: kartikirans
2 Replies

2. Shell Programming and Scripting

AWK, Perl or Shell? Unique strings and their maximum values from 3 column data file

I have a file containing data like so: 2012-01-02 GREEN 4 2012-01-02 GREEN 6 2012-01-02 GREEN 7 2012-01-02 BLUE 4 2012-01-02 BLUE 3 2012-01-02 GREEN 4 2012-01-02 RED 4 2012-01-02 RED 8 2012-01-02 GREEN 4 2012-01-02 YELLOW 5 2012-01-02 YELLOW 2 I can't always predict what the... (4 Replies)
Discussion started by: rich@ardz
4 Replies

3. Shell Programming and Scripting

Select combination unique using shell script

Hi All, bash-3.00$ gzgrep -i '\ ExecuteThread:' /******/******/******/******/stdout.log.txt.gz <Jan 7, 2012 5:54:55 PM UTC> <Error> <WebLogicServer> <BEA-000337> < ExecuteThread: '414' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "696" seconds working on the request... (4 Replies)
Discussion started by: osmanux
4 Replies

4. Shell Programming and Scripting

Shell script to count unique rows in a CSV

HI All, I have a CSV file of 30 columns separated by ,. I want to get a count of all unique rows written to a flat file. The CSV file is around 5000 rows The first column is a time stamp and I need to exclude while counting unique Thanks, Ravi (4 Replies)
Discussion started by: Nani369
4 Replies

5. Shell Programming and Scripting

Problem running Perl Script with huge data files

Hello Everyone, I have a perl script that reads two types of data files (txt and XML). These data files are huge and large in number. I am using something like this : foreach my $t (@text) { open TEXT, $t or die "Cannot open $t for reading: $!\n"; while(my $line=<TEXT>){ ... (4 Replies)
Discussion started by: ad23
4 Replies

6. Shell Programming and Scripting

Perl script error to split huge data one by one.

Below is my perl script: #!/usr/bin/perl open(FILE,"$ARGV") or die "$!"; @DATA = <FILE>; close FILE; $join = join("",@DATA); @array = split( ">",$join); for($i=0;$i<=scalar(@array);$i++){ system ("/home/bin/./program_name_count_length MULTI_sequence_DATA_FILE -d... (5 Replies)
Discussion started by: patrick87
5 Replies

7. Shell Programming and Scripting

Shell script to check numbers!

Hello All, I have 3 types of files. The names of which starts with P,I,M like P********* D********* M********* now I need to do some operations witht hese files.. so if file name starts with P or p then do the operation for P file... fi else (20 Replies)
Discussion started by: smarty86
20 Replies

8. UNIX for Dummies Questions & Answers

Getting unique list of numbers using grep

Hi, I am going to fetch a list of numbers that starts with "0032" from a file with a format like the given below: " 0032459999 0032458888 0032457777 0032451111 0032452222 0032453333 0032459999 0032458888 0032457777 0032451111 0032452222 0032453333 " I want to get a unique... (6 Replies)
Discussion started by: tinku
6 Replies

9. UNIX for Dummies Questions & Answers

Shell Script Unique Identifier Question

i All I have scripting question. I have a file "out.txt" which is generated by another script the file contains the following my_identifier8859574 logout The number is generated in the script and I have put the my_identifier bit in front of it as a unique identifier I now have... (7 Replies)
Discussion started by: grahambo2005
7 Replies

10. UNIX for Dummies Questions & Answers

To get unique numbers from two files

here i have two files: file 1 1 2 3 4 5 5 6 7 8 9 file 2 4 5 6 6 8 8 (6 Replies)
Discussion started by: i.scientist
6 Replies
Login or Register to Ask a Question