02-27-2017
How to identify varying unique fields values from a text file in UNIX?
Hi,
I have a huge unsorted text file. We wanted to identify the unique field values in a line and consider those fields as a primary key for a table in upstream system.
Basically, the process or script should fetch the values from each line that are unique compared to the rest of the lines in the file.
If there are 150 bytes in a line for a file that is containing around 100,000 lines and I wanted to find how many bytes on the line (150 bytes) can be formed as a primary key?
I know the file has to be sorted based on the entire 150 bytes and aftre that I am not sure how can I identify the uniqueness between lines?
Please help.
Thanks,
Mani A
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
Hi,
I need some help in knowing how I can append tabs at the end of each line...
The data looks something like this:
field1, field2, field3, field4
1 2
3 4 5
I have values in field1 and field 2 in the first row and I would like to append tab on field3 and field4 for the first row..and in... (6 Replies)
Discussion started by: madhunk
6 Replies
2. Shell Programming and Scripting
Greetings,
I would like to extract records from a fixed width text file that have unique field elements.
Data is structured like this:
John A Smith NY
Mary C Jones WA
Adam J Clark PA
Mary Jones WA
Fieldname / start-end position
Firstname 1-10... (8 Replies)
Discussion started by: sitney
8 Replies
3. Shell Programming and Scripting
I have a situation where I am reading a text file line-by-line. Those lines of data contain comma separated fields of data. However, each line can vary in the number of fields it can contain. What I need to do is parse apart each line and write each field of data found (left to right) into a file.... (7 Replies)
Discussion started by: 2reperry
7 Replies
4. Shell Programming and Scripting
Hi,
I have a file like this:
Some_String_Here 123 123 123 321 321 321 3432 3221 557 886 321 321
I would like to find only the unique values in the files and get the following output:
Some_String_Here 123 321 3432 3221 557 886
I am trying to get this done using awk. Can someone please... (5 Replies)
Discussion started by: Legend986
5 Replies
5. Shell Programming and Scripting
Hi all,
I have got a problem while comparing 2 text files and the result should contains the unique values(Non repeatable).
For eg:
file1.txt
1
2
3
4
file2.txt
2
3
So after comaping the above 2 files I should get only 1 and 4 as the output. Pls help me out. (7 Replies)
Discussion started by: smarty86
7 Replies
6. Shell Programming and Scripting
My data is something like as shown below. Out of this i want the details of alarms (ex: 1947147711,1947147081......) and the fields( ex :sw=tacmwafabb9:shelf=1:slot=5-2:pport=2)
Once i have these details separated, i want the count of these excluding the duplicates. What is the best possible way... (7 Replies)
Discussion started by: rdhanek
7 Replies
7. Shell Programming and Scripting
I have high values (such as ÿÿÿÿ) in a text file contained in an Unix AIX server. I need to identify all the records
which are having these high values and also get the position/column number in the record structure if possible. Is there
any Unix command by which this can be done to :
1.... (5 Replies)
Discussion started by: devina
5 Replies
8. Shell Programming and Scripting
Good morning all,
I have a problem that is one step beyond a standard awk compare.
I would like to compare three files which have several thousand records against a fourth file. All of them have a value in each row that is identical, and one value in each of those rows which may be duplicated... (1 Reply)
Discussion started by: nashton
1 Replies
9. UNIX for Dummies Questions & Answers
Hi would like to ask you guys any advise regarding my problem
I have this kind of data
file.txt
111111111,20
111111111,50
222222222,70
333333333,40
444444444,10
444444444,20
I need to get this
file1.txt
111111111,70
222222222,70
333333333,40
444444444,30
using this code I can... (6 Replies)
Discussion started by: reks
6 Replies
10. Shell Programming and Scripting
datafile:
2017-03-24 10:26:22.098566|5|'No Route for Sndr:RETEK RMS 00040 /ZZ Appl:PF Func:PD Txn:832 Group Cntr:None ISA CntlNr:None Ver:003050 '|'2'|'PFI'|'-'|'EAI_ED_DeleteAll'|'EAI_ED'|NULL|NULL|NULL|139050594|ActivityLog|
2017-03-27 02:50:02.028706|5|'No Route for... (7 Replies)
Discussion started by: SkySmart
7 Replies
LEARN ABOUT DEBIAN
dbfdump
SHAPELIB(1) User Commands SHAPELIB(1)
NAME
dbfdump - dump xBase DBF files as text
SYNOPSIS
dbfdump [-h] [-m] [-r] file
DESCRIPTION
Dumps the contents of file to standard output. The first line contains the field names appearing in file, and each of the following lines
contains the field values of a record. Field names and values are padded by spaces to their field widths. Empty fields are printed as the
string "(NULL)".
OPTIONS
-h Prints the column field definitions before other output. Each field definition consists of a line of the form
Field: index, Type=type, Title=`name', Width=width, Decimals=precision
where index is the zero offset column number of the field; the type indicates the datatype of the field value and is either "Inte-
ger", "Real" or "String"; name is the field's name; width is the number of bytes reserved for the field's value; and precision is
the number of decimal places of precision for "Real" type fields, and is zero for "Integer" and "String" type fields.
-m Prints each record in multiline format separated by empty lines. The first line of a record gives the number of the record in the
form
Records: record_index
where record_index is the zero offset number of the record in the file, and then each field of the record appears on its own line in
the format
name: value
-r Prints the exact bytes occurring in file for field values and suppresses printing "(NULL)" for empty values.
EXIT STATUS
0 Successful program execution.
1 Missing file argument.
2 Failed to open file.
3 There are no fields in file.
DIAGNOSTICS
The following diagnostics may be issued on stdout:
DBFOpen(file,"r") failed.
There are no fields in this table!
AUTHORS
Frank Warmerdam (warmerdam@pobox.com) is the maintainer of the shapelib shapefile library. Joonas Pihlaja (jpihlaja@cc.helsinki.fi) wrote
this man page.
BUGS
Unless the -r option is given, values in numeric fields that overflow the int or double types of the C language are printed as plus or
minus a huge number. For integer fields the huge value is HUGE_VALL from <stdlib.h> and for real fields it is HUGE_VALF.
SEE ALSO
dbf_dump(1), dbfcreate(1), dbfadd(1), shpadd(1), shpcreate(1), shpdump(1), shprewind(1)
shapelib OCTOBER 2004 SHAPELIB(1)