10 More Discussions You Might Find Interesting
1. UNIX for Dummies Questions & Answers
Hi
How can I extract sequences from a fasta file with respect a certain criteria? The beginning of my file (containing in total more than 1000 sequences) looks like this:
>H8V34IS02I59VP
SDACNDLTIALLQIAREVRVCNPTFSFRWHPQVKDEVMRECFDCIRQGLG
YPSMRNDPILIANCMNWHGHPLEEARQWVHQACMSPCPSTKHGFQPFRMA... (6 Replies)
Discussion started by: Marion MPI
6 Replies
2. Shell Programming and Scripting
Hi....I need one help....
I'm having a files which is having the data as follows...
a
b
c c
d d d
e
f
Now I need to find out distinct characters from this file and the output should be as follows -
a
b
c
d
e
f
Can you please help me on this? I'm using KSH script. (18 Replies)
Discussion started by: Krishanu Saha
18 Replies
3. Shell Programming and Scripting
Hi, I have the following file:
LOG:015608::ERR:2310:map_spsrec:Invalid parameter
LOG:015608::ERR:2471:map_dgdrec:Invalid parameter
LOG:015608::ERR:2487:map_nnmrec:Invalid number
LOG:015608::ERR:2310:map_nmrec:Invalid number
LOG:015608::ERR:2438:map_nmrec:Invalid number
As a delimiter I... (2 Replies)
Discussion started by: apenkov
2 Replies
4. Shell Programming and Scripting
I am a beginner to scripting, please help me in this regard.
How do I create a script that provides a count of distinct values of all the fields in the pipe delimited file ? I have 20 different files with multiple columns in each file. I needed to write a generic script where I give the number... (2 Replies)
Discussion started by: vukkusila
2 Replies
5. UNIX for Dummies Questions & Answers
I am a beginner to scripting, please help me in this regard.
How do I create a script that provides a count of distinct values of all the fields in the pipe delimited file ? I have 20 different files with multiple columns in each file. I needed to write a generic script where I give the number... (1 Reply)
Discussion started by: vukkusila
1 Replies
6. Shell Programming and Scripting
this is a little more complex than that. I have a text file and I need to find all the distinct words that appear in a line after the word TABLESPACE
when I grep for just the word tablespace, I get:
how do i parse this a little better so i have a smaller file to read?
This is just an... (4 Replies)
Discussion started by: guessingo
4 Replies
7. UNIX for Dummies Questions & Answers
How do I create a script that provides a count of distinct values of a particular field in a file utilizing commonly available UNIX commands (sh or awk)?
Field1|Field2|Field3|Field4
AAA|BBB|CCC|DDD
111|222|333|777
AAA|EEE|ZZZ|EEE
111|555|333|444
AAA|EEE|CCC|DDD
111|222|555|444
For... (2 Replies)
Discussion started by: Refresher
2 Replies
8. Shell Programming and Scripting
Hi
I have a pipe delimited file. I am trying to grab the DISTINCT value from the second field. The file is something like:
1233|apple|ron
1234|apple|elephant
1235|egg|man
the output I am trying to get from second field is apple,egg (apple coming only once)
Thanks
simi (4 Replies)
Discussion started by: simi28
4 Replies
9. Shell Programming and Scripting
Datafile has the following data seperated by :
FIELD1:FIELD2:FIELD3
D1:/opt/9.1.9:Y
D2:/opt/10.1.10:Y
D3:/opt/9.1.9:Y
D4:/opt/8.1.8:Y
D5:/opt/8.1.8:Y
D6:/opt/9.1.9:Y
D7:/opt/9.1.9:Y
D8:/opt/10.1.10:Y
D9:/opt/9.1.9:Y
D10:/opt/10.1.10:Y
I want to do some operations only on the distinct... (2 Replies)
Discussion started by: pbekal
2 Replies
10. UNIX for Dummies Questions & Answers
Hi, buddies out there.
I have a text file ( only one column ) which I created using vi editor. The file contains duplicate rows and I would like to select distinct rows, how to go on it using unix command:
file content =
apple
apple
orange
watermelon
apple
orange
Can it be done... (7 Replies)
Discussion started by: merry susana
7 Replies
bup-margin(1) General Commands Manual bup-margin(1)
NAME
bup-margin - figure out your deduplication safety margin
SYNOPSIS
bup margin [options...]
DESCRIPTION
bup margin iterates through all objects in your bup repository, calculating the largest number of prefix bits shared between any two
entries. This number, n, identifies the longest subset of SHA-1 you could use and still encounter a collision between your object ids.
For example, one system that was tested had a collection of 11 million objects (70 GB), and bup margin returned 45. That means a 46-bit
hash would be sufficient to avoid all collisions among that set of objects; each object in that repository could be uniquely identified by
its first 46 bits.
The number of bits needed seems to increase by about 1 or 2 for every doubling of the number of objects. Since SHA-1 hashes have 160 bits,
that leaves 115 bits of margin. Of course, because SHA-1 hashes are essentially random, it's theoretically possible to use many more bits
with far fewer objects.
If you're paranoid about the possibility of SHA-1 collisions, you can monitor your repository by running bup margin occasionally to see if
you're getting dangerously close to 160 bits.
OPTIONS
--predict
Guess the offset into each index file where a particular object will appear, and report the maximum deviation of the correct answer
from the guess. This is potentially useful for tuning an interpolation search algorithm.
--ignore-midx
don't use .midx files, use only .idx files. This is only really useful when used with --predict.
EXAMPLE
$ bup margin
Reading indexes: 100.00% (1612581/1612581), done.
40
40 matching prefix bits
1.94 bits per doubling
120 bits (61.86 doublings) remaining
4.19338e+18 times larger is possible
Everyone on earth could have 625878182 data sets
like yours, all in one repository, and we would
expect 1 object collision.
$ bup margin --predict
PackIdxList: using 1 index.
Reading indexes: 100.00% (1612581/1612581), done.
915 of 1612581 (0.057%)
SEE ALSO
bup-midx(1), bup-save(1)
BUP
Part of the bup(1) suite.
AUTHORS
Avery Pennarun <apenwarr@gmail.com>.
Bup unknown- bup-margin(1)