05-21-2018
Welcome to the forum.
Why is B4-B6 considered a duplicate? I can't see it twice or more in the input line.
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
Hello Experts,
I have two files named old and new. Below are my example files. I need to compare and print the records that only exist in my new file. I tried the below awk script, this script works perfectly well if the records have exact match, the issue I have is my old file has got extra... (4 Replies)
Discussion started by: forumthreads
4 Replies
2. Shell Programming and Scripting
The log reads as follows.
fname1;lname1;eid1;addr;pincode1;
fname2;lname2;eid2;addr2;pincode2;
fname3;lname3;eid3;addr3;pincode3;
fname4;lname4;eid;addr4;pincode4;
how do i extract only fname and save it in an array
similarly for lname and so on
i tried reading a file and cutting each... (5 Replies)
Discussion started by: vkca
5 Replies
3. Shell Programming and Scripting
Hi
I need a script that removes the duplicate records and write it to a new file
for example I have a file named test.txt and it looks like
abcd.23
abcd.24
abcd.25
qwer.25
qwer.26
qwer.98
I want to pick only $1 and compare with the next record and the output should be
abcd.23... (6 Replies)
Discussion started by: antointoronto
6 Replies
4. Shell Programming and Scripting
hi guys,
basically what i'm trying to do is fetching a set of columns from an oracle database like so...
my_row=`sqlplus -s user/pwd << EOF
set head off
select user_id, username from all_users where rownum = 1;
EOF`
echo $my_row
the code above returns...
1 ADSHOCKER
so then i... (3 Replies)
Discussion started by: adshocker
3 Replies
5. Shell Programming and Scripting
I have a file with the following format:
fields seperated by "|"
title1|something class|long...content1|keys
title2|somhing class|log...content1|kes
title1|sothing class|lon...content1|kes
title3|shing cls|log...content1|ks
I want to remove all duplicates with the same "title field"(the... (3 Replies)
Discussion started by: dtdt
3 Replies
6. Shell Programming and Scripting
Here is my task :
I need to sort two input files and remove duplicates in the output files :
Sort by 13 characters from 97 Ascending
Sort by 1 characters from 96 Ascending
If duplicates are found retain the first value in the file
the input files are variable length, convert... (4 Replies)
Discussion started by: ysvsr1
4 Replies
7. Shell Programming and Scripting
Hi I have a below file structure.
200,1245,E1,1,E1,,7611068,KWH,30, ,,,,,,,,
200,1245,E1,1,E1,,7611070,KWH,30, ,,,,,,,,
300,20140223,0.001,0.001,0.001,0.001,0.001
300,20140224,0.001,0.001,0.001,0.001,0.001
300,20140225,0.001,0.001,0.001,0.001,0.001
300,20140226,0.001,0.001,0.001,0.001,0.001... (1 Reply)
Discussion started by: tejashavele
1 Replies
8. Shell Programming and Scripting
values=(1 2 3 5 4 2 3 1 6 8 3 5 )
#i need the output like this by removing the duplicates
1
2
3
5
4
6
8
#i dont need sorting in my program
#plz explain me as simple using for loop
#os-ubuntu ,shell=bash (5 Replies)
Discussion started by: Meeran Rizvi
5 Replies
9. Shell Programming and Scripting
I have a below file and I wanted to remove the leading zeros in each field separated by pipe
File:
01/09/2017|2017/09/06|2017/02/06|02/06/2017|02/06/2017 06:50:06 AM|2017/02/06|02/06/2017|02/07/2017 05:45:06 AM|
02/08/2017|2017/08/06|2017/09/06|02/05/2017|02/07/2017 05:40:06... (4 Replies)
Discussion started by: Joselouis
4 Replies
10. UNIX for Beginners Questions & Answers
Hi,
I have following input file:
niki niki
niki1 niki
niki2 niki,niki2
niki3 niki,niki3,niki
niki4 niki4,blabla
niki5 jkjkl
niki6 niki60,niki6
I would like to delete lines with identical matches completely and remove the selfmatches in the other lines.
... (2 Replies)
Discussion started by: niki0211
2 Replies
LEARN ABOUT DEBIAN
bup-margin
bup-margin(1) General Commands Manual bup-margin(1)
NAME
bup-margin - figure out your deduplication safety margin
SYNOPSIS
bup margin [options...]
DESCRIPTION
bup margin iterates through all objects in your bup repository, calculating the largest number of prefix bits shared between any two
entries. This number, n, identifies the longest subset of SHA-1 you could use and still encounter a collision between your object ids.
For example, one system that was tested had a collection of 11 million objects (70 GB), and bup margin returned 45. That means a 46-bit
hash would be sufficient to avoid all collisions among that set of objects; each object in that repository could be uniquely identified by
its first 46 bits.
The number of bits needed seems to increase by about 1 or 2 for every doubling of the number of objects. Since SHA-1 hashes have 160 bits,
that leaves 115 bits of margin. Of course, because SHA-1 hashes are essentially random, it's theoretically possible to use many more bits
with far fewer objects.
If you're paranoid about the possibility of SHA-1 collisions, you can monitor your repository by running bup margin occasionally to see if
you're getting dangerously close to 160 bits.
OPTIONS
--predict
Guess the offset into each index file where a particular object will appear, and report the maximum deviation of the correct answer
from the guess. This is potentially useful for tuning an interpolation search algorithm.
--ignore-midx
don't use .midx files, use only .idx files. This is only really useful when used with --predict.
EXAMPLE
$ bup margin
Reading indexes: 100.00% (1612581/1612581), done.
40
40 matching prefix bits
1.94 bits per doubling
120 bits (61.86 doublings) remaining
4.19338e+18 times larger is possible
Everyone on earth could have 625878182 data sets
like yours, all in one repository, and we would
expect 1 object collision.
$ bup margin --predict
PackIdxList: using 1 index.
Reading indexes: 100.00% (1612581/1612581), done.
915 of 1612581 (0.057%)
SEE ALSO
bup-midx(1), bup-save(1)
BUP
Part of the bup(1) suite.
AUTHORS
Avery Pennarun <apenwarr@gmail.com>.
Bup unknown- bup-margin(1)