10 More Discussions You Might Find Interesting
1. UNIX for Advanced & Expert Users
Hi,
I have received a file which is 20 GB. We would like to split the file into 4 equal parts and process it to avoid memory issues.
If the record delimiter is unix new line, I could use split command either with option l or b.
The problem is that the line terminator is |##|
How to use... (5 Replies)
Discussion started by: Ravi.K
5 Replies
2. Shell Programming and Scripting
Hi,
I have one tab delimited file which is having multiple store_ids in first column seprated by pipe.I want to split the file on the basis of store_id(separating 1st record in to 2 records ).
I tried some more options like below with using split,awk etc ,But not able to get proper output. can... (1 Reply)
Discussion started by: jaggy
1 Replies
3. UNIX for Dummies Questions & Answers
Hi Friends,
source
....
col1,col2,col3
a,b,1;2;3
here colom delimeter is comma(,).
here we dont know what is the max length of col3 means now we have 1;2;3 next time i will receive 1;2;3;4;5;etc...
required output
..............
col1,col2,col3
a,b,1
a,b,2
a,b,3
please give me... (5 Replies)
Discussion started by: bab.galary
5 Replies
4. Shell Programming and Scripting
Hi,
Help required to split record value and write to new row.
Input
a~b~c~value in ('3','4','5')~test
output
a~b~c~3~test
a~b~c~4~test
a~b~c~5~test
input
a~b~c~value in ('3','4')~test
output
a~b~c~3~test
a~b~c~4~test (8 Replies)
Discussion started by: Jairaj
8 Replies
5. UNIX for Dummies Questions & Answers
Hi ,
I have file with all records in one line, which needs to split it to have a fixed length.Am trying to execute the below script for the same
FILENAME="$1"
while line LINE
do
echo $LINE | awk 'BEGIN{n=1}{while(substr($0,n,10)){print substr($0,n,10);n+=10}}'
done < $FILENAME
it... (4 Replies)
Discussion started by: nishantrk
4 Replies
6. Shell Programming and Scripting
Hi,
My inputfile contains field separaer is ^.
12^inms^
13^fakdks^ssk^s3
23^avsd^
13^fakdks^ssk^a4
I wanted to print only 2 delimiter occurence i.e
12^inms^
23^avsd^ (4 Replies)
Discussion started by: Jairaj
4 Replies
7. Shell Programming and Scripting
I want to keep only records contain length is 10 other records should remove from my original file without redirecting to other output file.
Source
1234567890
123456789011234
abcdefghil
Expected Result
1234567890
abcdefghil (9 Replies)
Discussion started by: Jairaj
9 Replies
8. Shell Programming and Scripting
Hi ,
I have a requirement to split the record based on particular match using UNIX.
Case1:
Input Record :
10.44.48.63;"Personals/Dating;sports";1441
Output Records :
10.44.48.63;Personals/Dating;1441;Original
10.44.48.63;sports;1441;Dummy
Case2:
Input Record : ... (5 Replies)
Discussion started by: mksuneel
5 Replies
9. Shell Programming and Scripting
-Hi, I have a problem with parcing/spliting a file record into two parts and assigning the split parts to two viriables. The record is as follows:
ftrn facc ttrd feed xref fsdb fcp ruldb csdb omom fordr ftxn fodb fsdc texc oxox reng ttrn ttxn fqdb ... (5 Replies)
Discussion started by: aoussenko
5 Replies
10. Shell Programming and Scripting
UNIX Scripting
Hi I am trying to read a record and split it into multiple records
My Record looks like this
1001A0010@B0010*&^0)C0012hgdj&6sD0020fhfri93kivmepi9
where
UniqueID is 1001
segments are A,B,C,D
length of each segment is 4 characters after the segment 0010 for A
0010 for B 0012... (5 Replies)
Discussion started by: pukars4u
5 Replies
bup-margin(1) General Commands Manual bup-margin(1)
NAME
bup-margin - figure out your deduplication safety margin
SYNOPSIS
bup margin [options...]
DESCRIPTION
bup margin iterates through all objects in your bup repository, calculating the largest number of prefix bits shared between any two
entries. This number, n, identifies the longest subset of SHA-1 you could use and still encounter a collision between your object ids.
For example, one system that was tested had a collection of 11 million objects (70 GB), and bup margin returned 45. That means a 46-bit
hash would be sufficient to avoid all collisions among that set of objects; each object in that repository could be uniquely identified by
its first 46 bits.
The number of bits needed seems to increase by about 1 or 2 for every doubling of the number of objects. Since SHA-1 hashes have 160 bits,
that leaves 115 bits of margin. Of course, because SHA-1 hashes are essentially random, it's theoretically possible to use many more bits
with far fewer objects.
If you're paranoid about the possibility of SHA-1 collisions, you can monitor your repository by running bup margin occasionally to see if
you're getting dangerously close to 160 bits.
OPTIONS
--predict
Guess the offset into each index file where a particular object will appear, and report the maximum deviation of the correct answer
from the guess. This is potentially useful for tuning an interpolation search algorithm.
--ignore-midx
don't use .midx files, use only .idx files. This is only really useful when used with --predict.
EXAMPLE
$ bup margin
Reading indexes: 100.00% (1612581/1612581), done.
40
40 matching prefix bits
1.94 bits per doubling
120 bits (61.86 doublings) remaining
4.19338e+18 times larger is possible
Everyone on earth could have 625878182 data sets
like yours, all in one repository, and we would
expect 1 object collision.
$ bup margin --predict
PackIdxList: using 1 index.
Reading indexes: 100.00% (1612581/1612581), done.
915 of 1612581 (0.057%)
SEE ALSO
bup-midx(1), bup-save(1)
BUP
Part of the bup(1) suite.
AUTHORS
Avery Pennarun <apenwarr@gmail.com>.
Bup unknown- bup-margin(1)