04-30-2009
4 GB delimited-textfile on ONE LINE
I have delimited-text files ( > 4GB ) and is just one line.
OS: HP-UX 11.23
Awk / cut / sed all have line_max limitations. & unable to read one line in (Buffered-mode).
Sample file:
xxxx|adsfadf|Afdsa|adsf|afds|Asdfas|ads|Afds|Asdf| .....till forever,
I want to put a carriage return every 17 fields.
xxx|asdfad|asdfadsf|adfsadsf
afdsadsf|adfsadsf|Afds|adsf|Asdf
...this has to be in -unbuffered mode..
as i tried to fold -w 500 inputfile > outputfile
then while loop (line-by-line) .. read multiples of 17 fields & append remaing to next line..
see trials below:
#!/usr/bin/sh -x
inputfile=$1
foldfile=${inputfile}.fold
outputfile=${inputfile}.new
i=0
multiplier=17 ### This is 17 Fields to be split
##gawk -F"|" 'OFS="," { for (i=1;i<=NF;i += 17) { print $i, $(i+1 ), $(i+2) , $(i+3) , $(i+4) , $(i+5) , $(i+6) , $(i+7) , $(i+8) , $(i+9) , $(i+10) , $(i+11) , $(i+12) , $(i+13) , $(i+14) , $(i+15) , $(i+16) } fflush(""); } ' $inputfile > $outputfile
fold -w 500 $inputfile > $foldfile
truncated_fields=''
while read line
do
line=`echo $truncated_fields$line`
fields_per_line=`echo $line | awk -F"|" '{print NF}'`
## ex: 69 / 17 = 4 loops + 1 remaining field
fields_multiples_multiplier=`expr ${fields_per_line} / ${multiplier}`
start_index_of_trunc_fields=`expr ${fields_multiples_multiplier} \* ${multiplier} + 1`
if [ $start_index_of_trunc_fields -gt $fields_per_line ]; then
$start_index_of_trunc_fields=$fields_per_line
fi
while [ $i -ne $fields_multiples_multiplier ]
do
start=`expr ${multiplier} \* ${i} + 1 `
end=`expr ${start} + 16 `
echo $line | cut -d"|" -f${start}-${end} >> $outputfile
i=`expr ${i} + 1`
done
truncated_fields=`echo $line | cut -d"|" -f${start_index_of_trunc_fields}-${fields_per_line}`
## echo $fields_multiples_multiplier $start_index_of_trunc_fields $fields_per_line
## echo $truncated_fields
## sleep 10
done < $foldfile
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
I have a text file of lines like:
A=5|B=7|G=4|C=3|P=4|...
In other words, each line is a pipe-delimited set of pairs of strings of the form "X=Y".
What I want to do is find the token starting with "C", and print it and its value (so I'd want to print "C=3" in the example above).
I'm... (11 Replies)
Discussion started by: monkeys
11 Replies
2. Shell Programming and Scripting
i need to cut the string in a textfile but each line has a specific way of cutting it (different lengths)
i have a for loop that gets the string line per line, then each line has to be compared:
for x in `cat tmp2.txt`; do
if; then
echo 'BAC'
elif
... (6 Replies)
Discussion started by: izuma
6 Replies
3. Shell Programming and Scripting
Hi!
I need to do the following:
(1) I wan't to extract a line of a textfile (defined by a numer) and store it into a variable...
(2) ...but I want to cut out a part of the line which is between two tokens and store just this to the variable
Example:
BlaBlaBla Bla2Bla2Bla2 *pPointerOne;... (4 Replies)
Discussion started by: Michi21609
4 Replies
4. Shell Programming and Scripting
Hi, I'm starting a little project with a shell script but I'm don't know how to do it. Maybe someone can help me.
I have un text file like this :
I'd like to do a script who will extract from my file from @ADDLINE1@ to @ADDLINE4@ only and I have no idea how to do this.
Any idea ?
... (7 Replies)
Discussion started by: Poulki
7 Replies
5. UNIX for Dummies Questions & Answers
How can i make a script to add text before the first word on a line in a textfile :
Example:
Old line:
is my place
New line:
this is my place
Please use and tags when posting code, data or logs etc. to preserve formatting and enhance readability, thanks. (3 Replies)
Discussion started by: mjanssen
3 Replies
6. Shell Programming and Scripting
Hello everyone,
I don't really know anything about scripting, but I have to manage to make this script, out of necessity.
#!/bin/bash
while read -r line; do #I'm reading from a big wordlist
instructions using $line
done
Is there a way to automatically write the $line number the script... (4 Replies)
Discussion started by: bobylapointe
4 Replies
7. Shell Programming and Scripting
Awk gurus,
Greatly appreciate for any kind of assistance from the expert community
Input line:
abc,11.22.33.44,xyz,7-8-9-10
pqr,111.222.333.444,wxy,1-2-3
def,22.33.44.55,stu,7-8
used the gsub function below but it changes all of the "-" delimiter:
awk 'gsub("-",",")'
Desired... (4 Replies)
Discussion started by: ux4me
4 Replies
8. Shell Programming and Scripting
Hello everybody
I have a text file which has the following format:
nmm "text20140601.033954text" "text" "text"
"text" , ... , "text"
"text" , ... , Lat 36.3247 Lon 16.0588 Depth 8
"text", ... , "text"
"text", ..., CovXX 1.65 CovYY 2.32 CovZZ 1.2
"text" , ..., "text
nmm ... (6 Replies)
Discussion started by: phaethon
6 Replies
9. Shell Programming and Scripting
My first post, so don't kill me :)
Say i open some textfile with some example like this.
on the table are handy, bread and wine
Now i know exactly what is in and i want to separate and sorted it in terminal to an existing file with another 2 existing lines in like this:
table
plane ... (3 Replies)
Discussion started by: schwatter
3 Replies
10. Shell Programming and Scripting
Hi All,
I am new to Unix and I have one challenge and below are the details.
I have pipe delimited text file in that data has span into multiple lines instead of single line.
Sample data.
Data should be like below for entire file.
41|216|398555|77|provided complete NP outcome data ... (21 Replies)
Discussion started by: Narasimhasss
21 Replies
SUBST(1) General Commands Manual SUBST(1)
NAME
subst - substitute definitions into file(s)
SYNOPSIS
subst [ -e editor ] -f substitutions victim ...
DESCRIPTION
Subst makes substitutions into files, in a way that is suitable for customizing software to local conditions. Each victim file is altered
according to the contents of the substitutions file.
The substitutions file contains one line per substitution. A line consists of two fields separated by one or more tabs. The first field
is the name of the substitution, the second is the value. Neither should contain the character `#', and use of text-editor metacharacters
like `&' and `' is also unwise; the name in particular is best restricted to be alphanumeric. A line starting with `#' is a comment and
is ignored.
In the victims, each line on which a substitution is to be made (a target line) must be preceded by a prototype line. The prototype line
should be delimited in such a way that it will be taken as a comment by whatever program processes the file later. The prototype line must
contain a ``prototype'' of the target line bracketed by `=()<' and `>()='; everything else on the prototype line is ignored. Subst
extracts the prototype, changes all instances of substitution names bracketed by `@<' and `>@' to their values, and then replaces the tar-
get line with the result.
OPTIONS
-e Substitutions are done using the sed(1) editor, which must be found in either the /bin or /usr/bin directories. To specify a dif-
ferent executable, use the ``-e'' flag.
EXAMPLE
If the substitutions file is
FIRST 111
SECOND 222
and the victim file is
x = 2;
/* =()<y = @<FIRST>@ + @<SECOND>@;>()= */
y = 88 + 99;
z = 5;
then ``subst -f substitutions victim'' changes victim to:
x = 2;
/* =()<y = @<FIRST>@ + @<SECOND>@;>()= */
y = 111 + 222;
z = 5;
FILES
victimdir/substtmp.new new version being built
victimdir/substtmp.old old version during renaming
SEE ALSO
sed(1)
DIAGNOSTICS
Complains and halts if it is unable to create its temporary files or if they already exist.
HISTORY
Written at U of Toronto by Henry Spencer.
Rich $alz added the ``-e'' flag July, 1991.
BUGS
When creating a file to be substed, it's easy to forget to insert a dummy target line after a prototype line; if you forget, subst ends up
deleting whichever line did in fact follow the prototype line.
25 Feb 1990 SUBST(1)