Sponsored Content
Top Forums Shell Programming and Scripting Extract large list of substrings Post 302233883 by cfajohnson on Monday 8th of September 2008 05:34:12 PM
Old 09-08-2008
Quote:
Originally Posted by dcfargo
I have a very long string (millions of characters).

Where do you have it? Is it in a file? In a variable?

Are there any newlines in the string?
Quote:
I have a file with start location and length that is thousands of rows long:

Start Length
5 10
16 21
44 100
215 37
...

I'd like to extract the substring that corresponds to the start and length from each row of the list:

I tried just using a large awk '{print substr($1,5,10), "\n", substr($1,16,21) "\n", substr($1,44,100) "\n", substr($1,215,37)...}' infile > outfile &

command

But it seems to hang likely because the Bash line is too long.

I have no problem extracting portions of a multimegabyte string using bash's parameter expansion:
Code:
## Assuming the string is in 'infile'
string=$( < infile )
while read start length
do
  printf "%s\n" "${string:$start:$length}"
done < /path/to/file/with/startpoints_and_lengths

 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

List large files

Hi I need to list all files in the system: 1. Greater than specific size 2. All files sorted by size How can I do that? Thanks in advance. (2 Replies)
Discussion started by: GNMIKE
2 Replies

2. Shell Programming and Scripting

Need to extract 7 characters immediately after text '19' from a large file.

Hi All!! I have a large file containing millions of record. My purpose is to extract 7 characters immediately after text '19' from this file (including text '19') and save the result in new file. So, my OUTPUT would be as under : 191234561 194567894 192789005 198839408 and so on..... ... (7 Replies)
Discussion started by: parshant_bvcoe
7 Replies

3. Shell Programming and Scripting

Extract data from large file 80+ million records

Hello, I have got one file with more than 120+ million records(35 GB in size). I have to extract some relevant data from file based on some parameter and generate other output file. What will be the besat and fastest way to extract the ne file. sample file format :--... (2 Replies)
Discussion started by: learner16s
2 Replies

4. Shell Programming and Scripting

extract unique pattern from large text file

Hi All, I am trying to extract data from a large text file , I want to extract lines which contains a five digit number followed by a hyphen , like 12345- , i tried with egrep ,eg : egrep "+" text.txt but which returns all the lines which contains any number of digits followed by hyhen ,... (19 Replies)
Discussion started by: shijujoe
19 Replies

5. Shell Programming and Scripting

Extract information into large variable

Hello people :) That's here my first message to your forum, so I guess it would be fine ^^ I have a request about a code I want to use. Actually, my system use a large variable, including much informations but those informations can change by more and I want to extract one of thoses... (26 Replies)
Discussion started by: WolwX
26 Replies

6. Shell Programming and Scripting

Extract three substrings from a logfile

I have a log file like below. 66.249.73.11 - - "UCiZ7QocVqYAABgwfP8AAHAA" "US" "Mediapartners-Google" "-" www.mahashwetha.com.sg "GET... (2 Replies)
Discussion started by: Tuxidow
2 Replies

7. Shell Programming and Scripting

Curl download zip extract large xml file

Hi i have a php script that works 100% however i don't want this to run on php because of server limits etc. Ideally if i could convert this simple php script to a shell script i can set it up to run on a cron. My mac server has curl on it. So i am assuming i should be using this to download the... (3 Replies)
Discussion started by: timgolding
3 Replies

8. UNIX for Dummies Questions & Answers

Extract spread columns from large file

Dear all, I want to extract around 300 columns from a very large file with almost 2million columns. There are no headers, but I can find out which column numbers I want. I know I can extract with the function 'cut -f2' for example just the second column but how do I do this for such a large... (1 Reply)
Discussion started by: fndijk
1 Replies

9. Shell Programming and Scripting

Need to extract 8 characters from a large file.

Hi All!! I have a large file containing millions of records. My purpose is to extract 8 characters immediately from the given file. 222222222|ZRF|2008.pdf|2008|01/29/2009|001|B|C|C 222222222|ZRF|2009.pdf|2009|01/29/2010|001|B|C|C 222222222|ZRF|2010.pdf|2010|01/29/2011|001|B|C|C... (5 Replies)
Discussion started by: pavand
5 Replies

10. UNIX for Beginners Questions & Answers

Command to extract empty field in a large UNIX file?

Hi All, I have records in unix file like below. In this file, we have empty fields from 4th Column to 22nd Column. I have some 200000 records in a file. I want to extract records only which have empty fields from 4th field to 22nd filed. This file is comma separated file. what is the unix... (2 Replies)
Discussion started by: rakeshp
2 Replies
SUBSTR(3)								 1								 SUBSTR(3)

substr - Return part of a string

SYNOPSIS
string substr (string $string, int $start, [int $length]) DESCRIPTION
Returns the portion of $string specified by the $start and $length parameters. PARAMETERS
o $string - The input string. Must be one character or longer. o $start - If $start is non-negative, the returned string will start at the $start'th position in $string, counting from zero. For instance, in the string ' abcdef', the character at position 0 is ' a', the character at position 2 is ' c', and so forth. If $start is negative, the returned string will start at the $start'th character from the end of $string. If $string is less than or equal to $start characters long, FALSE will be returned. Example #1 Using a negative $start <?php $rest = substr("abcdef", -1); // returns "f" $rest = substr("abcdef", -2); // returns "ef" $rest = substr("abcdef", -3, 1); // returns "d" ?> o $length - If $length is given and is positive, the string returned will contain at most $length characters beginning from $start (depend- ing on the length of $string). If $length is given and is negative, then that many characters will be omitted from the end of $string (after the start position has been calculated when a $start is negative). If $start denotes the position of this trunca- tion or beyond, false will be returned. If $length is given and is 0, FALSE or NULL, an empty string will be returned. If $length is omitted, the substring starting from $start until the end of the string will be returned. Example #2 Using a negative $length <?php $rest = substr("abcdef", 0, -1); // returns "abcde" $rest = substr("abcdef", 2, -1); // returns "cde" $rest = substr("abcdef", 4, -4); // returns false $rest = substr("abcdef", -3, -1); // returns "de" ?> RETURN VALUES
Returns the extracted part of $string; or FALSE on failure, or an empty string. CHANGELOG
+--------------+---------------------------------------------------+ | Version | | | | | | | Description | | | | +--------------+---------------------------------------------------+ |5.2.2 - 5.2.6 | | | | | | | If the $start parameter indicates the position | | | of a negative truncation or beyond, false is | | | returned. Other versions get the string from | | | start. | | | | +--------------+---------------------------------------------------+ EXAMPLES
Example #3 Basic substr(3) usage <?php echo substr('abcdef', 1); // bcdef echo substr('abcdef', 1, 3); // bcd echo substr('abcdef', 0, 4); // abcd echo substr('abcdef', 0, 8); // abcdef echo substr('abcdef', -1, 1); // f // Accessing single characters in a string // can also be achieved using "square brackets" $string = 'abcdef'; echo $string[0]; // a echo $string[3]; // d echo $string[strlen($string)-1]; // f ?> Example #4 substr(3) casting behaviour <?php class apple { public function __toString() { return "green"; } } echo "1) ".var_export(substr("pear", 0, 2), true).PHP_EOL; echo "2) ".var_export(substr(54321, 0, 2), true).PHP_EOL; echo "3) ".var_export(substr(new apple(), 0, 2), true).PHP_EOL; echo "4) ".var_export(substr(true, 0, 1), true).PHP_EOL; echo "5) ".var_export(substr(false, 0, 1), true).PHP_EOL; echo "6) ".var_export(substr("", 0, 1), true).PHP_EOL; echo "7) ".var_export(substr(1.2e3, 0, 4), true).PHP_EOL; ?> The above example will output: 1) 'pe' 2) '54' 3) 'gr' 4) '1' 5) false 6) false 7) '1200' ERRORS
/EXCEPTIONS Returns FALSE on error. Example #5 <?php var_dump(substr('a', 1)); // bool(false) ?> SEE ALSO
strrchr(3), substr_replace(3), preg_match(3), trim(3), mb_substr(3), wordwrap(3), String access and modification by character. PHP Documentation Group SUBSTR(3)
All times are GMT -4. The time now is 01:39 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy