Can't input large file to variable


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Can't input large file to variable
# 1  
Old 06-25-2018
Can't input large file to variable

I have a file that I'm trying to place into a variable.
I've tried the following:

Code:
DATE=`date +%Y%m%d`
filez=$(cat /tmp/Test_$DATE.txt)

Code:
DATE=`date +%Y%m%d`
filez=$(</tmp/Test_$DATE.txt)

Code:
DATE=`date +%Y%m%d`
filez=$(`cat /tmp/Test_$DATE.txt`)

None of these lines allows the file to go into the variable, unless the file is small. If it is a small file, then it works. If it is a large file, I get the following error:

Code:
builds/ul11u3sru-gate/components/bash/bash-4.4/lib/sh/strtrans.c:66: cannot allocate 1448426861 bytes (3758187128 bytes allocated)
./scripttest.bash: xmalloc: /builds/ul11u3sru-gate/components/bash/bash-4.4/lib/sh/strtrans.c:66: cannot allocate 1448426861 bytes (3758187128 bytes allocated)
./scripttest.bash: xmalloc: /builds/ul11u3sru-gate/components/bash/bash-4.4/lib/sh/strtrans.c:66: cannot allocate 1448426861 bytes (3758187128 bytes allocated)

Anyone know how to fix this? Apparently the file size causes it.
# 2  
Old 06-25-2018
Now, that I'd call ambitious: Define a 5GB variable!

Not sure you exceeded any system / configuration parameter, though. Looks more like you hit a limit of the virtual memory system / manager.

What be the reason to read an entire file into a variable in the first place?
This User Gave Thanks to RudiC For This Post:
# 3  
Old 06-25-2018
There are memory limits and this depends a lot on the version of bash you operating system, etc. You should restructure your script so it doesn't need to read the whole file into a variable.

Consider reading 1 line at a time from the file, or using external utilities to manipulate the file for you.
# 4  
Old 06-25-2018
You almost certainly can accomplish exactly what you want without assigning that file to a variable. Few languages but shell afford you such easy and direct methods of streaming data anywhere you want.

What exactly do you want to do? Why do you want a 5 gig variable - what will you do with it?
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Checking the file depending on the input variable

Hi I have a requirement of taking time as input variable outside the script.depending on the time it will check the file output .like ./script.sh <30 min> so script.sh should run every 5 minutes ie.6 times to check the output file.Can any one please help here. (7 Replies)
Discussion started by: netdbaind
7 Replies

2. Shell Programming and Scripting

XML variable for input in same input file

Dear All , i stuck in one problem executing xml .. i have input xml as <COMMAND name="ARRANGEMENT.WRITE" timestamp="0" so="initial"> <SVLOBJECT> <LONG name="CSP_PMNT_ID" val="-1"/> <MONEY name="CSP_CEILING" amount="0.0" currency="AUD"/> ... (6 Replies)
Discussion started by: arvindng
6 Replies

3. Shell Programming and Scripting

awk input large file

Hi...Does anyone know how to input huge file about 25 GB to awk if single file then this works awk '{print}' <hugefile suppose if have to use something like this awk FNR==NR{x=$0;next}{print $0,x}' hugefile1 hugefile2 then how to redirect ? and is there any provision to assign memory... (12 Replies)
Discussion started by: Akshay Hegde
12 Replies

4. Shell Programming and Scripting

Need to delete large set of files (i.e) close to 100K from a directory based on the input file

Hi all, I need a script to delete a large set of files from a directory under / based on an input file and want to redirect errors into separate file. I have already prepared a list of files in the input file. Kndly help me. Thanks, Prash (36 Replies)
Discussion started by: prash358
36 Replies

5. Shell Programming and Scripting

awk built-in variable for input file

Hi guys, Does awk have a built-in variable which I can use to display the input file it's currently reading? I'm currently concatenating multiple files using awk and later on do some parsing. But for now, I want to add an extra column in the main output data file - basically putting in the... (3 Replies)
Discussion started by: Det7
3 Replies

6. Shell Programming and Scripting

Reading from a File and Using as an Input variable

I need to know how the the string constant from Input File should be read and provide as input data for the script . INPUT FILE CONST VARIABLE myname=/root/dir/syslog/myname1 myname=/root/dir/syslog/myname2 myname=/root/dir/syslog/myname3 urname=/root/dir/syslog/urname1... (6 Replies)
Discussion started by: baraghun
6 Replies

7. Shell Programming and Scripting

Find and analyze variable Strings in a large file.

Hi guys, I have multiple files (>5000) which I have in a folder. I read every file name and put it in a tmp variable "i" so that i can use it in the following task. I have a large .txt file (>50 MB) in which I want to find the variable "i" and the lines after this variable, so that I can use... (4 Replies)
Discussion started by: Ashitaka007
4 Replies

8. Shell Programming and Scripting

KSH: Compare variable to $1 in an input file

Hello, I am working with KSH on AIX and I have 2 files generated from different sources... as seen below: FILE1 FILE2 AAA AAA@ABS0001C BBB BBB@ABS0003D CCC CCC@ABS0023A DDD DDD@ABC0145D EEE EEE@ABS0090A FFF FFF@ABS0002A GGG GGG@ABC0150D HHH FILE1 is main main data source,... (4 Replies)
Discussion started by: right_coaster
4 Replies

9. UNIX and Linux Applications

Input a variable and write to a file using awk

Hi I am trying to edit a csv file. Bacically I need to input a search variable and the value that must be changed in one of the fields corresponding to that searched variable. My csv file looks like so: 1,1A,5 1,1B,2 1,1C,3 2,2A,7 2,2B,4 2,2C,0 3,3A,1 3,3B,6 3,3C,4 I want to... (4 Replies)
Discussion started by: ladyAnne
4 Replies

10. Shell Programming and Scripting

How to get variable input from a file

Hi guys, I want to input some variables from a file. I tried it with following script, for i in 1 2 3 4 do read varible1 varible2 < ddd.txt echo $varible1 $varible2 done ... (1 Reply)
Discussion started by: maheshsri
1 Replies
Login or Register to Ask a Question