Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Strange Phenomena with records filed in variable

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Strange Phenomena with records filed in variable
# 1  
Old 09-25-2014
Strange Phenomena with records filed in variable
Trying to find out whether there is a limit for the number of records that can be stored in a variable I set up this small script:

Code:
#!/usr/bin/ksh 
for ((i = 1; i < 21; i++)) 
do 
  n=$(($i*100)) 
  echo "Trying $n records:" 
  recs=$(head -$n error.log) 
  echo "$recs" | wc 
done

Strangely somewhere beyond 1500 records some bytes/words/records are added to the data. The output looks like this:
Code:
Trying 100 records: 
     100    1701   14803 
Trying 200 records: 
     200    3405   29497 
Trying 300 records: 
     300    5105   44207 
Trying 400 records: 
     400    6802   58874 
Trying 500 records: 
     500    8654   74208 
Trying 600 records: 
     600   10464   89113 
Trying 700 records: 
     700   12309  104241 
Trying 800 records: 
     800   14067  119066 
Trying 900 records: 
     900   15913  134357 
Trying 1000 records: 
    1000   17700  149270 
Trying 1100 records: 
    1100   19556  164703 
Trying 1200 records: 
    1200   21441  180414 
Trying 1300 records: 
    1300   23329  195869 
Trying 1400 records: 
    1400   25225  211358 
Trying 1500 records: 
    1500   27204  227203 
Trying 1600 records: 
    1603   29132  243107 
Trying 1700 records: 
    1703   31039  258401 
Trying 1800 records: 
    1815   32952  274627 
Trying 1900 records: 
    1915   34816  289880 
Trying 2000 records: 
    2015   36687  305311

What is going on here?
# 2  
Old 09-25-2014
What happens when you change:
Code:
recs=$(head -$n error.log) 
echo "$recs" | wc

to
Code:
head -$n error.log | wc

# 3  
Old 09-25-2014
Changed script:

Code:
do
  n=$(($i*100))
  echo "Trying $n records:"
  head -$n error.log | wc
  recs=$(head -$n error.log)
  echo "$recs" | wc
done

New output:
Code:
Trying 100 records:
     100    1701   14803
     100    1701   14803
Trying 200 records:
     200    3405   29497
     200    3405   29497
Trying 300 records:
     300    5105   44207
     300    5105   44207
Trying 400 records:
     400    6802   58874
     400    6802   58874
Trying 500 records:
     500    8654   74208
     500    8654   74208
Trying 600 records:
     600   10464   89113
     600   10464   89113
Trying 700 records:
     700   12309  104241
     700   12309  104241
Trying 800 records:
     800   14067  119066
     800   14067  119066
Trying 900 records:
     900   15913  134357
     900   15913  134357
Trying 1000 records:
    1000   17700  149270
    1000   17700  149270
Trying 1100 records:
    1100   19556  164703
    1100   19556  164703
Trying 1200 records:
    1200   21441  180414
    1200   21441  180414
Trying 1300 records:
    1300   23329  195869
    1300   23329  195869
Trying 1400 records:
    1400   25225  211358
    1400   25225  211358
Trying 1500 records:
    1500   27204  227203
    1500   27204  227203
Trying 1600 records:
    1600   29129  243110
    1603   29132  243107
Trying 1700 records:
    1700   31036  258404
    1703   31039  258401
Trying 1800 records:
    1800   32937  274642
    1815   32952  274627
Trying 1900 records:
    1900   34801  289895
    1915   34816  289880
Trying 2000 records:
    2000   36672  305326
    2015   36687  305311

# 4  
Old 09-25-2014
OK, so the problem can be in the command substitution statement, since trailing white space is discarded (so if there are empty lines or spaces at the end of the head output, they will be removed)

Try:
Code:
recs=$(head -$n error.log; printf x)
echo "${recs%x}" | wc

But that will not be your problem here, since there apear to be more lines..

There can be an additional problem because you are using echo which is not standardized and may interpret special characters. Instead you could try
Code:
recs=$(head -$n error.log; printf x)
printf "%s\n" "${recs%x}" | wc

Last edited by Scrutinizer; 09-25-2014 at 09:08 PM..
# 5  
Old 10-01-2014
Sorry for the late response, shell shock kept me busy the past days.

I found the problem: \n is interpreted as newline, so a line in the file like ...

Code:
C:\Users\AD~1\AppData\Local\Temp\notes32C5CD\lnw01.gif

becomes

Code:
C:\Users\AD~1\AppData\Local\Temp
otes32C5CD\lnw01.gif

As shown above in my previous posts, the number of lines and words increases, while the number of bytes decreases.

What can be done so that such escape sequences are taken literally by printf or echo?
# 6  
Old 10-01-2014
This is bad:

Code:
printf "${some_arbitrary_variable}\n"

This is how it's intended to be used:

Code:
printf "%s\n" "${some_arbitary_varible}"

This User Gave Thanks to Corona688 For This Post:
Cochise
# 7  
Old 10-01-2014
See also answer #4
This User Gave Thanks to Scrutinizer For This Post:
Cochise
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Replace variable value in first file based on records in second

Hello , I have below files a) File A <?xml version="1.0" encoding="UTF-8" standalone="no"?> <root xmlns="http://aaa/1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema" version="2.0"> <project name="source"> <mapping name="m_Source"> <parameter... (3 Replies)
Discussion started by: Pratik4891
3 Replies

2. Shell Programming and Scripting

Assign number of records to a variable

How does one assign a variable, x to equal the number of records in a different file. I have a simple command such as below: awk -F "\t" '(NR>5) { if(($x == "0/0")) { print $0} }' a.txt > a1.txt but I want x to equal the number of records in a different file, b.txt (10 Replies)
Discussion started by: Geneanalyst
10 Replies

3. Shell Programming and Scripting

Bash - concatenate string - strange variable scoping

Hello, I am trying to concatenate a string in a bash script like this: runCmd="docker run -e \"IMAGE_NAME=$IMAGE_NAME\" " env | grep "$ENV_SUFFIX" | while read line; do envCmd="-e \"${line}\" " runCmd=$runCmd$envCmd echo $runCmd # here concatenation works fine done echo... (3 Replies)
Discussion started by: czabak
3 Replies

4. Programming

Strange value of the double type variable: -nan(0x8000000000000)

I am confused by the value of "currdisk->currangle" after adding operation. Initially the value of "currdisk->currangle" is 0.77500000000000013, but after adding operation, it's changed to "-nan(0x8000000000000)", Can anyone explain ? Thanks! The following is the occasion of gdb debugging. 3338 ... (8 Replies)
Discussion started by: 915086731
8 Replies

5. Shell Programming and Scripting

Prefix a variable in the first column of all the records of the files with and without header

In a bash shell, I have to prefix a variable to two .CSV files File1.CSV and File2.CSV. One of the files has a header and the other one is with no header in the below format: "value11","value12","value13","value14","value15","value16" "value21","value22","value23","value24","value25","value26"... (7 Replies)
Discussion started by: dhruuv369
7 Replies

6. Shell Programming and Scripting

Strange variable comparison result in awk

So, I'm making a little awk script that generates a range-based histogram of a set of numbers. I've stumbled onto a strange thing. Toward the end of the process, I have this test: if ( bindex < s ) "bindex" is the "index" of my "bin" (the array element that gets incremented whenever a... (2 Replies)
Discussion started by: treesloth
2 Replies

7. Shell Programming and Scripting

filter out all the records which are having space in the 8th filed of my file

I have a file which is having fileds separtaed by delimiter. Ex: C;4498;qwa;cghy;;;;40;;222122 C;4498;sample;city;;;;34 2;;222123 C;4498;qwe;xcbv;;;;34-2;;222124 C;4498;jj;sffz;;;;41;;222120 C;4498;eert;qwq;;;;34 A;;222125 C;4498;jj;szxzzd;;;;34;;222127 out of these records I... (3 Replies)
Discussion started by: indusri
3 Replies

8. Shell Programming and Scripting

Using a variable to select records with awk

As part of a bigger task, I had to read thru a file and separate records into various batches based on a field. Specifically, separate records based on the value in the batch field as defined below. The batch field left-justified numbers. The datafile is here > cat infile 12345 1 John Smith ... (5 Replies)
Discussion started by: joeyg
5 Replies

9. Shell Programming and Scripting

Using awk to search variable length records

New to awk and need some help. I have a script that I would like to make more compact. I want to read a file and grab every field, from every record, except the last field. The records are variable length and have varying number of fields. A record will have at least two fields, but can have... (9 Replies)
Discussion started by: synergy_texas
9 Replies
Login or Register to Ask a Question