Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

awk (Too many open files)

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting awk (Too many open files)
# 8  
Old 01-29-2018
Nice solution disedorgue.

However, I did have some issues running under bash 4.4.12(3) in a cygwin environment. It appears that the pipe write sometimes blocks waiting for a reader, causing things to lock up after a sha1 is calculated for the first line.

After a little bit of playing with it I came up with this modification, that appears to be a little more stable and reduces the number of FIFOs to 3.

Here I use file descriptor 7 and 9 for the read and write ends of std0. Avoiding using the external awk program, should also have it run faster:

Code:
func_openssl ()
{ 
  exec 7< std0
  openssl > /dev/null < <(
    while read -u 7 mdp
    do
      printf "%s\n" "dgst -sha256 -out out0 in0"
      printf "%s\n" "$mdp" >in0
    done
  )
}

func_mngopenssl ()
{
    exec 9> std0
    while read a1 b1 c1 d1 e1 f1; do
        printf "%s\n" "$e1" >&9
        IFS=" " read discard e1sha < out0
        printf "%s\n" "$a1 $b1 $c1 $d1 $e1sha $f1"
    done
    exec 9>&-
    exec 7<&-
}

trap 'rm std0 in0 out0' EXIT
mkfifo std0 in0 out0

func_openssl & func_mngopenssl <fichier

# 9  
Old 01-29-2018
After some more testing it appears that the 2 in and out FIFOs are needed to avoid some syncing issues which cause records to be missed. Update below reverts to using in0 in1 out0 and out1.

Using pipe std0 and FD #7 and #9 to pass date to func_openssl() appears to not suffer from this issue on the systems I've tested so I've kept is as before.

Code:
func_openssl ()
{
  exec 7< std0
  openssl > /dev/null < <(
    while read -u 7 mdp
    do
      printf "%s\n" "dgst -sha256 -out out$((++k%2)) in$((k%2))"
      printf "%s\n" "$mdp" >in$((k%2))
    done
  )
}

func_mngopenssl ()
{
    exec 9> std0
    while read a1 b1 c1 d1 e1 f1; do
        echo "$e1" >&9
        IFS=" " read discard e1sha < out$((++j%2))
        printf "%s\n" "$a1 $b1 $c1 $d1 $e1sha $f1";
    done;
    exec 9>&-
    exec 7<&-
}

trap 'rm std0 in0 out0 in1 out1' EXIT
mkfifo std0 in0 out0 in1 out1

func_openssl & func_mngopenssl <fichier

This User Gave Thanks to Chubler_XL For This Post:
disedorgue
# 10  
Old 01-30-2018
Hello Chubler_XL,
Your solution is more worked and certainly more stable because the file descriptors are already existing before the first use of openssl.

Regards.
# 11  
Old 02-07-2018
Hi,
In another french forum for similar problem, I give a solution like as:
Code:
#!/bin/bash
mkfifo in0 in1 out0 out1
openssl < <(awk 'BEGIN{while(1){print "dgst -out out0 -sha256 in0\ndgst -out out1 -sha256 in1"}}') > /dev/null &
while true ;  do IFS=" " read T X <out$((h++%2)) ; echo -n "$X" ; done &

while read a1 a2 a3 a4 a5 a6  ; do echo -n "$a1 $a2 $a3 $a4 " ; echo -n $a5 >in$((h++%2)) ; echo " $a6" ; done <file

exec 2>/dev/null
jobs -p | xargs kill
rm -f in0 in1 out0 out1

Duration of this method on my computer for 100000 iterations:
real 0m42,974s
user 0m33,780s
sys 0m29,528s

Duration of your previous method on my computer for 100000 iterations:
real 1m3,779s
user 0m32,084s
sys 0m27,428s
# 12  
Old 02-07-2018
Quote:
Originally Posted by disedorgue
Hi,
In another french forum for similar problem, I give a solution like as:
Code:
#!/bin/bash
mkfifo in0 in1 out0 out1
openssl < <(awk 'BEGIN{while(1){print "dgst -out out0 -sha256 in0\ndgst -out out1 -sha256 in1"}}') > /dev/null &
while true ;  do IFS=" " read T X <out$((h++%2)) ; echo -n "$X" ; done &

while read a1 a2 a3 a4 a5 a6  ; do echo -n "$a1 $a2 $a3 $a4 " ; echo -n $a5 >in$((h++%2)) ; echo " $a6" ; done <file

exec 2>/dev/null
jobs -p | xargs kill
rm -f in0 in1 out0 out1

I am getting some synchronization issues with that code:

file:
Code:
word_01 word_02 word_03 word_04 word_05 word_06 word_07
word_11 word_12 word_13 word_14 word_15 word_16 word_17
word_21 word_22 word_23 word_24 word_25 word_26 word_27
word_31 word_32 word_33 word_34 word_35 word_36 word_37
word_41 word_42 word_43 word_44 word_45 word_46 word_47
word_51 word_52 word_53 word_54 word_55 word_56 word_57
word_61 word_62 word_63 word_64 word_65 word_66 word_67
word_71 word_72 word_73 word_74 word_75 word_76 word_77
word_81 word_82 word_83 word_84 word_85 word_86 word_87
word_91 word_92 word_93 word_94 word_95 word_96 word_97
word_A1 word_A2 word_A3 word_A4 word_A5 word_A6 word_A7
word_B1 word_B2 word_B3 word_B4 word_B5 word_B6 word_B7

results:
Code:
word_01 word_02 word_03 word_04  word_06 word_07
word_11 word_12 word_13 word_14 81233e976f89c167d338d9a8cba97fb1e063b4c4514f06693a65c067c36afcdd word_16 word_17
word_21 word_22 word_23 word_24 28829970d9ac7c53e14a246c98fb11e9cb24f4c91a3b03cacf227144cd135334 word_26 word_27
word_31 word_32 word_33 word_34 5a85b70bc0a50d7fa9ad630652ded2e48a3716aa75438eaef7557312c01d7eb5 word_36 word_37
word_41 word_42 word_43 word_44 973568825cf106a3331aea7c42d05b2bdcbd4b4b2761e1efc9043ad5537485cc word_46 word_47
word_51 word_52 word_53 word_54 4fb3737155d728cf1efe4581e980707b40184bedc59eec9523848aa27d4e78b0 word_56 word_57
word_61 word_62 word_63 word_64 4b3a529aa0b718a93c8133647c8cb170887f86d2f10886a39154b88880421f8c word_66 word_67
word_71 word_72 word_73 word_74 14b325115cf1ac8d72920470fe9b6047b753bdd9b33738e856997a3dc29eaef1 word_76 word_77
word_81 word_82 word_83 word_84 d04ce42feb8e2804061d371e0c9c0dedb1f337ca21a33fe6020fcf1f288631d0 word_86 word_87
word_91 word_92 word_93 word_94 005373e5b47d5114156460f869db467bf5d54c89e5fd530a4c395f8eb7c485c1 word_96 word_97
word_A1 word_A2 word_A3 word_A4 9700e3ec4af85ae7546d52bb0aa15f4c42c1aed94ba5da9c18273b4e16751ef0 word_A6 word_A7
word_B1 word_B2 word_B3 word_B4 97b2728ca61a338db68c64ad13b9b51db338d27d00b56f98acc529b5ff4929e9 word_B6 word_B7
d2679e2780dab58874e4a07e17d3944c6c3fe4e0edeb0b1eed767a92bc60d2ef

Code:
word_01 word_02 word_03 word_04  word_06 word_07
word_11 word_12 word_13 word_14 81233e976f89c167d338d9a8cba97fb1e063b4c4514f06693a65c067c36afcdd word_16 word_17
word_21 word_22 word_23 word_24 28829970d9ac7c53e14a246c98fb11e9cb24f4c91a3b03cacf227144cd135334 word_26 word_27
word_31 word_32 word_33 word_34 5a85b70bc0a50d7fa9ad630652ded2e48a3716aa75438eaef7557312c01d7eb5 word_36 word_37
973568825cf106a3331aea7c42d05b2bdcbd4b4b2761e1efc9043ad5537485ccword_41 word_42 word_43 word_44  word_46 word_47
word_51 word_52 word_53 word_54 4fb3737155d728cf1efe4581e980707b40184bedc59eec9523848aa27d4e78b0 word_56 word_57
word_61 word_62 word_63 word_64 4b3a529aa0b718a93c8133647c8cb170887f86d2f10886a39154b88880421f8c word_66 word_67
word_71 word_72 word_73 word_74 14b325115cf1ac8d72920470fe9b6047b753bdd9b33738e856997a3dc29eaef1 word_76 word_77
word_81 word_82 word_83 word_84 d04ce42feb8e2804061d371e0c9c0dedb1f337ca21a33fe6020fcf1f288631d0 word_86 word_87
word_91 word_92 word_93 word_94 005373e5b47d5114156460f869db467bf5d54c89e5fd530a4c395f8eb7c485c1 word_96 word_97
word_A1 word_A2 word_A3 word_A4 9700e3ec4af85ae7546d52bb0aa15f4c42c1aed94ba5da9c18273b4e16751ef0 word_A6 word_A7
97b2728ca61a338db68c64ad13b9b51db338d27d00b56f98acc529b5ff4929e9word_B1 word_B2 word_B3 word_B4  word_B6 word_B7
d2679e2780dab58874e4a07e17d3944c6c3fe4e0edeb0b1eed767a92bc60d2ef

Last edited by Chubler_XL; 02-07-2018 at 04:36 PM..
# 13  
Old 02-08-2018
I corrected like as:
Code:
#!/bin/bash
trap 'exec 2>/dev/null; jobs -p | xargs kill;rm -f in0 in1 out0 out1 outin0 outin1' EXIT
mkfifo in0 in1 out0 out1 outin0 outin1
openssl < <(awk 'BEGIN{while(1){print "dgst -out out0 -sha256 in0\ndgst -out out1 -sha256 in1"}}') > /dev/null &
while true ;  do read T X <out0 ; echo "$X" >outin0; read T X <out1 ; echo "$X" >outin1 ; done &

while read a1 a2 a3 a4 a5 a6  ; do echo -n $a5 >in$((h%2)) ; read X <outin$((h++%2)) ; echo "$a1 $a2 $a3 $a4 $X $a6" ; done <file

But, it's less efficient:
real 0m59,877s
user 0m35,016s
sys 0m35,712s

Regards.

I think that transform line below in awk, we could win of time:
Code:
while true ;  do read T X <out0 ; echo "$X" >outin0; read T X <out1 ; echo "$X" >outin1 ; done

But, awk doesn't like to work with fifo...

Otherwise, when we can, is better to use perl or python.
Example in perl:
Code:
perl -MDigest::SHA=sha256_hex -MMIME::Base64 -a -ne '$,=" ";chomp($F[4]=sha256_hex($F[4]));print @F,"\n"' <file

Duration for 100000 iterations:
real 0m0,976s
user 0m0,932s
sys 0m0,012s

EDIT: Add my final bash - awk - openssl version most efficient and synchrone:
Code:
#!/bin/bash

trap 'exec 2>/dev/null; jobs -p | xargs kill;rm -f in0 in1 out0 out1' EXIT
mkfifo in0 in1 out0 out1

exec 7<>out0
exec 8<>out1

openssl < <(awk 'BEGIN{while(1){print "dgst -out out0 -sha256 in0\ndgst -out out1 -sha256 in1"}}') > /dev/null &

while read a1 a2 a3 a4 a5 a6  ; do echo -n $a5 >in$((h%2)) ; read T X <&$((7+(h++%2))) ; echo "$a1 $a2 $a3 $a4 $X $a6" ; done <file

duration to 100000 iterations:
real 0m38,300s
user 0m21,292s
sys 0m23,884s
Last edited by disedorgue; 02-09-2018 at 05:30 PM.. Reason: Add final version script shell more efficient
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

awk date too many open files

Dear Community; I have a csv file with msb and lsb in $3 and $5 fields which provides the epochtime (factor 65536). Further I need to convert epochtime to readable datetime. But am getting an error. File Sample: 5000a,1000,20671,0,16421,0,1,NULL,0 5000b,1000,20974,0,-16284,0,1,NULL,0... (4 Replies)
Discussion started by: mystition
4 Replies

2. Shell Programming and Scripting

awk output yields error: awk:can't open job_name (Autosys)

Good evening, Im newbie at unix specially with awk From an scheduler program called Autosys i want to extract some data reading an inputfile that comprises jobs names, then formating the output to columns for example 1. This is the inputfile: $ more MapaRep.txt ds_extra_nikira_usuarios... (18 Replies)
Discussion started by: alexcol
18 Replies

3. Shell Programming and Scripting

awk throws makes too many open files

Hi, I have a below awk script. BEGIN { FS=","; } { system("curl -v -H \"Authorization: SSWS test" -H \"Accept: application/json\" -H \"Content-Type: application/json\" -X POST \"https://tes.test.com/api/v1/users?activate=false\" -d \'{ \"profile\": { \"firstName\": \"" $1 "... (7 Replies)
Discussion started by: Krrishv
7 Replies

4. Shell Programming and Scripting

Awk: can't open error

Hi , In a directory i've the files in the following format pay:year:mon:11789604 pay:year:mon:17675644 --- and i need to get 4th part of the above file name so i used awk command in the below code #!/bin/ksh for test_data in pay* do txt_awk = awk -F':' '{print $4;}' $test_data ... (7 Replies)
Discussion started by: smile689
7 Replies

5. Shell Programming and Scripting

Awk: can't open file!!

To whom it may concern, Please help! I am trying to use awk to open all the files in a folder and remove one row and append everything into an output file. for X in `ls /Users/misssmith/Desktop/birdseed/`; do awk '(NR==546412)' /Users/misssmith/Desktop/birdseed/$X >>... (6 Replies)
Discussion started by: salpie
6 Replies

6. Shell Programming and Scripting

How to open large datafie in awk?

I just tried awk '{print}' all.plo awk: cannot open all.plo (Value too large for defined data type)awk '{print $8"-"$7"-"$6,$9,$4,$5,$12,$15}' all.plo awk: cannot open all.plo (Value too large for defined data type) datafile size is 4.8GB any other provision ? only cat works FS is... (4 Replies)
Discussion started by: Akshay Hegde
4 Replies

7. Shell Programming and Scripting

awk: can't open the file

I have a shell script with the following awk command. awk '$1 == "GEOG_SRV"{split($2,a,":");print a}' /etc/qm when I run it manually command line its fine and giving the output. bash-3.00$ awk '$1 == "GEOG_SRV"{split($2,a,":");print a}' /etc/qm 9010 But in the script its... (5 Replies)
Discussion started by: Tuxidow
5 Replies

8. Shell Programming and Scripting

awk Can't open file

Hello all, I am creating a small bash script to cycle through some files that are the results of an analysis I have run (I am a statistician). There are many folders called job001, job002,...,job135. Each folder has a text file called results.txt. I want to create two summary files of the... (3 Replies)
Discussion started by: jem43
3 Replies

9. Shell Programming and Scripting

awk: can't open

Hi Guys, I was just trying to run a bunch of test scripts which ran perfectly before i changed servers to a different solaris machine. But i just changed it to run on a different solaris version and my awk statement in the script shows an error. awk BEGIN{s=ARGV; r=ARGV; \ ... (2 Replies)
Discussion started by: wick3dsunny
2 Replies

10. Shell Programming and Scripting

awk: cannot open inputf

I'd like some help with awk too. Somewhere in my bash script i have written something like this: awk -v outputfile=$finalFile -v inputf=$file 'BEGIN { commands } { more commands } ΕND{ }' inputf where the file is a .input file with data. I also tried this: awk -v... (2 Replies)
Discussion started by: little_fairy
2 Replies
Login or Register to Ask a Question