Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

awk (Too many open files)

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting awk (Too many open files)
# 1  
Old 01-19-2018
awk (Too many open files)
Hi Team,

This is my first post, hope I am doing it right.
I have a large file, like 6 GB. Its a proxy file so vendor requested to change username from logs for saving the confidentiality of the user.

This is the script I created (With the help of Google):

Code:
awk '{
    tmp="echo " $5 " | sha256sum | cut -f5 "
tmp | getline cksum
$5=cksum
print
}' < file.txt

So the solution will be to encrypt the username (which is the 5th word per each line), that way vendor will keep track of user navigation but will not know who they are.

Problem:
This is the error I am getting:

awk: cmd. line:2: (FILENAME=- FNR=5433) fatal: cannot open pipe `echo ZS317AP | sha256sum | cut -f5 ' (Too many open files)

Looks like I need to close the file somewhere, butI donīt know where.
Could you please help me to prevent that error.

Thank you very much!
# 2  
Old 01-19-2018
Code:
awk '{
    tmp="echo " $5 " | sha256sum | cut -f5 "
tmp | getline cksum
$5=cksum
print
close (tmp)
}' < file.txt

Moderator's Comments:
Mod Comment Please use CODE tags as required by forum rules!
Last edited by RudiC; 01-19-2018 at 07:49 PM.. Reason: Added CODE tags.
This User Gave Thanks to rdrtx1 For This Post:
tealc
# 3  
Old 01-19-2018
Hi,
sha256sum is slower that openssl if it use correctly, demo:
Three approach to compute 1000 hash and duration:
Your approach:
Code:
$ time for i in {00001..01000}; do echo $i| sha256sum ; done | cut -d " " -f1 | tail -1
e621b00e04e35902627e74db08413fec5e307d459866a67a3a0b1a6fcaed05fd

real	0m2,762s
user	0m0,220s
sys	0m0,744s

openssl standard approach:
Code:
$ time for i in {00001..01000}; do echo $i| openssl dgst -sha256 ; done | cut -d " " -f2 | tail -1
e621b00e04e35902627e74db08413fec5e307d459866a67a3a0b1a6fcaed05fd

real	0m7,910s
user	0m4,188s
sys	0m1,188s

openssl other approach (and better) :
Code:
$ mkfifo in0 in1
$ time openssl < <(
 j=1
 while read mdp
 do
  x=$(( j++ % 2 ))
  echo "dgst -sha256 in$x"
  echo "$mdp" >in$x
done < <(for i in {00001..01000}; do echo $i;done)
) | cut -d " " -f3 | tail -2
e621b00e04e35902627e74db08413fec5e307d459866a67a3a0b1a6fcaed05fd


real	0m0,181s
user	0m0,096s
sys	0m0,044s
$ rm in0 in1

This User Gave Thanks to disedorgue For This Post:
tealc
# 4  
Old 01-22-2018
Thank you @rdrtx1 , tested and worked!

@disedorgu Thanks for showing me that approach, but how can I use it? I need the 5th word for every line in a file to be changed. I Tried using your code, removing the:

Code:
(for i in {00001..01000}; do echo $i;done)

But didnīt worked.

Thank you All for your great help!
# 5  
Old 01-22-2018
In your case, you must create 2 functions and 5 fifo for work fine:
First function:
Code:
func_openssl ()
{
    openssl < <(
 j=1
 while read mdp
 do
  x=$(( j++ % 2 ))
  echo "dgst -sha256 -out out$x in$x"
  echo "$mdp" >in$x
done <std0
)
}

second function:
Code:
func_mngopenssl ()
{
    read a1 b1 c1 d1 e1 f1;
    echo "$e1" > std0;
    o=1;
    while read a2 b2 c2 d2 e2 f2; do
        x=$(( o++ %2 ));
        echo "$e2" > std0;
        e3=$(awk '{print $NF}' out$x);
        echo "$a1;$b1;$c1;$d1;$e3;$f1";
        a1="$a2";
        b1="$b2";
        c1="$c2";
        d1="$d2";
        e1="$e2";
        f1="$f2";
    done;
    x=$(( o++ %2 ));
    e3=$(awk '{print $NF}' out$x);
    echo "$a1;$b1;$c1;$d1;$e3;$f1"
}

create fifo std0 in0 in1 out0 out1 in your curent directory:
Code:
mkfifo std0 in0 in1 out0 out1

And you can use as:
Code:
func_openssl >/dev/null &  func_mngopenssl <fichier

You can change your word separator as (here, I choose ';' as separator) :
Code:
func_openssl >/dev/null &  IFS=\; func_mngopenssl <fichier

File example:
Code:
$ cat /tmp/words.csv
word_01;word_02;word_03;word_04;word_05;word_06;word_07
word_11;word_12;word_13;word_14;word_15;word_16;word_17
word_01;word_02;word_03;word_04;word_05;word_06;word_07
word_11;word_12;word_13;word_14;word_15;word_16;word_17
word_01;word_02;word_03;word_04;word_05;word_06;word_07
word_11;word_12;word_13;word_14;word_15;word_16;word_17
word_01;word_02;word_03;word_04;word_05;word_06;word_07
word_11;word_12;word_13;word_14;word_15;word_16;word_17
word_01;word_02;word_03;word_04;word_05;word_06;word_07
word_11;word_12;word_13;word_14;word_15;word_16;word_17
word_01;word_02;word_03;word_04;word_05;word_06;word_07
word_11;word_12;word_13;word_14;word_15;word_16;word_17

execution:
Code:
$ func_openssl >/dev/null & IFS=\; func_mngopenssl </tmp/words.csv
[1] 31328
word_01;word_02;word_03;word_04;f0a73d8cb2a8fb7b724132be9853f9cf72ac00ca42abeaf7bce1bbcfd299d594;word_06;word_07
word_11;word_12;word_13;word_14;467cb4d2ed408093ce85766df9836a759e7f3dcd710308f1a2bba10fe0b921c6;word_16;word_17
word_01;word_02;word_03;word_04;f0a73d8cb2a8fb7b724132be9853f9cf72ac00ca42abeaf7bce1bbcfd299d594;word_06;word_07
word_11;word_12;word_13;word_14;467cb4d2ed408093ce85766df9836a759e7f3dcd710308f1a2bba10fe0b921c6;word_16;word_17
word_01;word_02;word_03;word_04;f0a73d8cb2a8fb7b724132be9853f9cf72ac00ca42abeaf7bce1bbcfd299d594;word_06;word_07
word_11;word_12;word_13;word_14;467cb4d2ed408093ce85766df9836a759e7f3dcd710308f1a2bba10fe0b921c6;word_16;word_17
word_01;word_02;word_03;word_04;f0a73d8cb2a8fb7b724132be9853f9cf72ac00ca42abeaf7bce1bbcfd299d594;word_06;word_07
word_11;word_12;word_13;word_14;467cb4d2ed408093ce85766df9836a759e7f3dcd710308f1a2bba10fe0b921c6;word_16;word_17
word_01;word_02;word_03;word_04;f0a73d8cb2a8fb7b724132be9853f9cf72ac00ca42abeaf7bce1bbcfd299d594;word_06;word_07
word_11;word_12;word_13;word_14;467cb4d2ed408093ce85766df9836a759e7f3dcd710308f1a2bba10fe0b921c6;word_16;word_17
word_01;word_02;word_03;word_04;f0a73d8cb2a8fb7b724132be9853f9cf72ac00ca42abeaf7bce1bbcfd299d594;word_06;word_07
word_11;word_12;word_13;word_14;467cb4d2ed408093ce85766df9836a759e7f3dcd710308f1a2bba10fe0b921c6;word_16;word_17
[1]+  Done                    func_openssl > /dev/null

Here, output separator is forced to ';' , but you can modified code in function.
This User Gave Thanks to disedorgue For This Post:
tealc
# 6  
Old 01-23-2018
Thank you @disedorgue! Your help is much apreciated. This is helping me a lot.

One more thing, the output looks like this:

Code:
2018-01-16;22:17:21;1396;10.23.13.215;OPENSSLENCRYPED S-U-Proxy-DefaultAccessAR

But I need it to look without the ";" , since the initial separator is space " "
Like this:

Code:
2018-01-16 22:17:21 1396 10.23.13.215 OPENSSLENCRYPED S-U-Proxy-DefaultAccessAR

Thank you Kind sir!

Moderator's Comments:
Mod Comment Please use CODE tags (for data as well) as required by forum rules!
Last edited by RudiC; 01-23-2018 at 10:00 AM.. Reason: Added CODE tags.
# 7  
Old 01-23-2018
just change like this (in red) :
Code:
func_mngopenssl ()
{
    read a1 b1 c1 d1 e1 f1;
    echo "$e1" > std0;
    o=1;
    while read a2 b2 c2 d2 e2 f2; do
        x=$(( o++ %2 ));
        echo "$e2" > std0;
        e3=$(awk '{print $NF}' out$x);
        echo "$a1 $b1 $c1 $d1 $e3 $f1";
        a1="$a2";
        b1="$b2";
        c1="$c2";
        d1="$d2";
        e1="$e2";
        f1="$f2";
    done;
    x=$(( o++ %2 ));
    e3=$(awk '{print $NF}' out$x);
    echo "$a1 $b1 $c1 $d1 $e3 $f1"
}

And launch like as (first exemple in my last post) :
Code:
func_openssl >/dev/null &  func_mngopenssl <fichier

Regards.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

awk date too many open files

Dear Community; I have a csv file with msb and lsb in $3 and $5 fields which provides the epochtime (factor 65536). Further I need to convert epochtime to readable datetime. But am getting an error. File Sample: 5000a,1000,20671,0,16421,0,1,NULL,0 5000b,1000,20974,0,-16284,0,1,NULL,0... (4 Replies)
Discussion started by: mystition
4 Replies

2. Shell Programming and Scripting

awk output yields error: awk:can't open job_name (Autosys)

Good evening, Im newbie at unix specially with awk From an scheduler program called Autosys i want to extract some data reading an inputfile that comprises jobs names, then formating the output to columns for example 1. This is the inputfile: $ more MapaRep.txt ds_extra_nikira_usuarios... (18 Replies)
Discussion started by: alexcol
18 Replies

3. Shell Programming and Scripting

awk throws makes too many open files

Hi, I have a below awk script. BEGIN { FS=","; } { system("curl -v -H \"Authorization: SSWS test" -H \"Accept: application/json\" -H \"Content-Type: application/json\" -X POST \"https://tes.test.com/api/v1/users?activate=false\" -d \'{ \"profile\": { \"firstName\": \"" $1 "... (7 Replies)
Discussion started by: Krrishv
7 Replies

4. Shell Programming and Scripting

Awk: can't open error

Hi , In a directory i've the files in the following format pay:year:mon:11789604 pay:year:mon:17675644 --- and i need to get 4th part of the above file name so i used awk command in the below code #!/bin/ksh for test_data in pay* do txt_awk = awk -F':' '{print $4;}' $test_data ... (7 Replies)
Discussion started by: smile689
7 Replies

5. Shell Programming and Scripting

Awk: can't open file!!

To whom it may concern, Please help! I am trying to use awk to open all the files in a folder and remove one row and append everything into an output file. for X in `ls /Users/misssmith/Desktop/birdseed/`; do awk '(NR==546412)' /Users/misssmith/Desktop/birdseed/$X >>... (6 Replies)
Discussion started by: salpie
6 Replies

6. Shell Programming and Scripting

How to open large datafie in awk?

I just tried awk '{print}' all.plo awk: cannot open all.plo (Value too large for defined data type)awk '{print $8"-"$7"-"$6,$9,$4,$5,$12,$15}' all.plo awk: cannot open all.plo (Value too large for defined data type) datafile size is 4.8GB any other provision ? only cat works FS is... (4 Replies)
Discussion started by: Akshay Hegde
4 Replies

7. Shell Programming and Scripting

awk: can't open the file

I have a shell script with the following awk command. awk '$1 == "GEOG_SRV"{split($2,a,":");print a}' /etc/qm when I run it manually command line its fine and giving the output. bash-3.00$ awk '$1 == "GEOG_SRV"{split($2,a,":");print a}' /etc/qm 9010 But in the script its... (5 Replies)
Discussion started by: Tuxidow
5 Replies

8. Shell Programming and Scripting

awk Can't open file

Hello all, I am creating a small bash script to cycle through some files that are the results of an analysis I have run (I am a statistician). There are many folders called job001, job002,...,job135. Each folder has a text file called results.txt. I want to create two summary files of the... (3 Replies)
Discussion started by: jem43
3 Replies

9. Shell Programming and Scripting

awk: can't open

Hi Guys, I was just trying to run a bunch of test scripts which ran perfectly before i changed servers to a different solaris machine. But i just changed it to run on a different solaris version and my awk statement in the script shows an error. awk BEGIN{s=ARGV; r=ARGV; \ ... (2 Replies)
Discussion started by: wick3dsunny
2 Replies

10. Shell Programming and Scripting

awk: cannot open inputf

I'd like some help with awk too. Somewhere in my bash script i have written something like this: awk -v outputfile=$finalFile -v inputf=$file 'BEGIN { commands } { more commands } ΕND{ }' inputf where the file is a .input file with data. I also tried this: awk -v... (2 Replies)
Discussion started by: little_fairy
2 Replies
Login or Register to Ask a Question