Change value for POSIX


 
Thread Tools Search this Thread
Top Forums UNIX for Advanced & Expert Users Change value for POSIX
# 15  
Old 09-01-2017
Quote:
Originally Posted by Abhayman
Hi,

I am using Bash shell.

I have set of Ids stored in a file and I am copying them in batches in array.


Code:
# Store Distinct ContainerIds  in Array
distinct_array=`sed ':a;N;$!ba;s/\n/ /g' output/EmployeeId_distinct.txt`
declare -a arr=($distinct_array)
echo " Total Number of Distinct Ids Stored in Array ${#arr[@]}"
batchsize=200


This is how I am calling the mongo

Code:
for((i=0; i < ${#arr[@]}; i+=batchsize))
 do
   IFS=,
   part=( "${arr[@]:i:batchsize}" )
  
   sed -i  "2i permissibleCars = [  ${part[*]} ]"  query/employee_Id_count.js
   mongo localhost:27045/employee_db -u user_user -p password123 < query/employee_Id_count.js >> output/employee_Id_count.txt 
   cat query/employee_Id_count.js >> query/employee_Id_count_total.js
done


My sample employee id would be like "XYZ:16772767:586748411"

mployee_Id_count.js file :--

Code:
DBQuery.shellBatchSize = 224361901 ; 
permissibleCars = ["XYZ:16772767:58675748411" ..... more 200 ids ]
db.getCollection('employee_contracts_nrt').aggregate([
{$match:
      {         employeeClass : "ownload",
                    EmployeeId: {"$in": permissibleCars},
                                "methods.name": "image",
                "methods.status": "ACTIVE"

        } },
{"$group" : {_id:"$EmployeeId", count:{$sum:1}}}
],
{ allowDiskUse: true}
);

It works perfectly fine if I pass say 150 batchsize but if crosses 4096 bytes it starts failing.


Moderator's Comments:
Mod Comment Please use CODE tags as required by forum rules!
The entire premise behind this thread seems to be totally unrelated to the description provided. The values of the _POSIX_ARG_MAX and ARG_MAX limits affect the total byte count of the strings that can be passed to a member of the exec() family of system calls.

You said in post #8 (quoted above) that mongo is always invoked in your shell script with the command:
Code:
   mongo localhost:27045/employee_db -u user_user -p password123 < query/employee_Id_count.js >> output/employee_Id_count.txt

The arguments passed to the system to invoke mongo from this command-line is a constant (unless you change exported environment variables between calls to mongo no matter how large the files localhost:27045/employee_db, query/employee_Id_count.js, and output/employee_Id_count.txt are (or for output/employee_Id_count.txt will become as a result of running this command).

I have no idea what mongo is or what it is trying to do, but I find the argument that the value of _POSIX_ARG_MAX is keeping mongo from doing what you want it to do to indicate a gross misunderstanding of how the _POSIX_ARG_MAX limit is intended to be used.

When you say mongo starts failing with larger lists, what exactly are the diagnostic messages produced? What are the symptoms of the failure?

What is mongo trying to do? Is it trying to send a database query to a system across a network where the system running that database has less than a megabyte of RAM?
This User Gave Thanks to Don Cragun For This Post:
# 16  
Old 09-01-2017
Hi Dan,

I think you are right . I did lot more analysis in last 24 hrs and it seems the issue is completely with array . It seems due to huge size the array is failing to store it and causes a failure at mongo end.
Code:
2017-08-31T18:48:03.286+0000 E QUERY    [thread1] SyntaxError: unterminated string literal @(shell):2:4077
2017-08-31T18:48:03.317+0000 E QUERY    [thread1] SyntaxError: missing ; before statement @(shell):1:5
2017-08-31T18:48:03.348+0000 E QUERY    [thread1] SyntaxError: missing ; before statement @(shell):1:10
2017-08-31T18:48:03.379+0000 E QUERY    [thread1] SyntaxError: missing ; before statement @(shell):1:3
2017-08-31T18:48:03.409+0000 E QUERY    [thread1] SyntaxError: missing ; before statement @(shell):1:3
assert: command failed: {
        "ok" : 0,
        "errmsg" : "bad query: BadValue: $in needs an array",
        "code" : 16810
} : aggregate failed
_getErrorWithCode@src/mongo/shell/utils.js:25:13
doassert@src/mongo/shell/assert.js:16:14
assert.commandWorked@src/mongo/shell/assert.js:370:5
DBCollection.prototype.aggregate@src/mongo/shell/collection.js:1319:5
@(shell):1:1


I did try to change the ulimit -s to unlimited so that it stores the value but I feel that is also not working . I checked my virtual memory to see if that can help but couldnt figure anything concrete .

Code:
              total        used        free      shared  buff/cache   available
Mem:        3878024      347264     2409500       25108     1121260     3161284
Swap:             0           0           0

Regards.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. OS X (Apple)

POSIX compliance...

Thanks to all you guys about posix compliance I have learnt an enormous amount over the last few days. I have written a program that is an Egg Timer with simple animation. I now realise how sophisticated 'bash' is compared to full posix compliance. The code below has passed all of the tests from... (11 Replies)
Discussion started by: wisecracker
11 Replies

2. Programming

POSIX Thread Help

I want to create a program that creates 2 child process, and each of them creates 2 threads, and each thread prints its thread id. I0ve allread done that the outuput isn't the outuput i want. When a run the following comand "$./a.out | sort -u | wc -l" I have the folowing output 2 $: It should... (3 Replies)
Discussion started by: pharaoh
3 Replies

3. UNIX for Advanced & Expert Users

System V or POSIX

Hi , I am using UNIX network programming Vol1 (by R Stevens) book to learn about IPC. I would be using HP-UX,Solaris and Linux at my work. I have sections for POSIX and for System V in that book. I am quite confused in indentifying those OSs as POSIX or SYstem V. Can anyone please... (1 Reply)
Discussion started by: kumaran_5555
1 Replies

4. UNIX for Advanced & Expert Users

Posix threads

Hi, consider the code below: #include <stdio.h> . . struct myStruct { char *message ; int id; }; . . . void *thread_function( void *ptr ); nt main() { pthread_t thread1, thread2 ,thread3 ; struct myStruct nico1; (2 Replies)
Discussion started by: Behnaz
2 Replies

5. Programming

Posix

HI, When i am configuring php in SUN Solaris. I am getting the below error. configure: error: Your system seems to lack POSIX threads. Do i need to install POSIX? If so can somebody let me know where can i download POSIX for Solaris 8? Thanks, (2 Replies)
Discussion started by: Krrishv
2 Replies

6. Programming

POSIX threads

Hello ! Let's supose I have a main function in C , and two POSIX threads. I give you an example down : int main() { int something; char else; void *FirstThread(); void *SecondThread(); .. <start those two pthreads ..> return 0;} void *FirstThread() { ... } void *SecondThread()... (2 Replies)
Discussion started by: !_30
2 Replies

7. UNIX for Dummies Questions & Answers

how to read POSIX?

how to read POSIX? poe six or not? (3 Replies)
Discussion started by: robin.zhu
3 Replies

8. Programming

Unix(posix)

Please,does anybody can give me any general info about unix(posix) ? (1 Reply)
Discussion started by: Haris Astreos
1 Replies

9. Programming

ANSI C vs POSIX

can somebody explain about the ANSI C vs POSIX. say i was using open and fopen, i know that open is POSIX, and fopen is ANSI C. i read that that POSIX is a system call and ANSI C is like a standard library function. wouldn't the fopen function has to call on open function anyway to open any kind... (2 Replies)
Discussion started by: bb00y
2 Replies

10. UNIX for Dummies Questions & Answers

Posix and linux

What is posix? What is the relation between Posix, Unix and linux? (1 Reply)
Discussion started by: darbarilal
1 Replies
Login or Register to Ask a Question