Shell Programming and Scripting

Another possible pitfall: It may be waiting in buffers between programs, rather than truly dead.

After testing chubler_XL suggestion:

Code:

nc -ul -p 56045 -q -1 -vv | tee /tmp/pre-awk.dat | awk -F: 'NF > 1 {print strftime("%d-%m-%Y %H:%M:%S"),$0  }' | tee /tmp/pre-split.dat | split -d -a3 -C 1M --additional-suffix=`date +_AIVDM_%F_%H%M%S`.txt - aivdm/ &

I can say that it seems that the split command is stopping the whole sentence because there some data that arrives to pre-awk.data and not to pre-split.data

hicksd8 I'm working on GNU/Linux 9.4 and the file stream could be infinite, I'm dividing it with split in 1M files and curiously the process always ends between 3-4 files (3.5-4M). I dont think its a problem with the file size but maybe a buffer size or buffer waiting as Corona668 is pointing.

Any clues on how to trace this? I'm not having any error message

It can be a problem of the ssh session?

I mean, if I do something like

AISdecode.sh

Code:

#!/bin/sh
#CONTINUOUS AIS DECODE TO PERIODICALLY JSON FILES
nc -ul -p 56045 -q -1 | awk -F: 'NF > 1 {print strftime("%d-%m-%Y %H:%M:%S"),$0;fflush();  }' > awk.txt 2>&1 &

and then execute ./AISdecode.sh &
should I have any problem when the ssh session closes?

Note that in this case I've added and fflush(); to the awk sentence, eliminated the split sentence and reidirect stderror and stdout to a file just to tests by parts and see if I can find the problem. I'm my last try without 2>&1 & it stopped with a file of 3.8 M, now I'm trying this, lets see.

Any more ideas? This should be working...

To avoid issues with session timeout use screen


Is it a real pain to try out BSD nc variant to check if you are having same symptoms ?

To really detected what is happening one would need to tcpdump (or equivalent) and examine the generated files after, looking for suspected behavior.
I see no other way of telling what is going on, if network level is the problem of some sort.

When did the problem surfaced ?
Did it stop working in some point after working for some time or ?

Hope that helps
Regards
Peasant.

Would it make sense to start from the other side and flood the network and reception pipe from the sender side, by e.g. catting over MBytes and MBytes?

Thanks Peasant, I will check screen

What do you mean by BSD nc variant?
The problem is a new issue, I mean is a new problem, I recently obtained this data stream and I want to save it continuously but at the moment I can't get a satisfactory result, from time to time the pipe stop receiving data and saving to the file.

I guess that from time to time some packect get lost or the transmission temporarely stops or the pipe buffer if filled so even when the data still being received the data storing get stuck.

I will keep trying

The problem with buffers, if that is the problem, would be the buffer NOT filling, storing up data and waiting until its full.