Continuous nc data acquisition fails ocassionally


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Continuous nc data acquisition fails ocassionally
# 1  
Old 05-23-2018
Continuous nc data acquisition fails ocassionally

Hi,

I'm receiving text data on a server trough UDP packets.

The data are encoded messages separated by blank lines, like this:

Code:
!AIVDM,1,1,,A,13FPE?PP08OG@cPH:Gi8OwwB0<0h,0*06

!AIVDM,1,1,,A,13Ebj60000wH2E:H:mKIF2GB2<17,0*49

!AIVDM,1,1,,B,4028nAAv9Kh0awESU0HRIP102D3f,0*5D

I'm using the following sentence to adquire that data| add a timestamp and remove the blank line| separate in 1MegaByte files:

Code:
nc -ul -p 56045 -q -1 -vv | awk -F: 'NF > 1 {print strftime("%d-%m-%Y %H:%M:%S"),$0  }'| split -d -a3 -C 1M --additional-suffix=`date +_AIVDM_%F_%H%M%S`.txt - aivdm/ &

It seems to work but after a period of 5-8 hours the splited text files stop to increase their size and content, and I can't get more data till I kill the processes. When the adquisitions stops I get no error on screen and the processes appear if I call ps.

Any idea of what may be happening?

All the best
# 2  
Old 05-23-2018
I wonder if it's network-related. Are the sender and receiver on the same subnet?
This User Gave Thanks to Corona688 For This Post:
# 3  
Old 05-23-2018
No, they are not. But if the sender ocassionally stops sending packets (this can happens), nc with -q -1 should listen and wait forever isn't?
# 4  
Old 05-23-2018
It would be interesting to see where the data is being lost.

Why don't you throw some tee commands in those pipelines and when it happens again you will know if it's nc awk or split that's losing the data:

Code:
nc -ul -p 56045 -q -1 -vv | tee /tmp/pre-awk.dat | awk -F: 'NF > 1 {print strftime("%d-%m-%Y %H:%M:%S"),$0  }' | tee /tmp/pre-split.dat | split -d -a3 -C 1M --additional-suffix=`date +_AIVDM_%F_%H%M%S`.txt - aivdm/ &

# 5  
Old 05-24-2018
Are you using GNU or BSD netcat ?

With BSD netcat -k option will listen forever, with GNU i do not think it works.

Both should be available on linux distributions
On debian :
Code:
~$ apt-cache search netcat-
netcat-traditional - TCP/IP swiss army knife
netcat-openbsd - TCP/IP swiss army knife # this is the one you want with -k option.

Regards
Peasant.
This User Gave Thanks to Peasant For This Post:
# 6  
Old 05-24-2018
Thank Chubler_XL i will try and tell you.
Yesterday i tried a simpler:
Code:
nc -ul -p 56045 -q -1 -vv >data.txt

and there was no problem, so it seems that it has to be the awk or split sentence.

Thanks Peasant, I'm using GNU but I guess that the nc -q -1 option should work and wait forever and the tried that I did yesterday seems to confirm this. At this moment I guess that the problem is in the awk or split sentence

All the best
# 7  
Old 05-24-2018
I don't think you've said what operating system or filesystem type this is. Is there a possibility that you're hitting a maximum file size for the filesystem? When it stops writing to the file, is the file near or about the same size each time?
Login or Register to Ask a Question

Previous Thread | Next Thread

5 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Continuous for loop

Hi All, Please help ***************** a=100 and run for loop that will minus 30 from 100 an will display value run loop will run till display value will be 0 ***************** Thanking you (1 Reply)
Discussion started by: Praful Pednekar
1 Replies

2. Shell Programming and Scripting

Continuous Copying from a directory to another.

Hello Folks, Looking for a script, where i can check for the existing of the files in a directory and copying to another. How can i achieve it in loop for over period of time. whatever files comes into the folder copied in another without duplicate and should be continuous loop. ... (8 Replies)
Discussion started by: sadique.manzar
8 Replies

3. UNIX for Dummies Questions & Answers

Extracting data between continuous non empty xml tags

Hi, I need help in extracting only the phone numbers between the continuous non empty xml tags in unix. I searched through a lot of forum but i did not get exact result for my query. Please help Given below is the sample pipe delimited file. I have a lot of tags before and after... (6 Replies)
Discussion started by: zen01234
6 Replies

4. Shell Programming and Scripting

Continuous checking of a file

I have a requirement like this... I want to go to a particular server for which i have acess .I want to do a ssh to that server from one server and check if a file is theer or not..and i need the script to chcek continuosly till it finds the file.When it finds the file i want it to come out... (9 Replies)
Discussion started by: kanta_bhakti
9 Replies

5. UNIX for Advanced & Expert Users

Need info regarding CDP - continuous data protection

HI frnds, i am going to work on CDP(continuous data protection),does any one of them have any idea reg CDP ... any docs or Links...might be helpful to me in understanding it.... (1 Reply)
Discussion started by: deep
1 Replies
Login or Register to Ask a Question