Odd behavior from GDB while trying to cross-debug an embedded Linux application.


 
Thread Tools Search this Thread
Top Forums Programming Odd behavior from GDB while trying to cross-debug an embedded Linux application.
# 1  
Old 02-28-2019
Odd behavior from GDB while trying to cross-debug an embedded Linux application.

Some background:

The application normally runs on an embedded platform. Currently, for development purposes, I have the rootfs located @ /exports and the target is communicating over NFS. That way I can make a change on my local system, save the application @ /exports, and run the altered version to see if it will work without having to do a full install on the target itself.

Following this guide I was able to get GDB debugging working or at least I thought I was. However, as soon as I altered my code, built my project and copied/replaced the application with the altered application I received the following errors:

Truncated/partial Eclipse Debugger Console output:

Code:
For help, type "help".
Type "apropos word" to search for commands related to "word".
(gdb) 0x400007b0 in ?? () from target:/lib/ld-linux.so.3                                                                                                                                           
Reading /usr/lib/libasound.so.2 from remote target...
...
Reading /usr/lib/libgcc_s.so.1 from remote target...
Error while mapping shared library sections:
Remote communication error.  Target disconnected.: Connection reset by peer.
warning: Could not load shared library symbols for 4 libraries, e.g. /usr/lib/libc.so.6.
Use the "info sharedlibrary" command to see the complete listing.
Do you need "set solib-search-path" or "set sysroot"?
Error while reading shared library symbols for target:/usr/lib/libasound.so.2:
Can't read symbols from target:/usr/lib/libasound.so.2: Input/output error
...
Error while reading shared library symbols for target:/usr/lib/libm.so.6:
Can't read symbols from target:/usr/lib/libm.so.6: Input/output error

Output from my serial port terminal:

Code:
*** glibc detected *** gdbserver: double free or corruption (!prev):
0x000983d8 ****

As I mentioned I believe this gdbserver error may be erroneous and explaining why might be tricky so here goes. As I mentioned GDB seemed to be working. For instance, here is the GDB output I was getting:

Truncated/partial Eclipse Debugger Console output (before compiling and copying application):

Code:
For help, type "help".
Type "apropos word" to search for commands related to "word".
(gdb) 0x400007b0 in ?? () from target:/lib/ld-linux.so.3                                                                                                                                           
Cannot access memory at address 0x7ffd8614
...
Cannot access memory at address 0x7ffd8610

Temporary breakpoint 1, main (argc=0, argv=0x0) at main.cpp:413
413       TimeStuff::CStopWatch  dtorDuration;
Cannot access memory at address 0x7ffd8614
...
Cannot access memory at address 0x7ffd8610

So, like expected, the application runs and pauses at main. I can then resume the application and debug whatever I need. In fact, I used it to debug a problem then made a change and tested it. It worked, great! However, as soon as I went to debug the application for a different problem I received the errors you read earlier. So why do I think the gdbserver error is erroneous? Well because both applications are using the same exact version of glibc. In fact, the only differences between the two applications is the changes I made to the code and that code works, I know because I tested it.

That being said I am chooseing, XD, to ignore the error reported in the serial terminal and am concentrating on the errors reported by Eclipse, specifically this jumped out at me:

Code:
Remote communication error.  Target disconnected.: Connection reset by peer.

Not sure why there would be a remote communications error, any ideas what might be happening here? Are these glibc errors actually red herrings?

Last edited by Circuits; 02-28-2019 at 02:24 PM.. Reason: Attempting to make more sense XD
# 2  
Old 02-28-2019
I really doubt the crash is erroneous.

Quote:
So why do I think the gdbserver error is erroneous? Well because both applications are using the same exact version of glibc.
Crashes inside libc are almost always programmer-caused. This one is easy to cause:

Code:
#include <stdlib.h>

int main(void) {
        void *p=malloc(1000);
        free(p);
        free(p); // crash
}

Quote:
In fact, the only differences between the two applications is the changes I made to the code and that code works, I know because I tested it.
Never call code good because it "just works", especially when it crashes. What exact changes did you make to what exact code?

It could be the debugger that's crashing, I suppose. Unlikely but plausible.
# 3  
Old 03-01-2019
Unfortunately I got pulled off into a bug and had to put this aside for a few days but now I have some more time.



@Corona688


Just to be sure I remounted a brand new copy of the rootfs from a branch and then downloaded the Codebase for that branch and repeated the above exercise and got the same behavior. Another thing I find odd, in the debugger console at the end of the output I showed above it says:


Code:
A problem internal to GDB has been detected,
further debugging may prove unreliable.
 Quit this debugging session? (y or n)

but it's actually frozen. Entering y or n does nothing. So I have to manual terminate the debugging session.




I tried running gdb-multiarch from a terminal instead of using it through Eclipse and ended up with similar output except right at the end it says:

Code:
Segmentation fault (core dumped)

So, this doesn't seem to be an Eclipse problem. Maybe it is a gdb-multiarch problem?


EDIT: I take that back the output is jumping all over I tried again running gdb-multiarch with sudo and ended up with the same output as before, no mention of a seg fault.

Last edited by Circuits; 03-01-2019 at 06:15 PM..
# 4  
Old 03-01-2019
Quote:
Originally Posted by Corona688
What exact changes did you make to what exact code?
What exact changes did you make to the code?
# 5  
Old 03-04-2019
@Corona688 I reproduced the error without making any changes to the code to be sure that the changes I was making weren't effecting anything. I now believe the error to be a communications error with gdb-server on the embedded side. Gdb-server was installed onto the embedded target something like 5 or 6 years ago while gdb-multiarch is brand new.
This User Gave Thanks to Circuits For This Post:
Login or Register to Ask a Question

Previous Thread | Next Thread

8 More Discussions You Might Find Interesting

1. Cybersecurity

Odd behavior from passwd.

Hi Folks, Before I go off and start checking I'm just wondering if anyone has seen this behaviour before. # passwd e825390 Changing password for user e825390. New password: Retype new password: Retype new password: passwd: all authentication tokens updated successfully. As you can... (2 Replies)
Discussion started by: gull04
2 Replies

2. Programming

Can't debug: assert error with gdb (no problem without)

I'm sorry if the title is really criptic, but I don't know how to phrase my problem. I know I can't really ask for a solution, and I normally wouldn't but this is really escaping my abilities. Antefacts. I developed a program using the zeromq messaging library. I got to a point where the... (11 Replies)
Discussion started by: erupter
11 Replies

3. Programming

Debug two process Using GDB

Hi All I know How to attach a process to beubg it .But for my application I am using client as well server.Both are two separate process .Suppose I need to debug both .How to attach both of them together .Or I have to attach them separetly . Suppose client process id is 1325 and server is... (2 Replies)
Discussion started by: mr_deb
2 Replies

4. UNIX for Advanced & Expert Users

a question about debug using gdb(/lib32/libc.so.6: No such file or directory)

Hi all, (gdb) target remote 192.168.185.196:1389 Remote debugging using 192.168.185.196:1389 warning: Unable to find dynamic linker breakpoint function. GDB will be unable to debug shared library initializers and track explicitly loaded dynamic code. 0x2fbd0800 in ?? () (gdb) l 1 ... (0 Replies)
Discussion started by: yanglei_fage
0 Replies

5. Programming

gdb: problem while debug an executable file

I created one file (test.cpp)and complied it and get the executable file(test) and run it.It is giving the out put but when I am trying to debug that executable.I am getting the error -- please tell me the solutions?? Thanks in advance ..... (2 Replies)
Discussion started by: smartgupta
2 Replies

6. Programming

gdb: how to debug an executable file with a few command line papameters

If an executalbe file has several parameters, gdb can not pass parameters correctly. Let us see: run: ./executablefile1 agr1 arg2 arg3 debug: gdb executablefile1 run executalbefile1 arg1 arg2 arg3 then argv : executablefile1 argv : executablefile argv : arg1 ... (3 Replies)
Discussion started by: cdbug
3 Replies

7. HP-UX

Odd storage behavior

Hi, We have some troubles with our HP server (rx4640) running HP-UX 11.31. The server is attached to a JBod cabinet. If the JBod cabinet is powered on and we power on the server after then HP-UX can't find the devices (disks) at the cabinet. Does not help to run an ioscan -fnC disk. But if I power... (3 Replies)
Discussion started by: hoff
3 Replies

8. UNIX for Dummies Questions & Answers

Odd .sh behavior in script

Hello, I have been working on a what I thought was a fairly simple script for installing a software kit on Linux and Unix I am not new to scripting but am far from being fluent in sh scripting. any assistance would be appreciated. I have an odd bug occuring when executing the script. When... (2 Replies)
Discussion started by: robertmcol
2 Replies
Login or Register to Ask a Question