How to make use others' C library installed not for the system-wide (Ubuntu/Linux)?


 
Thread Tools Search this Thread
Top Forums Programming How to make use others' C library installed not for the system-wide (Ubuntu/Linux)?
# 1  
Old 01-14-2020
How to make use others' C library installed not for the system-wide (Ubuntu/Linux)?

I have downloaded and installed a library called htslib for specific bioinformatic use but not for the system (I'm using Ubuntu 18.04). Only parts of the library is needed for my exercise to parse data in a type called VCF format (basically tab-delimited file but contains many information in columns).
  1. My test code (vcf_parser01.c) is attached to make use the VCF API from the htslib, for which the vcf part seems to be what I need and the vcf.h/vcf.c files are also attached here. Maybe the whole htslib is needed, that's one of the parts I need to confirm.
  2. The htslib was compiled and installed locally without error through Makefile with the package, and the structure of the resulted folder is like:

Code:
.
├── aclocal.m4
├── bgzf.c
├── bgzf.o
├── bgzip.c
├── bgzip.o
├── config.h
......
|── cram
│   ├── cram_codecs.c
│   ├── cram_codecs.h
│   ├── cram_codecs.o
......
├── header.c
├── header.h
├── header.o
......
├── hts.c
├── htsfile.c
├── htsfile.o
├── hts_internal.h
├── htslib
│   ├── faidx.h
│   ├── hfile.h
│   ├── hts_defs.h
│   ├── hts_endian.h
│   ├── hts.h
......

├── kstring.c
├── kstring.o
├── vcf.c
├── vcf.o
......

3) To compile my test code, I used:
Code:
gcc -Wall -O3 -I ./htslib-1.10.2/htslib  -o vcf_parser01 vcf.c  vcf_parser01.c

4) There are many errors believed to the linker related such as:
Code:
vcf_parser01.c:(.text.startup+0x19): undefined reference to `hts_open'
vcf_parser01.c:(.text.startup+0x73): undefined reference to `hts_close'
/tmp/cc05JReL.o: In function `bcf_hdr_add_sample_len':
vcf.c:(.text+0x989): undefined reference to `hts_log'
/tmp/cc05JReL.o: In function `bcf_hdr_set_idx.isra.10':
vcf.c:(.text+0x116b): undefined reference to `hts_log'
vcf.c:(.text+0x11d5): undefined reference to `hts_resize_array_'
/tmp/cc05JReL.o: In function `bcf_subset_format.part.19':
vcf.c:(.text+0x18e5): undefined reference to `hts_realloc_or_die'
/tmp/cc05JReL.o: In function `bcf_hdr_format.constprop.30':
vcf.c:(.text+0x1e35): undefined reference to `ksprintf'
vcf.c:(.text+0x1e73): undefined reference to `ksprintf'
vcf.c:(.text+0x1ed3): undefined reference to `ksprintf'

There were totally 196 "undefined reference to" lines for 39 functions, which seems involving other libraries!
My question is:
How to debug this linker problem?

I think I understand the .o .so .a files, and the -I/-L options for gcc but not in full good catch.
A bigger picture in my mind is to learn how to make use of any non-standard C libraries from others, especially when documentation of the API is not very clear. I narrow down the topic in ANSI/GNU C in Linux platform only.
I may need a full course on this, but I know there are C experts in the forum. Really appreciate anybody could help me out.

Last edited by yifangt; 01-14-2020 at 02:50 PM..
# 2  
Old 01-15-2020
First, I suggest when troubleshooting, you should:

Use the full path name to file, not relative path name. This will insure there are no strange, unseen PATH issues. For example:

Code:
gcc -Wall -O3 -I ./htslib-1.10.2/htslib  -o vcf_parser01 vcf.c  vcf_parser01.c

I would change this to:

Code:
gcc -Wall -O3 -I /FULL/PATH/TO/HERE/htslib-1.10.2/htslib  -o vcf_parser01 vcf.c  vcf_parser01.c

Second, you should confirm that these objects are in the PATH, exactly, and that you have read / access permission for them.

Sometimes, even the best sys admins install code under one userid, then they work as another userid, and they do not have permissions to access the file. Happens all the time (at least to me, LOL).

This is generally the first two steps I always take (back to basics, before back to the future).
  • Insure your PATH(s) are correct and objects / symbols are in the PATH(s), correctly.
  • Check file and directly permissions for the userid you are using to build.

UNIX and Linux are funny things, they generally do what they are told to do and report back the "facts" as they find them. In your case, gcc cannot find required symbols and objects. This is generally because they cannot find them, i.e. not in the search PATH or the file/directory permissions are "not as required".

Please post back and let me know if you are certain your PATHs are correct and why, using the FULL PATH names to files and directories in your command line for gcc.

Since we are not "standing behind you, watching you type", we cannot "see" what directory you are in, so to be sure, it is always best to use FULL PATH names when troubleshooting a problem like this.

Thanks.
This User Gave Thanks to Neo For This Post:
# 3  
Old 01-15-2020
Undefined references means the compiler can't find compiled code for functions being called. If you want to use a subset of the library then you'll have to compile and link all the files used by the code you do want.
In otherwords - if you want to use method a in file a.c and it uses method b in b.c and it uses method c in c.c then you'll need to compile and link a.c, b.c and c,c. I you have an archive file you can link directly to it and the library won't be dynamically loaded at runtime. if you only have a shared library then you, if you are on a Linux system you can investigate rpath as a means to tell the linker where the library is to be found.


-Greg.
This User Gave Thanks to GRMartin For This Post:
# 4  
Old 01-15-2020
Thanks Neo and Greg!
Code:
$ pwd
/home/yifangt/Study/C/VCF
$ ls ${PWD}
config.h  for_post  hts_internal.h  htslib  htslib-1.10.2  htslib-1.10.2.tar.bz2  README  textutils_internal.h  tmp.err  vcf.c  vcf.o  vcf_parser01.c
$ gcc -Wall -O3 -I /home/yifangt/Study/C/VCF/htslib-1.10.2  -o vcf_parser01 vcf.c vcf_parser01.c  2>err.txt

The htslib-1.10.2.tar.bz2 is what I downloaded from the website, and the library was compiled from the pkg according to their Makefile.
By the way I tried -L option instead of -I for the gcc command line gave the same error.
The complete actual log file is attached here too for your reference.
Thank you very much again!
# 5  
Old 01-15-2020
You can compile and link using compiler options. The library is referenced using -l<its_name_here> and the path to the library using -L/path/to/lib.


-I is for the path to include directories.


However, if the library isn't in a directory in the linkers path when you run the executable you'll get an error. One way to solve this is to use rpath option.


someting like:

Code:
gcc -Wall -O3 -o vcf_parser01 vcf.c vcf_parser01.c -Wl,-rpath,/home/yifangt/Study/C/VCF -L/home/yifangt/Study/C/VCF -lhtslib-1.10.2  2>err.txt


Note: I'm guessing what you need and what each item is. -I should be used with the path to header -l prepended to the library name and -L preceding the path to the library.

Last edited by GRMartin; 01-15-2020 at 03:22 PM..
# 6  
Old 01-15-2020
library path(s) and subfolders within

Thanks!

Does the -L option ensure the sub-directories are searched recursively?

The -lhtslib-1.10.2 option of your command line may be wrong in this case as htslib-1.10.2 is a folder under which there are several sub-folders and many *.c *.h *.o *.pico files, but I understand your point. Two files: libhts.so and libhts.a may be the ones I need(permission is checked to be correct). So I tried:
Code:
$ gcc -Wall -O3 -o vcf_parser01 vcf.c vcf_parser01.c  -Wl,-rpath,/home/yifangt/Study/C/VCF  -L/home/yifangt/Study/C/VCF/htslib-1.10.2 -lhts 

  /tmp/ccSYd2K4.o: In function `bcf_hdr_set_idx.isra.10':
vcf.c:(.text+0x11d5): undefined reference to `hts_resize_array_'
/tmp/ccSYd2K4.o: In function `vcf_hdr_read':
vcf.c:(.text+0x66ad): undefined reference to `tbx_index_load3'
/tmp/ccSYd2K4.o: In function `vcf_write':
vcf.c:(.text+0xd57a): undefined reference to `hts_idx_tbi_name'
/tmp/ccSYd2K4.o: In function `bcf_index_load3':
vcf.c:(.text+0xdd55): undefined reference to `hts_idx_load3'
/tmp/ccSYd2K4.o: In function `bcf_idx_save':
vcf.c:(.text+0xe181): undefined reference to `sam_idx_save'
collect2: error: ld returned 1 exit status

Under the htslib-1.10.2 folder, there are three sub folders cram/, htslib/ and m4/ that may be involved.

In htslib/ sub-folder, *.h files only, so that's for the headers;
In cram/ and m4/ sub-folders, there are *.c, *.h and *.o files (and some other types, which seems for portability, and may not be needed, very likely to me!);
Also I have tried archive the *.o files into library in the format as mylibx.a or mylibx.so files, the error stayed the same. I have attached the link to download the library and my src code. Could it be possible for you to download the library to have a look at the code? That may be easier than what I explained.
Thanks a lot.

Last edited by yifangt; 01-15-2020 at 04:08 PM.. Reason: Add high lighter, correct typos
# 7  
Old 01-15-2020
Quote:
Originally Posted by yifangt
Thanks!

Does the -L option ensure the sub-directories are searched recursively?


No. Each folder requires a -L entry.



Quote:
Originally Posted by yifangt


The -lhtslib-1.10.2 option of your command line may be wrong in this case as htslib-1.10.2 is a folder under which there are several sub-folders and many *.c *.h *.o *.pico files, but I understand your point. Two files: libhts.so and libhts.a may be the ones I need(permission is checked to be correct). So I tried:


I presumed htslib-1.10-2 was your library. The -l option is for the library itself. If the library is libhts.so then use -lhts


Quote:
Originally Posted by yifangt


Could it be possible for you to download the library to have a look at the code? That may be easier than what I explained.
Thanks a lot.

I'll have a look.

--- Post updated at 09:38 PM ---

Ok. I didn't see a link but grabbed it from github.


If you use the archive its simplest. I'm going to call the root path of the directory where you build the library LIBROOT. You should replace this with the actual path.
Code:
gcc -Wall -O3 -o vcf_parser01 vcf.c vcf_parser01.c -I LIBROOT/htslib LIBROOT/libhts.a

should work.
This User Gave Thanks to GRMartin For This Post:
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

System-wide search

When looking for wherever a program or a filename appears in the system, a short scrip is "findinner" which another script calls with a long parameter list consisting of path names ending with ".sh" or ".menu". "findinner" looks like this: # If not .savenn file, show name and result of grep. #... (4 Replies)
Discussion started by: wbport
4 Replies

2. Red Hat

Changing system-wide for umask

Hi everybody, How can I change the default UMASK for non root users, e.g. I want the umask for every new created user will be 0044. Thanks (6 Replies)
Discussion started by: leo_ultra_leo
6 Replies

3. Linux

How to run User-mode Linux installed with synaptic package manager in Ubuntu 10.10

I have installed user-mode linux kernel in Ubuntu 10.10 with the help of Synaptic package manager. But I'm not getting how to run it. If we install it manually, we've to run it using the executable binary file. But here, I'm unable to locate any such file. Please help.... Thanking You.... ... (0 Replies)
Discussion started by: rohitadeshmukh1
0 Replies

4. Linux

System wide find and sort

Hi, I need to look for a config file (ldap.conf) and pick the latest modified file. `locate` tells me there are many ldap.conf's, some in /etc, /usr, /home, etc. Is there some way I can sort them by last modified time via bash? I was thinking maybe I could pipe the output of `locate` to `ls... (4 Replies)
Discussion started by: Housni
4 Replies

5. Shell Programming and Scripting

system wide password change

Hello, I am new to shell scripting and I was trying to write a script that would force a system wide password change except for admins. I am having some trouble and any help that someone could give me would be greatly appreciated. I am trying to do it by using the UID as the marker for anyone... (6 Replies)
Discussion started by: kilemark
6 Replies

6. Filesystems, Disks and Memory

How to mount/make a FAT system on Linux

Yea i was wondering how i would mount, and create a FAT directory that way i can save files in the FAT directory in a windows system and be able to access them on Linux systems. Or if there is any other way to share files between Linux and Windows. Any responds will help... thanks! (2 Replies)
Discussion started by: kyoist
2 Replies

7. UNIX for Dummies Questions & Answers

FIND function - system wide

Hi, I have a task to search for a file called 'Xstartup' in the whole system because there might be different versions of it which overrite eachother. Can anyone suggest a smart command to run this search ? The machine needs to scan every single folder beginning from root. Please help, I am... (5 Replies)
Discussion started by: DGoubine
5 Replies

8. UNIX for Advanced & Expert Users

System wide CDE setup

Does anyone know how to make system wide changes to the CDE's front panel icons? I dont know if it matters but im running Solaris 9. THanks (1 Reply)
Discussion started by: meyersp
1 Replies

9. UNIX for Dummies Questions & Answers

links working system wide

I have created symbolic links to several frequently used commands, for example: "lt" is a link to "ls -ltrgo|tail". What can I do to make these links available system-wide, or at least in the directories my coworkers are in most of the time? I have copied the link to several directories, and... (6 Replies)
Discussion started by: jpprial
6 Replies
Login or Register to Ask a Question