Need a key for "paste" command


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Need a key for "paste" command
# 1  
Old 08-19-2012
Need a key for "paste" command

Is there a way to use the "paste" command to combine files and have each line key off of a column? All this time I've been pasting files together and foolishly not checking the line count of each file. I have just found out that the line count is not the same for each file so the data is not being synchronised properly across the two files. I'd like to avoid using a for loop to grep one line of each file, and the key in that file, against each line of the correlating file as the files are large and would increase the overhead. Anyway I can tell paste to match lines that have a correlating key column?
# 2  
Old 08-19-2012
post sample input files, the key/column to join on and the desired output.
# 3  
Old 08-19-2012
One file is standard /etc/passwd entries:
Code:
user1:1:1:user1's comment field:/home/user1:/bin/ksh
user2:2:2:user2's comment field:/home/user2:/bin/ksh
user3:3:3:user3's comment field:/home/user3:/bin/ksh

The joining file is a class assigned to the user:
Code:
user2:CLASS_FIFTEEN
user1:CLASS_ONE
user3:ADMINISTRATIVE_CLASS

Both files would key off of the user's UNIX login name (user1, user2, user3, etc). Although I could sort the files and uniq them they're still not the same line count hence I need to make sure the paste matches the user login when I join them. GNU Linux using ksh (forgot to post that too).
# 4  
Old 08-19-2012
Sounds like a job for the join utility.

Regards,
Alister
# 5  
Old 08-19-2012
Thanks - this appears to be what I need however I'm not receiving the desired output. I've read the man page and googled a bit but all examples I see are with two files that either have 2 fileds or are whitespace-delimited. My files are not in this format. I tried:
Code:
join  -t: txt txt1

but this gives a line count of 50 whereas the two files contain 10 lines. I checked the two files and 8 of the entries should match up . Is there a way to specify the first and second field delimited by :?

Forgive me but I was trying to keep the data as clean as possible and I neglected to consider the lines are preceded by the server name so they look like this:
Code:
SERVER1:user1:1:1:user1's comment field:/home/user1:/bin/ksh
SERVER1:user2:2:2:user2's comment field:/home/user2:/bin/ksh
SERVER1:user3:3:3:user3's comment field:/home/user3:/bin/ksh

and
Code:
SERVER1:user2:CLASS_FIFTEEN
SERVER1:user1:CLASS_ONE
SERVER1:user3:ADMINISTRATIVE_CLASS

I tried using -1 and -2 like so (per the man page):
Code:
join -t$1:$2 -1 -2 file1 file2

But it's not working as $1 and $2 was just a guess at defining those fields as a delimiter.
# 6  
Old 08-19-2012
I don't know whether I understood it or not, but:

Code:
lem@biggy:/tmp$ cat file1
SERVER1:user1:1:1:user1's comment field:/home/user1:/bin/ksh
SERVER1:user2:2:2:user2's comment field:/home/user2:/bin/ksh
SERVER1:user3:3:3:user3's comment field:/home/user3:/bin/ksh
SERVER1:user8:8:8:user8's comment field:/home/user8:/bin/ksh
lem@biggy:/tmp$ cat file2
SERVER1:user2:CLASS_FIFTEEN
SERVER1:user1:CLASS_ONE
SERVER1:user3:ADMINISTRATIVE_CLASSO
SERVER1:user4:ADMINISTRATIVE_CLASSO
lem@biggy:/tmp$ sort -h file1 | uniq > file1sorted 
lem@biggy:/tmp$ sort -h file2 | uniq > file2sorted 
lem@biggy:/tmp$ join -t: -j2 -o 1.1 1.2 1.3 1.4 1.5 1.6 1.7 2.3 file1sorted file2sorted > newfile
lem@biggy:/tmp$ cat newfile
SERVER1:user1:1:1:user1's comment field:/home/user1:/bin/ksh:CLASS_ONE
SERVER1:user2:2:2:user2's comment field:/home/user2:/bin/ksh:CLASS_FIFTEEN
SERVER1:user3:3:3:user3's comment field:/home/user3:/bin/ksh:ADMINISTRATIVE_CLASSO

Is this acceptable?
--
Bye
# 7  
Old 08-19-2012
Quote:
Originally Posted by Lem
Code:
lem@biggy:/tmp$ sort -h file1 | uniq > file1sorted
lem@biggy:/tmp$ sort -h file2 | uniq > file2sorted
lem@biggy:/tmp$ join -t: -j2 -o 1.1 1.2 1.3 1.4 1.5 1.6 1.7 2.3 file1sorted file2sorted > newfile

The sorts should be restricted to the join key, but your sorts use the entire line.

The sort should not be numeric (human readable or otherwise). join expects a lexicographical sort in the same collation sequence.

Unless the sort key (the entire line in this case) begins with a valid number, the -h has no effect. The absence of a valid leading number is interpreted as a 0. Since all lines in your data sample compare equal to 0, the tie is broken using a lexicographical sort of the entire line.

The correct sort would be: sort -b -t: -k2,2

I realize that this is for a Linux machine, but -j2 instead of -12 -22? Why? You lose the ability to run that join on any implementation in exchange for saving 4 characters on the command line. A bad bargain, in my opinion.

Regards,
Alister
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Bash script - Print an ascii file using specific font "Latin Modern Mono 12" "regular" "9"

Hello. System : opensuse leap 42.3 I have a bash script that build a text file. I would like the last command doing : print_cmd -o page-left=43 -o page-right=22 -o page-top=28 -o page-bottom=43 -o font=LatinModernMono12:regular:9 some_file.txt where : print_cmd ::= some printing... (1 Reply)
Discussion started by: jcdole
1 Replies

2. UNIX for Dummies Questions & Answers

Using "mailx" command to read "to" and "cc" email addreses from input file

How to use "mailx" command to do e-mail reading the input file containing email address, where column 1 has name and column 2 containing “To” e-mail address and column 3 contains “cc” e-mail address to include with same email. Sample input file, email.txt Below is an sample code where... (2 Replies)
Discussion started by: asjaiswal
2 Replies

3. Web Development

How would I mod_rewrite "/~a1Pha" and "/=a1Pha" to "/paste/a1Pha.htm"? (internally & externally)

Basically I want to shorten URLs on my html pasting site (pasteht.ml), by using "/~a1Pha" instead of "/paste/a1Pha". The ID is 5 numbers and letters, both cases. For example: /~idnum serves /paste/idnum.htm /=idnum serves /paste/idnum.htm /paste/idnum redirects to /~idnum (to update any old... (0 Replies)
Discussion started by: phillips1012
0 Replies

4. UNIX for Dummies Questions & Answers

Unix "look" Command "File too large" Error Message

I am trying to find lines in a text file larger than 3 Gb that start with a given string. My command looks like this: $ look "string" "/home/patrick/filename.txt" However, this gives me the following message: "look: /home/patrick/filename.txt: File too large" So, I have two... (14 Replies)
Discussion started by: shishong
14 Replies

5. Shell Programming and Scripting

awk command to replace ";" with "|" and ""|" at diferent places in line of file

Hi, I have line in input file as below: 3G_CENTRAL;INDONESIA_(M)_TELKOMSEL;SPECIAL_WORLD_GRP_7_FA_2_TELKOMSEL My expected output for line in the file must be : "1-Radon1-cMOC_deg"|"LDIndex"|"3G_CENTRAL|INDONESIA_(M)_TELKOMSEL"|LAST|"SPECIAL_WORLD_GRP_7_FA_2_TELKOMSEL" Can someone... (7 Replies)
Discussion started by: shis100
7 Replies

6. Shell Programming and Scripting

"Join" or "Merge" more than 2 files into single output based on common key (column)

Hi All, I have working (Perl) code to combine 2 input files into a single output file using the join function that works to a point, but has the following limitations: 1. I am restrained to 2 input files only. 2. Only the "matched" fields are written out to the "matched" output file and... (1 Reply)
Discussion started by: Katabatic
1 Replies

7. UNIX for Dummies Questions & Answers

Explanation of "total" field in "ls -l" command output

When I do a listing in one particular directory (ls -al) I get: total 43456 drwxrwxrwx 2 root root 4096 drwxrwxrwx 3 root root 4096 -rwxrwxr-x 1 nobody nobody 3701594 -rwxrwxr-x 1 nobody nobody 3108510 -rwxrwxr-x 1 nobody nobody 3070580 -rwxrwxr-x 1 nobody nobody 3099733 -rwxrwxr-x 1... (1 Reply)
Discussion started by: proactiveaditya
1 Replies

8. Shell Programming and Scripting

Command Character size limit in the "sh" and "bourne" shell

Hi!!.. I would like to know what is maximum character size for a command in the "sh" or "bourne" shell? Thanks in advance.. Roshan. (1 Reply)
Discussion started by: Roshan1286
1 Replies

9. UNIX for Advanced & Expert Users

Command Character size limit in the "sh" and "bourne" shell

Hi!!.. I would like to know what is maximum character size for a command in the "sh" or "bourne" shell? Thanks in advance.. Roshan. (1 Reply)
Discussion started by: Roshan1286
1 Replies

10. UNIX for Dummies Questions & Answers

Command Character size limit in the "sh" and "bourne" shell

Hi!!.. I would like to know what is maximum character size for a command in the "sh" or "bourne" shell? Thanks in advance.. Roshan. (1 Reply)
Discussion started by: Roshan1286
1 Replies
Login or Register to Ask a Question