Visit Our UNIX and Linux User Community


Copying number by looking a large file


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Copying number by looking a large file
# 1  
Old 10-05-2011
Copying number by looking a large file

Hi All,

I have a big file which looks like this:

Code:
abc 34.32
cdf 343.45
computer 1.34
ladder 2.3422

I have some 100000 .TXT files which look like this:

Code:
computer
cdf
align

I have to open each of the text files and read the words from the text files. Then I have to look into that dictionary for the occurrence of the word and copy on the number in 1.num file. This is what I mean:

Contents of 1.num
Code:
1.34
343.45
0

As you can see, I have copied the numbers matching with words in 1.txt from that big file. 0 for no-occurrence.

I have to do this for all 100000 .txt files.

I have tried some lookup techniques but not working though.

Code:
for txt in *.txt; do                                                         
  awk '
     NR == FNR { a[$1]=$1 }
     NR != FNR { print $1, a[$1] }
  ' big_file.dat $txt >${txt%txt}num
done

I am doing this in Linux with BASH
# 2  
Old 10-05-2011
Code:
for txt in *.txt; do                                                         
  awk 'NR==FNR{a[$1]=$2;next} {print a[$1]==""?"0":a[$1]}' big_file.dat $txt  > ${txt%txt}num
done

This User Gave Thanks to rdcwayx For This Post:
# 3  
Old 10-05-2011
Seems it is not giving me the required output. What it is showing me in 1.num are the "words" which have matched rather than numbers, as I have shown in the example above. I am trying to edit your code and make it work. If I can do it, I'll post it here.

---------- Post updated at 01:27 PM ---------- Previous update was at 01:12 PM ----------

I am sorry. Its fine. It was the problem with my data file. Really sorry.

Previous Thread | Next Thread
Test Your Knowledge in Computers #789
Difficulty: Medium
Steve Wozniak was employed at Sun Microsystems before he founded Apple Computers.
True or False?

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Count number of lines between a pattern in a large file

1000CUS E Y4NYRETAIL 10010004HELIOPOLIS 110000500022360591000056XX EG 1101DEBY XXAD ZSSKY TSSROS 1102HANYNNYY@HOTMAIL.COM 210030/05/201301/06/2013AED 3100 OPE 3100 CLO 3100 The 1000CUS E Y NYCORPORATE 10010004HELIOPOLIS 110000500025270504550203XX EG 1101XXXQ FOR CTING AND... (1 Reply)
Discussion started by: john2022
1 Replies

2. UNIX for Dummies Questions & Answers

Delete large number of columns rom file

Hi, I have a data file that contains 61 columns. I want to delete all the columns except columns, 3,6 and 8. The columns are tab de-limited. How would I achieve this on the terminal? Thanks (2 Replies)
Discussion started by: lost.identity
2 Replies

3. Shell Programming and Scripting

Start copying large file while its still being restored from tape

Hello, I need to copy a 700GB tape-image file over a network. I want to start the copy process before the tape-image has finished being restored from the tape. The tape restore speed is about 78 Mbps and the file transfer speed over the network is about 45 Mbps I don't want to use a pipe, since... (7 Replies)
Discussion started by: swamik
7 Replies

4. Shell Programming and Scripting

Find line number of bad data in large file

Hi Forum. I was trying to search the following scenario on the forum but was not able to. Let's say that I have a very large file that has some bad data in it (for ex: 0.0015 in the 12th column) and I would like to find the line number and remove that particular line. What's the easiest... (3 Replies)
Discussion started by: pchang
3 Replies

5. UNIX for Dummies Questions & Answers

Copying a Large File

I have a large file that I append entries to the end of every few seconds. Its grown to >150MB. Its basically a log file but a perl script is writing to it. I need to make a copy of it to a new directory. I realize the latest entries occuring while the copy is taking place will not be recorded... (1 Reply)
Discussion started by: lforum
1 Replies

6. Shell Programming and Scripting

Copying of large files fail

Hi, I have a process which duplicates files for different environments. As the files arrive, my script (korn shell) makes copies of them (giving a unique name) and then renames the original file so that my process won't get triggered again. I don't like it either, but it's what we were told to... (4 Replies)
Discussion started by: GoldenEye4ever
4 Replies

7. UNIX for Dummies Questions & Answers

Copying large file problem on SVR4 Unix

We have 3 Unix servers all running SVR4 Unix 1.4. I have no problems copying files to and from 2 of the servers using either the rcp command or ftp but when i come to transfer large files to the third server the copy gives up part way through and crashes this server. Copying smaller files using RCP... (7 Replies)
Discussion started by: coatesd
7 Replies

8. Shell Programming and Scripting

script to splite large file to number of small files

Dear All, Could you please help me to split a file contain around 240,000,000 line to 4 files all equally likely , note that we need to maintain that the end of each file should started by start flage (MSISDN) and ended by end flag (End), also the number of the line between the... (10 Replies)
Discussion started by: ahmed.gad
10 Replies

9. Filesystems, Disks and Memory

Strange difference in file size when copying LARGE file..

Hi, Im trying to take a database backup. one of the files is 26 GB. I am using cp -pr to create a backup copy of the database. after the copying is complete, if i do du -hrs on the folders i saw a difference of 2GB. The weird fact is that the BACKUP folder was 2 GB more than the original one! ... (1 Reply)
Discussion started by: 0ktalmagik
1 Replies

10. UNIX for Dummies Questions & Answers

copying a large filesystem

Hi there In my organisation we have a solaris network with /home being automounted from /export/home on a central file server (usual stuff) however, the guy who originally set this up only allocated 3gb to /export/home and now we are really struggling for space. I have a new 18gb disk installed... (3 Replies)
Discussion started by: hcclnoodles
3 Replies

Featured Tech Videos