Sponsored Content
Top Forums UNIX for Dummies Questions & Answers merge files along with file names (awk)? Post 302570919 by Corona688 on Friday 4th of November 2011 04:26:42 PM
Old 11-04-2011
what is this 'fileID'? Is it a file or a command? I'm assuming a file.

Shell variables don't work inside awk. awk has a special FILENAME variable you can use instead.

You don't have to reopen betaout 5,000 times to process 5,000 files. You can do it once.

You don't have to store the entire list of files in a shell variable -- that's actually dangerous, if the file's big enough, bits will be lopped off the end.

Code:
while read i
do
        awk '{if($5 < 0.01) printf $0"\n"}' file.${i}.extension
done < fileID >> betaout

This User Gave Thanks to Corona688 For This Post:
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Merge two files whose names are given in other file

Hi, I have a pointer file ptr.txt. There may be any number of files mentioned in the ptr.txt file eg: cat ptr.txt /home/abc.txt /home/pqr.txt /home/xyz.txt I have to read this pointer file and merge the files given in the pointer file so that final file say... (1 Reply)
Discussion started by: harshada
1 Replies

2. Shell Programming and Scripting

merge two files using awk

Hi Guys, I wonder whether is possible to merge two files using awk. I have two files one with 7 columns and another one with 9 columns and the first column on both files is identical so will be my key to merge the files. Any ideas.Thanks in advance. Harby. (2 Replies)
Discussion started by: hariza
2 Replies

3. Shell Programming and Scripting

Merge files of differrent size with one field common in both files using awk

hi, i am facing a problem in merging two files using awk, the problem is as stated below, file1: A|B|C|D|E|F|G|H|I|1 M|N|O|P|Q|R|S|T|U|2 AA|BB|CC|DD|EE|FF|GG|HH|II|1 .... .... .... file2 : 1|Mn|op|qr (2 Replies)
Discussion started by: shashi1982
2 Replies

4. Shell Programming and Scripting

merge two files into one file use awk

Hi, guys. I have one question: I have two files: passwd and shadow (the number of records in these files are not equal)the contents of them are below: passwd: ************** ftp:x:24:24: sshd:x:71:65: uucp:x:10:14: brownj:x:5005:1000: sherrys: x :5006:1000: ... ************* ... (2 Replies)
Discussion started by: daikeyang
2 Replies

5. Shell Programming and Scripting

Merge files using AWK

I want to merge data from 2nd file to 1st file based on 1st column File1 ==== data1,12,comp1 data1,13,comp2 data3,14,, File2 ==== data1,11,host1,lit data2,11,host2,lit3 data3,11,host3,lit4 Required Ouput (5 Replies)
Discussion started by: greycells
5 Replies

6. Shell Programming and Scripting

merge two files with awk

I have two file like follows. I want to merge them according the first field of file1. The separator of file1 is tab, while the second one is ",". I tried some code, but can't get the results. File1: AABB 6072 28 5922 BBCC 316 147 162 CCDD 907 71 231 File2: CCDD,hTRBV12-4,hTRBJ2-3,319895... (7 Replies)
Discussion started by: xshang
7 Replies

7. UNIX for Dummies Questions & Answers

Merge files with file names added

I want to merge several files with identical format: file 1: rs3094315 0.0006105222804 0.9528743638 rs3131972 -0.05461465109 0.3139864854 rs3115860 -0.06041530955 0.3195499498 file 2: rs2073813 -0.06039552152 0.2956527097 rs11564776 -0.1864266568 ... (4 Replies)
Discussion started by: luoruicd
4 Replies

8. Shell Programming and Scripting

awk merge two files

file1 AAA3:WWW1:DDD1:XXX8:DDD2:XXX9 AAA6:WWW2:FFF1:XXX130:FFF1:XXX104:FFF1:XXX16 AAA7:WWW3:ZZZ1:XXX4:ZZZ2:XXX5:ZZZ3:XXX6:ZZZ4:XXX7file2 XXX8:EEE1:EEE2 XXX9:KKK1:KKK2 XXX130:OOO1:OOO2 XXX104:PPP1:PPP2 XXX16:RRR1:RRR1 XXX4:UUU1:UUU2 XXX5:III1:III2 XXX7:JJJ1:JJJ2Result... (2 Replies)
Discussion started by: vikus
2 Replies

9. Shell Programming and Scripting

Exclude certain file names while selectingData files coming in different names in a file name called

Data files coming in different names in a file name called process.txt. 1. shipments_yyyymmdd.gz 2 Order_yyyymmdd.gz 3. Invoice_yyyymmdd.gz 4. globalorder_yyyymmdd.gz The process needs to discard all the below files and only process two of the 4 file names available ... (1 Reply)
Discussion started by: dsravanam
1 Replies

10. Shell Programming and Scripting

Merge two files on awk/shell

Hi, i have two files like these: FILE 1 00:0f:35:1b:0c:00 1402691094.750049000 00:0f:35:1b:0c:00 1402691087.474893000 44:d3:ca:fd:a2:08 1402691091.865127000 30:e4:db:c1:df:de 1402691090.192464000 FILE 2_ 00:0F:35 Cisco Systems, Inc 30:E4:DB Cisco Systems, Inc I need a file 3, that... (5 Replies)
Discussion started by: bertiko
5 Replies
funmerge(1)							SAORD Documentation						       funmerge(1)

NAME
funmerge - merge one or more Funtools table files SYNOPSIS
funmerge [-w|-x] -f [colname] <iname1> <iname2> ... <oname> OPTIONS
-f # output a column specifying file from which this event came -w # adjust position values using WCS info -x # adjust position values using WCS info and save old values DESCRIPTION
funmerge merges FITS data from one or more FITS Binary Table files or raw event files. The first argument to the program specifies the first input FITS table or raw event file. If "stdin" is specified, data are read from the standard input. Use Funtools Bracket Notation to specify FITS extensions and row filters. Subsequent arguments specify additional event files and tables to merge. (NB: Stdin cannot not be used for any of these additional input file arguments.) The last argument is the out- put FITS file. The columns in each input table must be identical. If an input file begins with the '@' character, it is processed as an include file, i.e., as a text file containing event file names (as well as blank lines and/or comment lines starting with the '#' sign). If standard input is specified as an include file ('@stdin'), then file names are read from the standard input until EOF (^D). Event files and include files can be mixed on a command line. Rows from each table are written sequentially to the output file. If the switch -f [colname] is specified on the command line, an addi- tional column is added to each row containing the number of the file from which that row was taken (starting from one). In this case, the corresponding file names are stored in the header parameters having the prefix FUNFIL, i.e., FUNFIL01, FUNFIL02, etc. Using the -w switch (or -x switch as described below), funmerge also can adjust the position column values using the WCS information in each file. (By position columns, we mean the columns that the table is binned on, i.e., those columns defined by the bincols= switch, or (X,Y) by default.) To perform WCS alignment, the WCS of the first file is taken as the base WCS. Each position in subsequent files is adjusted by first converting it to the sky coordinate in its own WCS coordinate system, then by converting this sky position to the sky position of the base WCS, and finally converting back to a pixel position in the base system. Note that in order to perform WCS alignment, the appropriate WCS and TLMIN/TLMAX keywords must already exist in each FITS file. When performing WCS alignment, you can save the original positions in the output file by using the -x (for "xtra") switch instead of the -w switch (i.e., using this switch also implies using -w) The old positions are saved in columns having the same name as the original positional columns, with the added prefix "OLD_". Examples: Merge two tables, and preserve the originating file number for each row in the column called "FILE" (along with the corresponding file name in the header): [sh] funmerge -f "FILE" test.ev test2.ev merge.ev Merge two tables with WCS alignment, saving the old position values in 2 additional columns: [sh] funmerge -x test.ev test2.ev merge.ev This program only works on raw event files and binary tables. We have not yet implemented image and array merging. SEE ALSO
See funtools(7) for a list of Funtools help pages version 1.4.2 January 2, 2008 funmerge(1)
All times are GMT -4. The time now is 08:09 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy