06-04-2007
Treating bzip2 files as text files
To save space on our development box, I'd like to manipulate tab-delimited files (many of which are huge) in bzip2 format. For simple things, these commands work:
bunzip2 -c file.tab.bz2 | command | command
bzcat file.tab.bz2 | command | command
However, we do most of our work in Makefiles. So quite often I'll need something like this:
outputFile.tab : file1.tab file2.tab
someCommand $^ > $@
Since "someCommand" expects tab-delimited files, and doesn't accept piped input, it seems that I would need a way to either have a "make" variable which is a link to a shell command or a way to make something similar to a symbolic link in bash which allows me to make a shell variable into an alias for a command (bzcat file.tab.bz2).
Is there any way to do this? Here's what I'd like it to look/work like:
outputFile.tab.bz2 : file1.tab.bz2 file2.tab.bz2
someCommand $^ > $@
Or even:
outputFile.tab.bz2 : file1.tab.bz2
someCommand < $< > $@
Thanks in advance,
Shawn
10 More Discussions You Might Find Interesting
1. UNIX for Dummies Questions & Answers
How do I use the grep command to take mutiple text files in a folder and make one huge text file out of them. I'm using Mac OS X and can not find a text tool that does it so I figured I'd resort to the BSD Unix CLI for a solution... there are 5,300 files that I want to write to one huge file so... (7 Replies)
Discussion started by: coppertone
7 Replies
2. Shell Programming and Scripting
Hi Everyone,
I'm really new to all this so I'm really hoping someone can help. I have a directory with ~1000 lists from which I want to extract lines from and write to new files. For simplicity lets say they are shopping lists and I want to write out the lines corresponding to apples to a new... (2 Replies)
Discussion started by: born2phase
2 Replies
3. Shell Programming and Scripting
Hi all,
Say I've got a file /tmp/tempfile.txt which contains a list of files with full path.
Now I would wish to take all filenames in that tempfile as arguments and zip them in a single archived file. So how could I archive this?
Seems like I need to use the xargs command?
Thanks (5 Replies)
Discussion started by: isaacniu
5 Replies
4. Shell Programming and Scripting
Hello everyone,
I work under Ubuntu 11.10 (c-shell)
I need a script to create a new text file whose content is the text of another text files that are in the directory $DIRMAIL at this moment.
I will show you an example:
- On the one hand, there is a directory $DIRMAIL where there are... (1 Reply)
Discussion started by: tenteyu
1 Replies
5. Shell Programming and Scripting
Dear all,
I have a complex data file shown below,,,,,
A_ABCD_13208 0 0 4.16735 141044 902449 1293900 168919
C_ABCD_13208 0 0 4.16735 141044 902449 1293900 168919
A_ABCDEF715 52410.9 18598.2 10611 10754.7 122535 252426 36631.4
C_DBCDI_1353 0... (19 Replies)
Discussion started by: AAWT
19 Replies
6. UNIX for Dummies Questions & Answers
Hello, I have a situation where I want to change a line of text in multiple files, but the problem is that I want to change the text to something unique for each file.
For example, let's say I have five files named bob.txt, joe.txt, john.txt, tom.txt, and zach.txt. Each of these files has a... (5 Replies)
Discussion started by: Scatterbrain26
5 Replies
7. Shell Programming and Scripting
I have a text file with entries like
1186
5556
90844
7873
7722
12
7890.6
78.52
6679
3455
9867
1127
5642
..N so many records like this.
I want to split this file into multiple files like cluster1.txt, cluster2.txt, cluster3.txt, ..... clusterN.txt. (4 Replies)
Discussion started by: sammy777
4 Replies
8. Shell Programming and Scripting
I dont even have a sample script cause I dont know where to start from. My data lookes like this
> sat#16 #data: 15 site:UNZA baseline: 205.9151
0.008 -165.2465 35.8109 40.6685 21.9148 121.1446 26.4629 -18.4976 33.8722
0.017 -165.2243 48.2201 40.6908 ... (8 Replies)
Discussion started by: malandisa
8 Replies
9. Shell Programming and Scripting
Hi,
I have a huge file that has data something like shown below:
huge_file.txt
start regexp
Name=Name1
Title=Analyst
Address=Address1
Department=Finance
end regexp
some text
some text
start regexp
Name=Name2
Title=Controller
Address=Address2
Department=Finance
end regexp (7 Replies)
Discussion started by: r3d3
7 Replies
10. Shell Programming and Scripting
I have 2 txt files, 1.txt and 2.txt which contain the paths to files that need to be linked.
Example 1.txt:
/root/001/folder2/image4.nii.gz
/root/002/folder2/image4.nii.gz
Example 2.txt:
/root/001/folder2/image5.nii.gz
/root/002/folder2/image5.nii.gz
Each line represents images from... (7 Replies)
Discussion started by: LeftoverStew
7 Replies
paste(1) General Commands Manual paste(1)
Name
paste - merge file data
Syntax
paste file1 file2...
paste -dlist file1 file2...
paste -s [-dlist] file1 file2...
Description
In the first two forms, concatenates corresponding lines of the given input files file1, file2, etc. It treats each file as a column or
columns of a table and pastes them together horizontally (parallel merging).
In the last form, the command combines subsequent lines of the input file (serial merging).
In all cases, lines are glued together with the tab character, or with characters from an optionally specified list. Output is to the
standard output, so it can be used as the start of a pipe, or as a filter, if - is used in place of a file name.
Options
- Used in place of any file name, to read a line from the standard input. (There is no prompting).
-dlist Replaces characters of all but last file with nontabs characters (default tab). One or more characters immediately following -d
replace the default tab as the line concatenation character. The list is used circularly, i. e. when exhausted, it is reused. In
parallel merging (i. e. no -s option), the lines from the last file are always terminated with a new-line character, not from the
list. The list may contain the special escape sequences:
(new-line), (tab), \ (backslash), and (empty string, not a null
character). Quoting may be necessary, if characters have special meaning to the shell (for example, to get one backslash, use
-d"\\" ).
Without this option, the new-line characters of each but the last file (or last line in case of the -s option) are replaced by a
tab character. This option allows replacing the tab character by one or more alternate characters (see below).
-s Merges subsequent lines rather than one from each input file. Use tab for concatenation, unless a list is specified with -d
option. Regardless of the list, the very last character of the file is forced to be a new-line.
Examples
ls | paste -d" " -
list directory in one column
ls | paste - - - -
list directory in four columns
paste -s -d"
" file
combine pairs of lines into lines
Diagnostics
line too long
Output lines are restricted to 511 characters.
too many files
Except for -s option, no more than 12 input files may be specified.
See Also
cut(1), grep(1), pr(1)
paste(1)