Sponsored Content
Top Forums Shell Programming and Scripting Merging multiple files using lines from one file Post 302704843 by iconig on Sunday 23rd of September 2012 05:34:02 PM
Old 09-23-2012
Merging multiple files using lines from one file

I have been working of this script for a very long time and I have searched the internet for direction but I am stuck here.
I have about 3000 files with two columns each. The length of each file is 50000. Each of these files is named this way b.4, b.5, b.6, b.7, b.8, b.9, b.10, b.11, b.12 .......b.3000

For example b.4 and b.10 looks like this: b.4
Code:
1     1
2     1
3     1
4     0
5     1
6     0.75
7     1
8      1
8      0
9    0.34
10    0.45
.....
50000  1

b.10:
Code:
1     1
2     0.87
3     1
4     1
5     0.89
6      1
7      0
8   
9
10     1
.....
50000  1


All the other b. files have the same first column.

I now have another file named 'list'. list has just one column with 3000 lines.
cat $ ls list:
Code:
4
7
10

...
3000

I want to list to compare the b.4, b.5, b.6, b.7, b.8, b.9, b.10, b.11,....... to b.3000. Then print out two columns of 3000 lines from these files common to file "list". The output should look like this:
Code:
4      0         1
7      1         1     
10    0.45      1
..
3000   1         1

Code:
for i in 'home/John/list'
do
      echo $i > fn1
for files in 'home/John/b*'
do
       echo $file > ff.
       paste fn1 ffn fnuse.temp
       cp fnuse.temp fnuse
     done
done

Please guys, I need your help. Thanks

Last edited by Franklin52; 09-24-2012 at 04:54 AM.. Reason: Please use code tags for data and code samples
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Merging columns from multiple files in one file

Hi, I want to select columns from multiple files and combine them in one file. The files are simulation-data-files with 23 columns each and about 50 rows. I now use: cut -f 11 Sweep?wing-30?scale=0.?0?fan2?.txt | pr -3 | awk '{printf("\n%s\t%s\t%s",$1,$2,$3)}' > ../Data_Processed/output.txtI... (1 Reply)
Discussion started by: isgoed
1 Replies

2. Shell Programming and Scripting

Matching lines across multiple csv files and merging a particular field

I have about 20 CSV's that all look like this: "","","","","","","","","","","","","","","",""What I've been told I need to produce is the exact same thing, but with each file now containing the start_code from every other file where the email matches. It doesn't matter if any of the other... (1 Reply)
Discussion started by: Demosthenes
1 Replies

3. Shell Programming and Scripting

Merging information from multiple files to a single file

Hello, I am new to unix and need help with a problem. I have 2 files each containing multiple columns of information ie; File 1 : A B C D E 1 2 3 4 5 File 2 : F G 6 7 I would like to merge the information from File 2 to File 1 so that the data reads as follows; File 1: A... (4 Replies)
Discussion started by: crunchie
4 Replies

4. Shell Programming and Scripting

merging two .txt files by alternating x lines from file 1 and y lines from file2

Hi everyone, I have two files (A and B) and want to combine them to one by always taking 10 rows from file A and subsequently 6 lines from file B. This process shall be repeated 40 times (file A = 400 lines; file B = 240 lines). Does anybody have an idea how to do that using perl, awk or sed?... (6 Replies)
Discussion started by: ink_LE
6 Replies

5. Shell Programming and Scripting

merging multiple lines into single line

Hi, 1. Each message starts with date 2. There is blank line between each message 3. Each message does not contain same number of lines. Any help in merging multiple lines in each message to a single line is much appreciated. AIX: Korn Shell Error log file looks like below. ... (5 Replies)
Discussion started by: bala123
5 Replies

6. Shell Programming and Scripting

Merging multiple files from multiple columns

Hi guys, I have very basic linux experience so I need some help with a problem. I have 3 files from which I want to extract columns based on common fields between them. File1: --- rs74078040 NA 51288690 T G 461652 0.99223 0.53611 3 --- rs77209296 NA 51303525 T G 461843 0.98973 0.60837 3... (10 Replies)
Discussion started by: bartman2099
10 Replies

7. Shell Programming and Scripting

awk Merging multiple files with symbol representing new file

I just tried following ls *.dat|sort -t"_" -k2n,2|while read f1 && read f2; do awk '{print}' $f1 awk FNR==1'{print $1,$2,$3,$4,$5,"*","*","*" }' OFS="\t" $f2 awk '{print}' $f2 donegot following result 18-Dec-1983 11:45:00 AM 18.692 84.672 0 25.4 24 18-Dec-1983 ... (3 Replies)
Discussion started by: Akshay Hegde
3 Replies

8. Shell Programming and Scripting

Merging multiple lines

I do have a text file with multiple lines on it. I want to put the lines of text into a single line where ever there is ";" for example ert, ryt, yvig, fgr; rtyu, hjk, uio, hyu, hjo; ghj, tyu, gho, hjp, jklo, kol; The resultant file I would like to have is ert, ryt, yvig, fgr;... (2 Replies)
Discussion started by: Kanja
2 Replies

9. Shell Programming and Scripting

Merging multiple lines to columns with awk, while inserting commas for missing lines

Hello all, I have a large csv file where there are four types of rows I need to merge into one row per person, where there is a column for each possible code / type of row, even if that code/row isn't there for that person. In the csv, a person may be listed from one to four times... (9 Replies)
Discussion started by: RalphNY
9 Replies

10. UNIX for Beginners Questions & Answers

Merging multiple lines into single line based on one column

I Want to merge multiple lines based on the 1st field and keep into single record. SRC File: AAA_POC_DB.TAB1 AAA_POC_DB.TAB2 AAA_POC_DB.TAB3 AAA_POC_DB.TAB4 BBB_POC_DB.TAB1 BBB_POC_DB.TAB2 CCC_POC_DB.TAB6 OUTPUT ----------------- 'AAA_POC_DB','TAB1','TAB2','TAB3','TAB4'... (10 Replies)
Discussion started by: raju2016
10 Replies
GITCLI(7)							    Git Manual								 GITCLI(7)

NAME
       gitcli - Git command-line interface and conventions

SYNOPSIS
       gitcli

DESCRIPTION
       This manual describes the convention used throughout Git CLI.

       Many commands take revisions (most often "commits", but sometimes "tree-ish", depending on the context and command) and paths as their
       arguments. Here are the rules:

       o   Revisions come first and then paths. E.g. in git diff v1.0 v2.0 arch/x86 include/asm-x86, v1.0 and v2.0 are revisions and arch/x86 and
	   include/asm-x86 are paths.

       o   When an argument can be misunderstood as either a revision or a path, they can be disambiguated by placing -- between them. E.g.  git
	   diff -- HEAD is, "I have a file called HEAD in my work tree. Please show changes between the version I staged in the index and what I
	   have in the work tree for that file", not "show difference between the HEAD commit and the work tree as a whole". You can say git diff
	   HEAD -- to ask for the latter.

       o   Without disambiguating --, Git makes a reasonable guess, but errors out and asking you to disambiguate when ambiguous. E.g. if you have
	   a file called HEAD in your work tree, git diff HEAD is ambiguous, and you have to say either git diff HEAD -- or git diff -- HEAD to
	   disambiguate.

	   When writing a script that is expected to handle random user-input, it is a good practice to make it explicit which arguments are which
	   by placing disambiguating -- at appropriate places.

       o   Many commands allow wildcards in paths, but you need to protect them from getting globbed by the shell. These two mean different
	   things:

	       $ git checkout -- *.c
	       $ git checkout -- *.c

	   The former lets your shell expand the fileglob, and you are asking the dot-C files in your working tree to be overwritten with the
	   version in the index. The latter passes the *.c to Git, and you are asking the paths in the index that match the pattern to be checked
	   out to your working tree. After running git add hello.c; rm hello.c, you will not see hello.c in your working tree with the former, but
	   with the latter you will.

       o   Just as the filesystem .  (period) refers to the current directory, using a .  as a repository name in Git (a dot-repository) is a
	   relative path and means your current repository.

       Here are the rules regarding the "flags" that you should follow when you are scripting Git:

       o   it's preferred to use the non-dashed form of Git commands, which means that you should prefer git foo to git-foo.

       o   splitting short options to separate words (prefer git foo -a -b to git foo -ab, the latter may not even work).

       o   when a command-line option takes an argument, use the stuck form. In other words, write git foo -oArg instead of git foo -o Arg for
	   short options, and git foo --long-opt=Arg instead of git foo --long-opt Arg for long options. An option that takes optional
	   option-argument must be written in the stuck form.

       o   when you give a revision parameter to a command, make sure the parameter is not ambiguous with a name of a file in the work tree. E.g.
	   do not write git log -1 HEAD but write git log -1 HEAD --; the former will not work if you happen to have a file called HEAD in the
	   work tree.

       o   many commands allow a long option --option to be abbreviated only to their unique prefix (e.g. if there is no other option whose name
	   begins with opt, you may be able to spell --opt to invoke the --option flag), but you should fully spell them out when writing your
	   scripts; later versions of Git may introduce a new option whose name shares the same prefix, e.g.  --optimize, to make a short prefix
	   that used to be unique no longer unique.

ENHANCED OPTION PARSER
       From the Git 1.5.4 series and further, many Git commands (not all of them at the time of the writing though) come with an enhanced option
       parser.

       Here is a list of the facilities provided by this option parser.

   Magic Options
       Commands which have the enhanced option parser activated all understand a couple of magic command-line options:

       -h
	   gives a pretty printed usage of the command.

	       $ git describe -h
	       usage: git describe [options] <commit-ish>*
		  or: git describe [options] --dirty

		   --contains		 find the tag that comes after the commit
		   --debug		 debug search strategy on stderr
		   --all		 use any ref
		   --tags		 use any tag, even unannotated
		   --long		 always use long format
		   --abbrev[=<n>]	 use <n> digits to display SHA-1s

       --help-all
	   Some Git commands take options that are only used for plumbing or that are deprecated, and such options are hidden from the default
	   usage. This option gives the full list of options.

   Negating options
       Options with long option names can be negated by prefixing --no-. For example, git branch has the option --track which is on by default.
       You can use --no-track to override that behaviour. The same goes for --color and --no-color.

   Aggregating short options
       Commands that support the enhanced option parser allow you to aggregate short options. This means that you can for example use git rm -rf
       or git clean -fdx.

   Abbreviating long options
       Commands that support the enhanced option parser accepts unique prefix of a long option as if it is fully spelled out, but use this with a
       caution. For example, git commit --amen behaves as if you typed git commit --amend, but that is true only until a later version of Git
       introduces another option that shares the same prefix, e.g. git commit --amenity option.

   Separating argument from the option
       You can write the mandatory option parameter to an option as a separate word on the command line. That means that all the following uses
       work:

	   $ git foo --long-opt=Arg
	   $ git foo --long-opt Arg
	   $ git foo -oArg
	   $ git foo -o Arg

       However, this is NOT allowed for switches with an optional value, where the stuck form must be used:

	   $ git describe --abbrev HEAD     # correct
	   $ git describe --abbrev=10 HEAD  # correct
	   $ git describe --abbrev 10 HEAD  # NOT WHAT YOU MEANT

NOTES ON FREQUENTLY CONFUSED OPTIONS
       Many commands that can work on files in the working tree and/or in the index can take --cached and/or --index options. Sometimes people
       incorrectly think that, because the index was originally called cache, these two are synonyms. They are not -- these two options mean very
       different things.

       o   The --cached option is used to ask a command that usually works on files in the working tree to only work with the index. For example,
	   git grep, when used without a commit to specify from which commit to look for strings in, usually works on files in the working tree,
	   but with the --cached option, it looks for strings in the index.

       o   The --index option is used to ask a command that usually works on files in the working tree to also affect the index. For example, git
	   stash apply usually merges changes recorded in a stash entry to the working tree, but with the --index option, it also merges changes
	   to the index as well.

       git apply command can be used with --cached and --index (but not at the same time). Usually the command only affects the files in the
       working tree, but with --index, it patches both the files and their index entries, and with --cached, it modifies only the index entries.

       See also http://marc.info/?l=git&m=116563135620359 and http://marc.info/?l=git&m=119150393620273 for further information.

GIT
       Part of the git(1) suite

Git 2.17.1							    10/05/2018								 GITCLI(7)
All times are GMT -4. The time now is 09:03 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy