Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Spacing off when files combined using awk or cat

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Spacing off when files combined using awk or cat
# 1  
Old 01-04-2016
Spacing off when files combined using awk or cat
I have 133 .txt files in a directory that I am combining into 1 file. The problem is when I use awk or cat to combine the files I get out put like this:

output
Code:
				
		85	138662360	KCNT1
		86	138662962	KCNT1
		82	138657053	KCNT1
		83	138657635	KCNT1

		95	138646881	KCNT1
		94	138642912	KCNT1
		98	138669261	KCNT1
		96	138662309	KCNT1
		97	138662661	KCNT1

I know the input and output do not match, but the format of the input is always the same, but it seems the spacing is off when I combine the files. If I do a copy and paste (copy file 1 then 2 and paste them into a text file) I get the desired output.

example input
Code:
name	31	Index	Chromosomal Position	Gene	Inheritance
		122	2106725	TSC2	AD
		124	2115481	TSC2	AD
		121	2105400	TSC2	AD
		82	135782221	TSC1	AD
		81	135782026	TSC1	AD
		126	2138218	TSC2	AD
		123	2113107	TSC2	AD
		125	2126142	TSC2	AD
name2	12	Index	Chromosomal Position	Gene	Inheritance
		1	43396568	SLC2A1	AD, AR
name3	20	Index	Chromosomal Position	Gene	Inheritance
		188	2135240	TSC1	AD
		179	2103379	TSC1 AD
		191	2137899	TSC2	AD
		181	2110617	TSC2	AD
		190	2137857	TSC2	AD
		189	2137806	TSC2	AD
		186	2133798	TSC2	AD
		187	2135074	TSC2	AD
		180	2105400	TSC2	AD
		183	2122822	TSC2	AD
		192	2138218	TSC2	AD
		185	2125937	TSC2	AD
		184	2125788	TSC2	AD
		193	2138269	TSC2	AD
		182	2112981	TSC2	AD

desired output
Code:
name	  31	Index	Chromosomal Position	Gene	Inheritance
                  82	135782221	TSC1	AD
                  81	135782026	TSC1	AD
name3  20	Index	Chromosomal Position	Gene	Inheritance
                  188	2135240	TSC1	AD
                  179	2103379	TSC1	AD
                  191	2137899	TSC1	AD

Last edited by cmccabe; 01-04-2016 at 03:07 PM.. Reason: added output
# 2  
Old 01-04-2016
Not the slightest idea what your problem is. Where does the combined output come into play? I can't find any of those lines in your desired output. What does "spacing is off" mean?
# 3  
Old 01-04-2016
I hope this helps but I think the problem is when I combine two files there are many new lines that the new file contains that are not there when I do a copy and paste. Thank you Smilie.

Code:
awk 'FNR==1{print ""}{print}' *.txt > example.txt

desired output (no spaces between new lines)

Code:
        name1 1 Index Chromosomal Position Gene Inheritance  
                    176 40757228 ADSL AR    
                    51 1.26E+08 ALDH7A1 AR   
                    49 1.26E+08 ALDH7A1 AR   
                    52 1.26E+08 ALDH7A1 AR   
                     50 1.26E+08 ALDH7A1 AR   
                    178 62857727 ARHGEF9 AD, AR  
                     13 1.6E+08 ATP1A2 AD     
        name2 2 Index Chromosomal Position Gene Inheritance    
                    102 52200340 SCN8A AD    
                    134 61991153 CHRNA4 AD   
                    136 62038585 KCNQ2 AD   
        name3 3 Index Chromosomal Position Gene Inheritance   
                    122 2106725 TSC2 AD    
                    124 2115481 TSC2 AD    
                    121 2105400 TSC2 AD      
        name4 4 Index Chromosomal Position Gene Inheritance    
                    4 43394661 SLC2A1 AD, AR   
                    22 1.67E+08 SCN1A AD     
        name5 5 Index Chromosomal Position Gene Inheritance    
                     75 52319081 EFHC1 AD, AR   
                      51 1.67E+08 SCN9A AD    
                       103 1.31E+08 SPTAN1 AD   
                      84 1.47E+08 CNTNAP2 AD   
                       134 6640393 TPP1 AR

example.txt (18.7 KB) 
Last edited by cmccabe; 01-04-2016 at 04:03 PM.. Reason: fixed format
# 4  
Old 01-04-2016
Code:
awk 'FNR==1{print ""}{print}' *.txt > example.txt

You are introducing those undesired lines with the highlighted red part.

Perhaps?
Code:
cat *.txt > example.txt

or
Code:
awk '1' *.txt > example.txt

# 5  
Old 01-04-2016
Unfortunately both the cat and awk commands produced the same output with the newlines between the name1, name2,name3.

output of command
Code:
name1


name2


name3

Code:
name1
name2
name3

Thank you Smilie.
# 6  
Old 01-04-2016
In that case, I have to conclude that some of the source contain those spaces.
You can test what files contain empty lines:

Code:
grep -n '^$' *.txt

It will output filename:linenumber: for each line found empty.

To eliminate empty lines:
Code:
awk 'NF' *.txt > example.txt

Last edited by Aia; 01-04-2016 at 04:48 PM..
# 7  
Old 01-04-2016
If the filenames have spaces in them:

Code:
annovar analysis CAP NGS-02 2015B-R2.txt
Annovar Analysis F41520
Annovar Analysis H52520

Is this ok? Thank you Smilie.

The issue seems to be that in each one of the individual text files there is a bunch of whitespace at the end of each file:
Code:
name1



name2



name3



name4




name5

I need to remove this white space from all 133 files before cat, maybe with a bash loop? Thank you Smilie
Last edited by cmccabe; 01-04-2016 at 05:42 PM.. Reason: added edit
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Lost redirecting stderr & stdout to 3 files - one each plus combined

Hi folks I need/want to redirect output (stdout, stderr) from an exec call to separate files. One for stderr only and two(!) different (!) ones for the combined output of stderr and stdout. After some research and testing i got this so far : (( exec ${command} ${command_parameters} 3>&1... (6 Replies)
Discussion started by: MDominok
6 Replies

2. Shell Programming and Scripting

Combined sed+awk for lookup csv file

have written a combined sed+awk to perform a lookup operation which works but looking to enhance it. looking to match a record using any of the comma separated values + return selected fields from the record - including the field header. so: cat foo make,model,engine,trim,value... (6 Replies)
Discussion started by: jack.bauer
6 Replies

3. Shell Programming and Scripting

awk combined with an IF

Hi everybody! I try to printout a csv-file with the exeption of cell $1 and $4. what i tried so far: awk '{for(i = 1; i<=NF; i++);if(i == 1 || i == 4);else print($i)}' file.csv ..any ideas how it work and why my example fails? Thanks in advance! IMPe (3 Replies)
Discussion started by: IMPe
3 Replies

4. UNIX for Dummies Questions & Answers

Grep and cat combined

Hello, i need to search one word (snp1) from many files and copy the content of the columns of this word in new file. example: file 1: SNP BP CHR P snp1 1 3 0.01 snp2 2 2 0.05 . . file 2: SNP BP CHR P snp1 1 3 0.06 snp2 2 2 0.3 output... (6 Replies)
Discussion started by: biopsy
6 Replies

5. Shell Programming and Scripting

awk or sed: change the color of a column w/o screwing up column spacing

Hey folks. I wrote a little awk script that summarizes /proc/net/dev info and then pipes it to the nix column command to set up column spacing appropriately. Here's some example output: Iface RxMBytes RxPackets RxErrs RxDrop TxMBytes TxPackets TxErrs TxDrop bond0 9 83830... (3 Replies)
Discussion started by: ryran
3 Replies

6. UNIX for Dummies Questions & Answers

awk for scientific notion and decimal combined data

Dea all, I have a question. I have a column of numbers with scientific notion and decimal combined data. I want to print it only if the number <0.05 using awk. however the very small numbers with scientific notion is not selected. Do any one know how to solve it? Thanks! example as below: ... (4 Replies)
Discussion started by: forevertl
4 Replies

7. Shell Programming and Scripting

Cat 114 files using grep or awk to pull muliple fields

Files xxxxxxx.txt ------------------------------------------------------------------------------------------------------------------------------------ Req.By: xxxxxxx WABUSH MINES - xxxxxx MINE (1001) Page: 1 Run on: 12/14/09... (4 Replies)
Discussion started by: sctxms
4 Replies

8. Shell Programming and Scripting

Merging of files with different headers to make combined headers file

Hi , I have a typical situation. I have 4 files and with different headers (number of headers is varible ). I need to make such a merged file which will have headers combined from all files (comman coluns should appear once only). For example - File 1 H1|H2|H3|H4 11|12|13|14 21|22|23|23... (1 Reply)
Discussion started by: marut_ashu
1 Replies

9. UNIX for Dummies Questions & Answers

Combined many txt files by columns

Hi, I want to combined 500 .txt files (each file only has one line) to a single txt file (A.txt). for example, A1.txt with 12 23 34 45 A2.txt with 56 67 78 89 ...etc, so I have A1.txt, A2.txt... , A499.txt, A500.txt (1 Reply)
Discussion started by: AMBER
1 Replies

10. Shell Programming and Scripting

Awk not printing the last combined column

nawk -F "|" 'FNR==NR {a=$2 OFS $3 OFS $4 OFS $5 OFS $6;next}\ {if ($5 in a)print $1,"test",$5,a, $2,$3,$4 OFS OFS OFS OFS OFS OFS OFS OFS $2-$3-$4 ; \ else print $1,"Database",$5 OFS OFS OFS OFS OFS OFS $2,$3,$4 OFS OFS OFS OFS OFS OFS OFS OFS $2-$3-$4 }' OFS="|" \ file1 file2 > file3 This... (5 Replies)
Discussion started by: pinnacle
5 Replies
Login or Register to Ask a Question