Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Content of filtered files write as columns into one file

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Content of filtered files write as columns into one file
# 1  
Old 05-20-2015
Content of filtered files write as columns into one file
Hi everybody!

i have a lot of files where i filter out data.
Code:
#!/bin/bash
f=sample_*.Spe
for i in $f `eval echo ls sample_*.Spe`
do
        if test -f "$i" 
        then
awk 'FNR==8 ||FNR==10 || (FNR>=13 && FNR<=268) {print $1}' $i > test$i.txt
paste test$i.txt test_f.txt > test_f.txt
        fi
done

what i have: several files where i filter out the first column:

what i want to have is --> write the filtered columns into one file - as a table
Code:
001             002               003     ...
05/13/2015   05/13/2015    05/13/2015
600              600              600
4                  8                 1
5                  6                 4
6                  4                 2
7                  2                 3
.                  .                 .
.                  .                 .
.                  .                 .

as you can see in the top example, i try to do it with paste, but that will not work propper. moreover i want a part of the filename in the top of every column
Code:
echo $i | cut -c 8-10

I dont have experience with awk-arrays - can someone help me to solve this in a reliable working script?

thanks in advance,
IMPe
# 2  
Old 05-20-2015
There are enough strange constructs in your sample shell script that I am not at all sure what you are trying to do, what your input files look like, nor what your output file is supposed to look like. It seems to be a very expensive way of copying the last regular file it processes to the file named test_f.txt.

Please show us three short sample input files (including their full filenames; not just the pattern that matches them), and show us the exact output you want to produce from those three sample input files. And, it is always helpful to know what operating system (including its release/version information) you're using and the version of the shell you're using.
This User Gave Thanks to Don Cragun For This Post:
IMPe
# 3  
Old 05-21-2015
Quote:
Originally Posted by Don Cragun
There are enough strange constructs in your sample shell script that I am not at all sure what you are trying to do, what your input files look like, nor what your output file is supposed to look like. It seems to be a very expensive way of copying the last regular file it processes to the file named test_f.txt.

Please show us three short sample input files (including their full filenames; not just the pattern that matches them), and show us the exact output you want to produce from those three sample input files. And, it is always helpful to know what operating system (including its release/version information) you're using and the version of the shell you're using.
Hi Don!

Sorry for being not accurate!
OS: Linux ass210 3.16.7-13-desktop #1 SMP PREEMPT Wed Mar 18 17:31:15 UTC 2015 (ba2afab) x86_64 x86_64 x86_64 GNU/Linux
Shell: GNU bash, version 4.2.53(1)-release (x86_64-suse-linux-gnu)
input file [sample_001.Spe]. Up to 40 files with the subsequent structure and ascending number after the underscore [ sample_001.Spe, sample_002.Spe ... sample_040.Spe] .
structure of the input file: i haven't put all the 255 lines; i stretched the main part, but all of them are numbers. moreover i marked the for me important parts blue.
Code:
$SPEC_ID:
No sample description was entered.
$SPEC_REM:
DET# 1
DETDESC# GBS178 Model 927 SN 9244437 Input 1
AP# Maestro Version 7.01
$DATE_MEA:
05/13/2015 09:21:27
$MEAS_TIM:
600 600
$DATA:
0 255
       0
       0
       0
       0
       0
.
.
.
      1
       0
       0
       0
       0
       0
       0
       0
       0
       0
       0
$ROI:
1
57 105
$PRESETS:
Live Time
600
0
$ENER_FIT:
-1.172989 0.081834
$MCA_CAL:
3
-1.172989E+000 8.183417E-002 0.000000E+000 keV
$SHAPE_CAL:
3
1.428736E+001 0.000000E+000 0.000000E+000

The for me interesting data i want to get out of each file by using
Code:
f=sample_*.Spe
for i in $f `eval echo ls sample_*.Spe`
do
        if test -f "$i" 
        then
awk 'FNR==8 ||FNR==10 || (FNR>=13 && FNR<=268) {print $1}' $i > test$i.txt
...

and put the extracted columns (i.e. 40 columns from 40 files) into one file.
furthermore the output file should have a csv-format to transfer it to M$-Excel. It should look like the following example; the 3 dots in every column and the 3 dots in every line should represent the vertical and horizontal ellipsis :
Code:
001           002           003          ...  040
05/13/2015    05/13/2015    05/13/2015   ...  05/13/2015
600           600           600          ...  600
4             8             1            ...  2
5             6             4            ...  3
6             4             2            ...  5
7             2             3            ...  7
.             .             .            ...  .
.             .             .            ...  .
.             .             .            ...  .
3             4             5            ...  9

can you please help me.

Thanks in advance,
IMPe
# 4  
Old 05-21-2015
You might want to try this instead:
Code:
#!/bin/bash
awk '
FNR == 1 {
	close(fn)
	seq = substr(FILENAME, 8, 3)
	fn = "test" FILENAME ".txt"
	printf("%-13s\n", seq) > fn
}
FNR == 8 || FNR == 10 || (FNR >= 13 && FNR <= 268) {
	printf("%-13s\n", $0) > fn
}' sample_*.Spe
paste -d ' ' testsample_*.Spe.txt > test_f.txt

It invokes awk once instead of 80 times, creates each of your testsample_*.Spe.txt files once instead of twice, invokes paste once instead of 80 times, and doesn't redirect the output of paste to one of its input files (which wiped out most of the work you were trying to do).

This assumes that your data doesn't have any fields wider than 13 characters and that you want data left aligned in each column (which matches the 1st two columns of output in your second sample output file.
This User Gave Thanks to Don Cragun For This Post:
IMPe
# 5  
Old 05-23-2015
Hi Don,

thanks a lot for solving my problem.
Unfortunatelly i don't understand your
script completely but i can work with it.
It gives me what i want propper and efficiently!

Thanks a lot!!
IMPe
# 6  
Old 05-23-2015
You're welcome.
I'm glad it is working for you.
What don't you understand about how it works?
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Counting number of times content in columns occurs in other files

I need to figure out how many times a location (columns 1 and 2) is present within a group of files. I figured using a combination of 'while read' and 'grep' I could count the number of instances but its not working for me. cat file.txt | while read line do grep $line *08-new.txt | wc -l... (6 Replies)
Discussion started by: ncwxpanther
6 Replies

2. Shell Programming and Scripting

Content should be filtered within brackets

Hello all, I have a string and would like to extract the content of the text within the brackets. Here is the string: Desc="file from to line" What I would like to have is the following: Filename="4009821_737498.out" FromLine=12965 ToLine=12355 Maybe I have to do this with... (6 Replies)
Discussion started by: API
6 Replies

3. Shell Programming and Scripting

Import 2 columns from 8 .csv files into pandas df (side by side) and write a new csv

I have 8 .csv files with 16 columns and "n" rows with no Header. I want to parse each of these .csv and get column and put the data into a new.csv. Once this is done, the new.csv should have 16 columns (2 from each input.csv) and "n" rows. Now, I want to just take the average of Column from... (3 Replies)
Discussion started by: Zam_1234
3 Replies

4. Shell Programming and Scripting

search for content in files. Name of files is in another file. Format as report.

Hi I have multiple files in a folder and one file which contains a list of files (one on each line). I was to search for a string only within these files and not the whole folder. I need the output to be in the form File1<tab>string instance 2<tab> string instance 2<tab>string instance 3... (6 Replies)
Discussion started by: pkabali
6 Replies

5. Shell Programming and Scripting

"Search for files in a file and write these files to another file which sould be in some other direc

Hi , Need to shell script to extracts files names and write those to another file in different directory. input file is inputfile.txt abc|1|bcd.dat 123 david123 123 rudy2345 124 tinku5634 abc|1|def.dat 123 jevid123 123 qwer2345 124 ghjlk5634 abc|1|pqr.txt 123 vbjnnjh435 123 jggdy876... (1 Reply)
Discussion started by: dssyadav
1 Replies

6. Shell Programming and Scripting

How can I write the specific content in the file through shell script

Hello, I need to do one thing that my script creates the file touch release.SPLASH_12_03_00_RC01.txt Now I want to update that file with some content e.g splashbuild::SPLASH_12_17_00_RC02.zip Thanks (1 Reply)
Discussion started by: anuragpgtgerman
1 Replies

7. Shell Programming and Scripting

split content and write to new record

Hi, Help required to split record value and write to new row. Input a~b~c~value in ('3','4','5')~test output a~b~c~3~test a~b~c~4~test a~b~c~5~test input a~b~c~value in ('3','4')~test output a~b~c~3~test a~b~c~4~test (8 Replies)
Discussion started by: Jairaj
8 Replies

8. Shell Programming and Scripting

How to replace some content of a file and write with a new name

Hi I have a control file which looks like this LOAD DATA INFILE '/home/scott/XXX.dat' PRESERVE BLANKS ............. ............. how can i change the content of this file and replace the file in the second line with anothe file name and write it back with another name to the disk? ... (5 Replies)
Discussion started by: mwrg
5 Replies

9. Shell Programming and Scripting

Mailing files with, ls filtered with grep

Hi guys Im new to forum so please dont be hard to me if I make any mistakes :) I want to the following task: 1. I have a file lets say file1, which contains job names with numbers which have failed to start and .... I can sort that file into another to get only the Job numbers with... (3 Replies)
Discussion started by: kl1ngac1k
3 Replies

10. UNIX for Dummies Questions & Answers

sorting file content on columns

guys i have a question: i'd like to sort files (as many I want) in columns so to visualize them one near the other...so let's say i have just 2 files: FILE1 John Mary Bridget FILE2 Anne Robert Mark i would like to obtain: John Anne Mary Robert Bridget ... (2 Replies)
Discussion started by: marshmallow
2 Replies
Login or Register to Ask a Question