Extract data in tabular format from multiple files


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Extract data in tabular format from multiple files
# 1  
Old 04-25-2013
Extract data in tabular format from multiple files

Hi,

I have directory with multiple files from which i need to extract portion of specif lines and insert it in a new file, the new file will contain a separate columns for each file data.

Example:

I need to extract Value_1 & Value_3 from all files and insert in output file as below:

File 1 Content:
Code:
/2013/April/23 Value_1="10"
/2013/April/23 Value_2="20"
/2013/April/23 Value_3="30"

File 2 Content:
Code:
/2013/April/23 Value_1="40"
/2013/April/23 Value_2="50"
/2013/April/23 Value_3="60"

File 3 Content:
Code:
/2013/April/23 Value_1="70"
/2013/April/23 Value_2="80"
/2013/April/23 Value_3="90"

Required File output:
Code:
10,30
40,60
70,90


Can someone please advise how to achieve this? Thanks

Last edited by Franklin52; 04-25-2013 at 06:49 AM.. Reason: Please use code tags
# 2  
Old 04-25-2013
some thing like the following should work, adjust *.dat to provide a suitable file list:
Code:
$ for file in  *.dat ; do
       echo "$(grep 'Value_1' $file | cut -d\= -f2),$(grep 'Value_3' $file | cut -d\= -f2)" >>results.csv
done
$ cat results.csv
"10","30"
"40","60"
"70","90"


Last edited by Skrynesaver; 04-25-2013 at 05:48 AM.. Reason: quoting and providing paramters to grep and generally repairing off the top of my head code. removed holding variables
# 3  
Old 04-25-2013
Thanks for the reply, the problem that there are 50 values I need to extract in each file and there are 5000 files in total, I can't use grep cause it will be so slow. can we achieve this using sed and awk? The needed values are located in fixed line numbers in all files
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Merge and Sort tabular data from different text files

I have 42 text files; each containing up to 34 lines with following structure; file1 H-01 23 H-03 5 H-05 9 H-02 14 . . file2 H-01 17 H-02 43 H-04 7 H-05 8 H-03 7 . . file3 (6 Replies)
Discussion started by: Syeda Sumayya
6 Replies

2. Shell Programming and Scripting

Script to generate Excel file or to SQL output data to Excel format/tabular format

Hi , i am generating some data by firing sql query with connecting to the database by my solaris box. The below one should be the header line of my excel ,here its coming in separate row. TO_CHAR(C. CURR_EMP_NO ---------- --------------- LST_NM... (6 Replies)
Discussion started by: dani1234
6 Replies

3. Shell Programming and Scripting

Convert data to a tabular format

How can i convert the below data to a simpler format :- cat tabular.txt User 1 Details :- First Name = Tom Middle Name = Last Name = Hanks Age = 40 Address = User 2 details :- First Name = Mike Middle Name = Last Name = Tyson Age = 50 Address = (2 Replies)
Discussion started by: lazydev
2 Replies

4. Shell Programming and Scripting

Generate tabular data based on a column value from an existing data file

Hi, I have a data file with : 01/28/2012,1,1,98995 01/28/2012,1,2,7195 01/29/2012,1,1,98995 01/29/2012,1,2,7195 01/30/2012,1,1,98896 01/30/2012,1,2,7083 01/31/2012,1,1,98896 01/31/2012,1,2,7083 02/01/2012,1,1,98896 02/01/2012,1,2,7083 02/02/2012,1,1,98899 02/02/2012,1,2,7083 I... (1 Reply)
Discussion started by: himanish
1 Replies

5. UNIX for Dummies Questions & Answers

Extract common data out of multiple files

I am trying to extract common list of Organisms from different files For example I took 3 files and showed expected result. In real I have more than 1000 files. I am aware about the useful use of awk and grep but unaware in depth so need guidance regarding it. I want to use awk/ grep/ cut/... (7 Replies)
Discussion started by: macmath
7 Replies

6. UNIX for Dummies Questions & Answers

Using AWK: Extract data from multiple files and output to multiple new files

Hi, I'd like to process multiple files. For example: file1.txt file2.txt file3.txt Each file contains several lines of data. I want to extract a piece of data and output it to a new file. file1.txt ----> newfile1.txt file2.txt ----> newfile2.txt file3.txt ----> newfile3.txt Here is... (3 Replies)
Discussion started by: Liverpaul09
3 Replies

7. UNIX for Dummies Questions & Answers

AWK, extract data from multiple files

Hi, I'm using AWK to try to extract data from multiple files (*.txt). The script should look for a flag that occurs at a specific position in each file and it should return the data to the right of that flag. I should end up with one line for each file, each containing 3 columns:... (8 Replies)
Discussion started by: Liverpaul09
8 Replies

8. UNIX for Dummies Questions & Answers

converting a tabular format data to comma seperated data in KSH

Hi, Could anyone help me in changing a tabular format output to comma seperated file pls in K-sh. Its very urgent. E.g : username empid ------------------------ sri 123 to username,empid sri,123 Thanks, Hema:confused: (2 Replies)
Discussion started by: Hemamalini
2 Replies

9. Shell Programming and Scripting

extract specific data from xml format file.

Hi, I need to extract the start time value (bold, red font) under the '<LogEvent ID="Timer Start">' tag (black bold) from a file with the following pattern. There are other LogEventIDs listed in the file as well, making it harder for me to extract out the specific start time that I need. . .... (7 Replies)
Discussion started by: 60doses
7 Replies

10. Shell Programming and Scripting

Anyways to find sentences with data format and extract it???

Hi guys,i got this problem which is..i need to find those sentences with date inside and extract them out,the input is somehow like this eg: $DATA42.GANTRY2.GA161147 DISKFILE 2007-10-16 11:56:45 SUPER.OPR \NETS.$Y4CB.#IN ... (4 Replies)
Discussion started by: cyberray
4 Replies
Login or Register to Ask a Question