Sponsored Content
Top Forums Shell Programming and Scripting Creating matrix from folders and subfolders Post 302746013 by Chubler_XL on Tuesday 18th of December 2012 03:36:08 PM
Old 12-18-2012
As you only have 83 subfolders gawk should be able to keep all the files open as it works (ulimit -n controls how many files gawk can have open) this will reduce the memory requirements from tens of Gb to a few Kb (and you should get output pretty much straight away).

Many traditional awk have a 15 open file limit and if you only have this awk your out of luck with this solution.

Code:
find index data -type f -print | awk -F/ '
/^index/ { F[++files]=$NF }
/^data/  { S[$(NF-1)] }
END {
    printf "\t"
    for(subdir in S) printf "\t%s",subdir;
    printf "\n"
    for(i=1;i<=files;i++) {
        while((getline < ("index/"F[i])) > 0) {
            printf "%s\t%s",F[i],$0
            for(subdir in S) {
                getline < ("data/"subdir"/"F[i])
                printf "\t%s",$0
            }
            printf "\n"
        }
        close(F[i])
        for(subdir in S) close("data/"subdir"/"F[i]);
    }
}'


Last edited by Chubler_XL; 12-18-2012 at 05:09 PM.. Reason: Fix indents + remove debug line
This User Gave Thanks to Chubler_XL For This Post:
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to create folders/subfolders using shellscript

Hi, Can any one help me how to create folders using shellscript.My requirement is: FolderName: Main/Main1 :Main/Main2 :Main/Main3 underSubFolder : Main1/A :Main1/B :Main1/C underSubfolder: A/A1 ... (2 Replies)
Discussion started by: ram2s2001
2 Replies

2. Shell Programming and Scripting

Script to Analyze folders and subfolders

I would like to know if there is a script out there that someone may have already written that I can use to analyze folders and sub folders on my AIX system. It can be in perl or a basic korn script. Thanks in advance. (7 Replies)
Discussion started by: seacros
7 Replies

3. UNIX for Advanced & Expert Users

find size of folders and its subfolders with the Owner details

HI, I have the following command that shows me the total size of folders and subfolders : du -hs *| sort -n result: 1.0M sandeep 1.4G sandy 1.4M important 1.6M files but I will need to know the size of folders and its subfolders( not size of individual files though)... (5 Replies)
Discussion started by: bsandeep_80
5 Replies

4. Windows & DOS: Issues & Discussions

Copy folders and subfolders from unix to windows

Sir From a unix machine some folders and their folders have to be copied to windows XP PC. Please help me with a batch file or a shell script. I am new to the the shell and batch files. Thanks in anticipation. sastry (3 Replies)
Discussion started by: chssastry
3 Replies

5. Shell Programming and Scripting

Search and Replace text in folders and Subfolders

Hi, I need help in writing a script to search a particular text in multiple files present in folders and sub folders and replace it with another string which also has special characters like '&', '|', etc.. I know sed command will be used to replace the text but i'm not sure how to use it for... (5 Replies)
Discussion started by: Asheesh
5 Replies

6. Shell Programming and Scripting

Help to move folders, subfolders and files from unix to windows

Hi Unix Gurus, I am able to copy only files that exist in the parent folder. My parent folder has sub folders and within sub folders there are lots files. I need to copy folder, sub folders and files from Unix to the remote windows SFTP location. The directory structure is something like... (1 Reply)
Discussion started by: shankar1dada
1 Replies

7. Ubuntu

Creating Matrix

Hi all, I'm a newbie in shell scripting and currently I'm trying to create a matrix using bash. The Output will look like this AB CDE FG 1 2 3 4 5 6 7 I'm stuck on the ABCDEFG display. printFlightSeats() { rows=7 columns=7 for ((i=0;i<=$rows;i++)) do (0 Replies)
Discussion started by: vinzping
0 Replies

8. Shell Programming and Scripting

List all the files in the present path and Folders and subfolders files also

Hi, I need a script/command to list out all the files in current path and also the files in folder and subfolders. Ex: My files are like below $ ls -lrt total 8 -rw-r--r-- 1 abc users 419 May 25 10:27 abcd.xml drwxr-xr-x 3 abc users 4096 May 25 10:28 TEST $ Under TEST, there are... (2 Replies)
Discussion started by: divya bandipotu
2 Replies

9. Shell Programming and Scripting

Move specific folders and subfolders in a directory

I am trying to move specific folders and subfolders within a directory using the below. I can see the folders to move and they are at the location, but I am getting an error. Thank you :). mv -v /home/cmccabe/Desktop/NGS/API/6-10-2016{bam/{validation,coverage},bedtools /media/cmccabe/"My... (6 Replies)
Discussion started by: cmccabe
6 Replies

10. UNIX for Beginners Questions & Answers

Command to change add permissions for a new user to all files in all subfolders and folders

Hi there! I'm new to Unix and haven't done command line stuff since MS-Dos and Turbo Pascal (hah!), I would love some help figuring out this basic command (what I assume is basic). I'd like to add a User to the permissions of all files in a folder and all files in all subfolders, as well... (9 Replies)
Discussion started by: Janjbrt
9 Replies
IGAWK(1)							 Utility Commands							  IGAWK(1)

NAME
igawk - gawk with include files SYNOPSIS
igawk [ all gawk options ] -f program-file [ -- ] file ... igawk [ all gawk options ] [ -- ] program-text file ... DESCRIPTION
Igawk is a simple shell script that adds the ability to have ``include files'' to gawk(1). AWK programs for igawk are the same as for gawk, except that, in addition, you may have lines like @include getopt.awk in your program to include the file getopt.awk from either the current directory or one of the other directories in the search path. OPTIONS
See gawk(1) for a full description of the AWK language and the options that gawk supports. EXAMPLES
cat << EOF > test.awk @include getopt.awk BEGIN { while (getopt(ARGC, ARGV, "am:q") != -1) ... } EOF igawk -f test.awk SEE ALSO
gawk(1) Effective AWK Programming, Edition 1.0, published by the Free Software Foundation, 1995. AUTHOR
Arnold Robbins (arnold@skeeve.com). Free Software Foundation Nov 3 1999 IGAWK(1)
All times are GMT -4. The time now is 08:56 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy