Sponsored Content
Top Forums Shell Programming and Scripting For loop and read from different directories Post 303001224 by drl on Sunday 30th of July 2017 02:49:47 PM
Old 07-30-2017
Hi.

Apologies for the lengthy post.

This is probably not an improvement as much as it is a meta-answer. This is how we went about solving problems like this by generalizing.

I'll assume that you could get the contents of the directories by looking at the content, and one such command might be:
Code:
ls -1 A > data1

and so on for B, C, etc.

We then would use a local code gather to obtain at least one line from each of the files data1, data2, etc.

The companion program is scatter.

As I have noted before we have not yet decided to publish our codes, but when we post meta-solutions like this, we can post the documentation. Then folks can decide whether this is a reasonable approach for them to pursue.

Here is a demonstration that exercises code gather:
Code:
 
#!/usr/bin/env bash

# @(#) s1       Demonstrate intersperse, shuffle of lines, gather.

# Utility functions: print-as-echo, print-line-with-visual-space, debug.
# export PATH="/usr/local/bin:/usr/bin:/bin"
LC_ALL=C ; LANG=C ; export LC_ALL LANG
pe() { for _i;do printf "%s" "$_i";done; printf "\n"; }
pl() { pe;pe "-----" ;pe "$*"; }
em() { pe "$*" >&2 ; }
db() { ( printf " db, ";for _i;do printf "%s" "$_i";done;printf "\n" ) >&2 ; }
db() { : ; }
C=$HOME/bin/context && [ -f $C ] && $C gather dixf scatter

FILE=${1-data1}

pl " Input data files data*:"
head data*

pl " Results, 1 item from each list:"
gather data*

pl " Results, 2 from column A, one from B and C:"
gather data1:2 data2 data3

pl " Results, as previous, but separator is newline:"
gather -s '\n' data1:2 data2 data3

pl " Help from gather:"
gather -h

pl " Details about gather, scatter:"
dixf gather scatter

exit 0

producing:
Code:
$ ./s1
Environment: LC_ALL = C, LANG = C
(Versions displayed with local utility "version")
OS, ker|rel, machine: Linux, 3.16.0-4-amd64, x86_64
Distribution        : Debian 8.8 (jessie) 
bash GNU bash 4.3.30
gather (local) 1.1
dixf (local) 1.49
scatter (local) 1.4

-----
 Input data files data*:
==> data1 <==
1.txt
2.txt
3.txt
4.txt
5.txt

==> data2 <==
a.txt
b.txt
c.txt
d.txt
e.txt

==> data3 <==
101.txt
102.txt
103.txt
104.txt
105.txt

-----
 Results, 1 item from each list:
1.txt
a.txt
101.txt
2.txt
b.txt
102.txt
3.txt
c.txt
103.txt
4.txt
d.txt
104.txt
5.txt
e.txt
105.txt

-----
 Results, 2 from column A, one from B and C:
1.txt 2.txt
a.txt
101.txt
3.txt 4.txt
b.txt
102.txt
5.txt 103.txt
c.txt
104.txt
d.txt
105.txt
e.txt

-----
 Results, as previous, but separator is newline:
1.txt
2.txt
a.txt
101.txt
3.txt
4.txt
b.txt
102.txt
5.txt
103.txt
c.txt
104.txt
d.txt
105.txt
e.txt

-----
 Help from gather:

gather - Read, shuffle, weave, intersperse lines from multiple inputs to STDOUT.

usage: gather [options] -- [files]

options:

--separator=SEP
  Set output token separator to SEP, default " ", and "\n" is
  accepted as NEWLINE.  The SEP is used when more than one line is
  read from a file. This allows one to easily capture short lines
  into a single output line.

--help (or -h)
  print this message and quit.

[files]
  filename1:count1 filename2:count2 ... filename3:count3
  Each filenamei will be read in succession, and will be written
  to standard output, continuing until EOF on every file.


-----
 Details about gather, scatter:

gather	Read, shuffle, weave, intersperse lines from multiple input files. (what)
Path    : ~/bin/gather
Version : 1.1
Length  : 224 lines
Type    : Perl script, ASCII text executable
Shebang : #!/usr/bin/env perl
Help    : probably available with -h,--help
Modules : (for perl codes)
 strict	1.08
 warnings	1.23
 English	1.09
 Carp	1.3301
 Getopt::Long	2.42
 feature	1.36_01

scatter	Write, deal, unravel disperse lines to multiple output files. (what)
Path    : ~/bin/scatter
Version : 1.4
Length  : 190 lines
Type    : Perl script, ASCII text executable
Shebang : #!/usr/bin/env perl
Modules : (for perl codes)
 strict	1.08
 warnings	1.23
 English	1.09
 Carp	1.3301
 Data::Dumper	2.151_01
 Getopt::Long	2.42
 feature	1.36_01

You can also, with the right shell, use embedded commands like this:
Code:
$ gather <( ls -1 A ) <( ls -1 B )
1.txt
a.txt
2.txt
b.txt
3.txt
c.txt
4.txt
d.txt
5.txt
e.txt

Best wishes ... cheers, drl
This User Gave Thanks to drl For This Post:
 

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Script to loop through all files and directories.

I'm trying to write a script that will loop through all files and directories down from a path I give it, and change the permissions and ACL. I was able to do the obvious way and change the files and folders on the same level as teh path...but I need it to continue on deeper into the file... (2 Replies)
Discussion started by: cheetobandito
2 Replies

2. Shell Programming and Scripting

How to loop through directories to touch files

Hi, Please help me on this. Suppose i have the following directory structure. /app/data /app/data/eng /app/data/med /app/data/bsc each of the directories data,data/eng,data/med,data/bsc holds files with date extension like a.20081230 b.20081230 and so on I need a script to loop... (9 Replies)
Discussion started by: sussane
9 Replies

3. Shell Programming and Scripting

Newbie: How to loop round sub-directories

I want a bit of shell script that will let me loop round all the sub-directories in a directory (i.e. ignoring any ordinary files in that directory). Let's say I just want to echo the names of the sub-directories. This sounds like it should be pretty easy - but not for me, it isn't! All help... (4 Replies)
Discussion started by: cjhancock
4 Replies

4. Shell Programming and Scripting

Loop to move files in different directories

Hi, I have various log files in different paths. e.g. a/b/c/d/e/server.log a/b/c/d/f/server.log a/b/c/d/g/server.log a/b/c/h/e/server.log a/b/c/h/f/server.log a/b/c/h/g/server.log a/b/c/i/e/server.log a/b/c/i/e/server.log a/b/c/i/e/server.log and above these have an archive folder... (6 Replies)
Discussion started by: acc01
6 Replies

5. Shell Programming and Scripting

For Loop Range Create Directories

Hello, I am a bit stumped on this. I am attempting to create 24 empty directories with a loop. Seems like I have incorrect syntax. When I run the following command I get the error below. Command $ for i in {2..24}; do mkdir $i_MAY_2011 ; doneError x 24 mkdir: missing operand Try `mkdir... (2 Replies)
Discussion started by: jaysunn
2 Replies

6. Shell Programming and Scripting

To read directories and file

Hi ; I want to write a shell script to read all files and directories(recursively) given in path along with their user permissions and store that result in one file as File path Userpermissions ===== =========== I m new to linux and my dont kno much abt shell scripting. I will... (5 Replies)
Discussion started by: ajaypadvi
5 Replies

7. UNIX for Dummies Questions & Answers

Read statement within while read loop

hi, this is my script #!/bin/ksh cat temp_file.dat | while read line do read test if ]; then break else echo "ERROR" fi done when i execute this code , the script does wait for the user input . it directly prints "ERROR" and terminates after the no. of times as there... (3 Replies)
Discussion started by: siva1612
3 Replies

8. Shell Programming and Scripting

Accessing multiple directories in loop

Hi Guys, I need to access multiple directories whcih is following similar structure and need to copy those files in desitination path. for eg : if ] then cd ${DIR}/Mon/loaded echo "copying files to $GRS_DIR" cp * ${DIR}/Mon/ echo "Files of Monday are Copied" fi if ] then... (5 Replies)
Discussion started by: rohit_shinez
5 Replies

9. Solaris

Giving read write permission to user for specific directories and sub directories.

I have searched this quite a long time but couldn't find the right method for me to use. I need to assign read write permission to the user for specific directories and it's sub directories and files. I do not want to use ACL. This is for Solaris. Please help. (1 Reply)
Discussion started by: blinkingdan
1 Replies
All times are GMT -4. The time now is 06:51 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy