Sponsored Content
Full Discussion: Help reformatting column
Top Forums Shell Programming and Scripting Help reformatting column Post 302991012 by torchij on Sunday 5th of February 2017 02:30:03 PM
Old 02-05-2017
Help reformatting column

Hello UNIX experts,

I'm stumped finding a method to reformat a column. Input file is a two column tab-delimited file. Essentially, for every term that appears in column 2, I would like to summarize whether that term appears for every entry in column 1. In other words, make a header for each term that appears in column 2, and then mark whether that appears for each entry in column 1. Somewhat complicated to explain, so I'll give an example:

Code:
$ cat input.txt
sample1	workflow1;workflow2;workflow3;workflow4;workflow5
sample2	workflow2;workflow3;workflow1;workflow5
sample3	workflow3;workflow1;workflow4;workflow5;workflow2
sample4	workflow3;workflow1;workflow4;workflow5;workflow2
sample5	workflow3;workflow1;workflow4;workflow5;workflow2
sample6	workflow1
sample7	workflow8

Code:
$ cat output.txt
Sample	workflow1	workflow2	workflow3	workflow4	workflow5	workflow8	
sample1	X	X	X	X	X	NA
sample2	X	X	X	NA	X	NA
sample3	X	X	X	X	X	NA
sample4	X	X	X	X	X	NA
sample5	X	X	X	X	X	NA
sample6	X	NA	NA	NA	NA	NA
sample7	NA	NA	NA	NA	NA	X

# note: X is "yes, this workflow exists for this sample"

The values "workflow1", "workflow2" etc...may contain special characters such as underscores, hyphens, colons etc... (ie. workflow-1, work:flow1), so matches must be exact. I wrote a working bash code, but it is terribly slow and poorly written. Right now, a 2,000 line file takes 20 minutes because my code sucks and runs through loops! For reference, my strategy:

Code:
# for each sample and workflow in $file, check whether workflow exists by through grep, then output a file with X or NA.
for SAMPLE in `cat $file.samples`; # column 1 from $file
 do
  echo "Preparing $SAMPLE"
  for WORKFLOW in `cat $file.workflows`; # all the unique terms from column 2
   do
    CHECK=`grep -w "$SAMPLE" $file | grep -oP ";${WORKFLOW};" | sed "s/\;//g"`;
    if [ "${CHECK}" == "${WORKFLOW}" ]; then
     echo "X" > $file.$SAMPLE.$WORKFLOW.reply; else echo "NA" >$file.$SAMPLE.$WORKFLOW.reply
    fi
   done;
  echo $SAMPLE > $file.sample;
  paste $file.sample $file.$SAMPLE.*.reply >> $file.sum.txt;
  rm $file.sample $file.$SAMPLE.*.reply;
 done

Many many thanks in advance!!

Torch
 

10 More Discussions You Might Find Interesting

1. Filesystems, Disks and Memory

reformatting a floppy!

i am trying to reformat a floppy i am using solaris 9 when i run this: rmformat -F quick /vol/dev/aliases/floppy0 it tells me that it cannot perform the operation on a mounted device. how do i unmount the device and format the floppy? (1 Reply)
Discussion started by: rmuhammad
1 Replies

2. UNIX for Dummies Questions & Answers

Reformatting file

Hi, How can I reformat a file (text file) using unix command. This file was FTP'd from Mainframe and contains some garbage character at the end of each line. Each line contains special characters '<soh>' at the end which should have been spaces when I view it in emacs or nedit. I couldnt do find... (2 Replies)
Discussion started by: mrjunsy
2 Replies

3. Shell Programming and Scripting

Help reformatting output

I have a command that gives me the output below: JAVA_HOME = C:/jdk1.5.0_11 Broker Performance Report for server 'app1' RMI_URL = rmis:// Parameter Kintana ItgDS DashboardDS ---------------------------- ------- ----- ----------- Connections count 41 ... (4 Replies)
Discussion started by: bwiebe
4 Replies

4. UNIX for Dummies Questions & Answers

Date reformatting

I currently have the following file containing sample values for a number of dates: Loc1 04 Jan 2007 0.95 0.9532 Loc1 05 Jan 2007 0.95 0.9513 Loc1 06 Jan 2007 0.95 0.9535 This continues for all months of the year and spans across several years. I am trying to reformat the dates so that... (2 Replies)
Discussion started by: figaro
2 Replies

5. UNIX for Dummies Questions & Answers

Date reformatting

I have a file with temperature measurements: Loc1,20090102,71.55 Loc1,20090103,71.65 Loc1,20090104,71.55 Loc1,20090105,71.54 Loc1,20090106,71.54 However, to load this into a database I would like to reformat the dates (column 2) from the yyyymmdd format to the yyyy-mm-dd format. I have... (2 Replies)
Discussion started by: figaro
2 Replies

6. Shell Programming and Scripting

Reformatting single column text file starting new line when finding particular string

Hi, I have a single colum file and I need to reformat the file so that it creates a new line every time it come to an IP address and the following lines are corresponding rows until it comes to the next IP address. I want to turn this 172.xx.xx.xx gwpusprdrp02_pv seinwnprd03... (7 Replies)
Discussion started by: kieranfoley
7 Replies

7. UNIX for Dummies Questions & Answers

Date reformatting

I have been reformatting dates from a data file to make them mysql compliant. 31-10-2011 Loc1 1-11-2011 Loc2 The first can be captured by this: sed -i '' -e "s#\(..\)-\(..\)-\(....\)#\3-\2-\1#" data.txt and leads to: 2011-10-31 Loc1 The second line is captured as follows: sed -i... (2 Replies)
Discussion started by: figaro
2 Replies

8. Shell Programming and Scripting

Reformatting Column into rows

I have a file that I need to reformat so that every time I match a certain string in the first column it prints to the string as the heading and under the sting it prints the remaining entries on the line that matched the string. For example, I need to reformat this xxx : yyy zzz 11 : 111 222... (4 Replies)
Discussion started by: kieranfoley
4 Replies

9. UNIX for Dummies Questions & Answers

Help reformatting input file

Hi, I have an input file that looks like this (columns are tab delimited: Data000005-RA GO:0003735 GO:0005840 GO:0006412 Data000005-RA GO:0003735 Data000009-RA GO:0003735 GO:0005622 GO:0005840 GO:0006412 ... (2 Replies)
Discussion started by: Fahmida
2 Replies

10. UNIX for Dummies Questions & Answers

Need help in reformatting the input

Hi i want to print line which is mentioned as below 615213:1;20150725;20250722;0|11;20150831;20150831;100|14;20150725;20160723;2 in below format. ' 615213: 1;20150725;20250722;0 615213: 11;20150831;20150831;100 615213: 14;20150725;20160723;2 please help me and suggest me how to... (9 Replies)
Discussion started by: scriptor
9 Replies
pam_sample(5)															     pam_sample(5)

NAME
pam_sample - a sample PAM module SYNOPSIS
/usr/lib/security/pam_sample.so.1 The SAMPLE service module for PAM is divided into four components: authentication, account management, password management, and session management. The sample module is a shared object that is dynamically loaded to provide the necessary functionality. SAMPLE Authentication Component The SAMPLE authentication module provides functions to test the PAM framework functionality using the pam_sm_authenticate(3PAM) call. The SAMPLE module implementation of the pam_sm_authenticate(3PAM) function compares the user entered password with the password set in the pam.conf(4) file, or the string "test" if a default test password has not been set. The following options can be passed in to the SAMPLE Authentication module: debug Syslog debugging information at the LOG_DEBUG level. passwd=newone Sets the password to be "newone." first_pass_good The first password is always good when used with the use_first_pass or try_first_pass option. first_pass_bad The first password is always bad when used with the use_first_pass or try_first_pass option. always_fail Always returns PAM_AUTH_ERR. always_succeed Always returns PAM_SUCCESS. always_ignore Always returns PAM_IGNORE. use_first_pass Use the user's initial password (entered when the user is authenticated to the first authentication module in the stack) to authenticate with the SAMPLE module. If the passwords do not match, or if this is the first authentica- tion module in the stack, quit and do not prompt the user for a password. It is recommended that this option only be used if the SAMPLE authentication module is designated as optional in the pam.conf configuration file. try_first_pass Use the user's initial password (entered when the user is authenticated to the first authentication module in the stack) to authenticate with the SAMPLE module. If the passwords do not match, or if this is the first authentica- tion module in the stack, prompt the user for a password. The SAMPLE module pam_sm_setcred(3PAM) function always returns PAM_SUCCESS. SAMPLE Account Management Component The SAMPLE Account Management Component implements a simple access control scheme that limits machine access to a list of authorized users. The list of authorized users is supplied as option arguments to the entry for the SAMPLE account management PAM module in the pam.conf file. Note that the module always permits access to the root super user. The option field syntax to limit access is shown below: allow= name[,name] allow= name [allow=name] The example pam.conf show below permits only larry to login directly. rlogin is allowed only for don and larry. Once a user is logged in, the user can use su if the user are sam or eric. login account require pam_sample.so.1 allow=larry dtlogin account require pam_sample.so.1 allow=larry rlogin account require pam_sample.so.1 allow=don allow=larry su account require pam_sample.so.1 allow=sam,eric The debug and nowarn options are also supported. SAMPLE Password Management Component The SAMPLE Password Management Component function ( pam_sm_chauthtok(3PAM)), always returns PAM_SUCCESS. SAMPLE Session Management Component The SAMPLE Session Management Component functions ( pam_sm_open_session(3PAM), pam_sm_close_session(3PAM)) always return PAM_SUCCESS. See attributes(5) for description of the following attributes: +-----------------------------+-----------------------------+ | ATTRIBUTE TYPE | ATTRIBUTE VALUE | |MT Level |MT-Safe with exceptions | +-----------------------------+-----------------------------+ pam(3PAM), pam_sm_authenticate(3PAM), pam_sm_chauthtok(3PAM), pam_sm_close_session(3PAM), pam_sm_open_session(3PAM), pam_sm_setcred(3PAM), libpam(3LIB), pam.conf(4), attributes(5) This module should never be used outside of a closed debug environment. The examples of the use_first_pass and try_first_pass options are obsolete for all other Solaris delivered PAM service modules The interfaces in libpam() are MT-Safe only if each thread within the multi-threaded application uses its own PAM handle. 16 Aug 2005 pam_sample(5)
All times are GMT -4. The time now is 04:31 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy