Sponsored Content
Top Forums Shell Programming and Scripting Grep -c text processing of a log file Post 302956712 by sea on Friday 2nd of October 2015 08:37:36 AM
Old 10-02-2015
Try:
Code:
while read _ _ _ _ _ _ _ JOBID
do
	count=$(grep "$JOBID" "./LOGFILE"|wc -l)
	[ $count -eq 2 ] && \
		echo "Job $JOBID: All good!" || \
		echo "Job $JOBID: Invalid amount of job entries ($count)..."
done<LOGFILE

hth
This User Gave Thanks to sea For This Post:
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Processing a text file

A file contains one name per line, such as: john doe jack bruce nancy smith sam riley When I 'cat' the file, the white space is treated as a new line. For example list=`(cat /path/to/file.txt)` for items in $list do echo $items done I get: john doe (1 Reply)
Discussion started by: TheCrunge
1 Replies

2. UNIX for Dummies Questions & Answers

text file processing

Hello! There is a text file, that contains hierarchy of menues, like: Aaaaa->Bbbbb Aaaaa->Cccc Aaaaa-> {spaces} Ddddd (it means that the full path is Aaaaa->Cccc->Ddddd ) Aaaaa-> {more spaces} Eeeee (it means that the full path is Aaaaa->Cccc->Ddddd->Eeeee ) Fffffff->Ggggg... (1 Reply)
Discussion started by: alias47
1 Replies

3. Shell Programming and Scripting

KSH script -text file processing NULL issues

I'm trying to strip any garbage that may be at the end of my text file and that part is working. The problem only seems to be with the really long lines in the file. When the head command is executed I am directing the output to a new file. The new file always get a null in the 4096 position but... (2 Replies)
Discussion started by: geauxsaints
2 Replies

4. Shell Programming and Scripting

processing file names using text files

Hi, I have to perform an iterative function on a set of 10 files. After the first round the output files are named differently than the input files. examples input file name = xxxx1.yyy output file name = xxxx1_0001.yyy I need to rename all of the output files to the original input... (5 Replies)
Discussion started by: ligander
5 Replies

5. UNIX for Advanced & Expert Users

perl text file processing using hash

Hi Experts, I have this requirement to process large files (200MB+).Format of files is like: recordstart val1 1 val2 2 val3 4 recordstart val1 5 val2 6 val3 1 val4 1 recordstart val1 ... (4 Replies)
Discussion started by: mtomar
4 Replies

6. Shell Programming and Scripting

Text processing of file

I have a text file which is a dataset. and I need to convert it into a CSV format The file is as follows : First line : -1 3:1 11:1 14:1 19:1 39:1 42:1 55:1 64:1 67:1 73:1 75:1 76:1 80:1 83:1 Second line " +1 5:1 11:1 15:1 32:1 39:1 40:1 52:1 63:1 67:1 73:1 74:1 76:1 78:1 83:1 There are a... (6 Replies)
Discussion started by: ajayram
6 Replies

7. UNIX for Dummies Questions & Answers

Take output of processing in text file

Hi ALL, I am presently using perl script mukesh.pl I just want to catch its output into another text file . So I am using > File.txt . I am getting output but i want the whole processing of the script into that file please let me know . Thanks in advance Cheers Mukesh (1 Reply)
Discussion started by: mumakhij
1 Replies

8. Programming

awk processing / Shell Script Processing to remove columns text file

Hello, I extracted a list of files in a directory with the command ls . However this is not my computer, so the ls functionality has been revamped so that it gives the filesizes in front like this : This is the output of ls command : I stored the output in a file filelist 1.1M... (5 Replies)
Discussion started by: ajayram
5 Replies

9. Shell Programming and Scripting

How to grep a log file for words listed in separate text file?

Hello, I want to grep a log ("server.log") for words in a separate file ("white-list.txt") and generate a separate log file containing each line that uses a word from the "white-list.txt" file. Putting that in bullet points: Search through "server.log" for lines that contain any word... (15 Replies)
Discussion started by: nbsparks
15 Replies

10. Shell Programming and Scripting

Text File with Binary Values processing

Hello all, I have a txt file containing millions of lines. Below is the example: {tx:be} head -50 file.txt Instr1: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx Instr1:... (6 Replies)
Discussion started by: Zam_1234
6 Replies
Slurm API(3)					     Slurm job information reporting functions					      Slurm API(3)

NAME
slurm_free_job_alloc_info_response_msg, slurm_free_job_info_msg, slurm_get_end_time, slurm_get_rem_time, slurm_get_select_jobinfo, slurm_load_jobs, slurm_pid2jobid, slurm_print_job_info, slurm_print_job_info_msg - Slurm job information reporting functions ISLURM_GET_REM_TIME, ISLURM_GET_REM_TIME2 - Fortran callable extensions SYNTAX
#include <stdio.h> #include <time.h> #include <slurm/slurm.h> #include <sys/types.h> void slurm_free_job_alloc_info_response_msg ( job_alloc_info_response_msg_t *job_alloc_info_msg_ptr ); void slurm_free_job_info_msg ( job_info_msg_t *job_info_msg_ptr ); int slurm_load_job ( job_info_msg_t **job_info_msg_pptr, uint32_t job_id, uint16_t show_flags, ); int slurm_load_jobs ( time_t update_time, job_info_msg_t **job_info_msg_pptr, uint16_t show_flags ); int slurm_notify_job ( uint32_t job_id, char *message ); int slurm_pid2jobid ( pid_t job_pid, uint32_t *job_id_ptr ); int slurm_get_end_time ( uint32_t jobid, time_t *end_time_ptr ); long slurm_get_rem_time ( uint32_t job_id ); void slurm_print_job_info ( FILE *out_file, job_info_t *job_ptr, int one_liner ); void slurm_print_job_info_msg ( FILE *out_file, job_info_msg_t *job_info_msg_ptr, int one_liner ); int slurm_get_select_jobinfo ( select_jobinfo_t jobinfo, enum select_data_type data_type, void *data ); int slurm_job_cpus_allocated_on_node_id ( job_resources_t *job_resrcs_ptr, int node_id ); int slurm_job_cpus_allocated_on_node ( job_resources_t *job_resrcs_ptr, const char *node_name ); FORTRAN EXTENSION
INTEGER*4 JOBID, REM_TIME REM_TIME = ISLURM_GET_REM_TIME(JOBID) REM_TIME = ISLURM_GET_REM_TIME2() ISLURM_GET_REM_TIME2() is equivalent to ISLURM_GET_REM_TIME() except that the JOBID is taken from the SLURM_JOB_ID environment variable, which is set by SLURM for tasks which it launches. Both functions return the number of seconds remaining before the job reaches the end of it's allocated time. ARGUMENTS
data_type Identifies the type of data to retrieve jobinfo. Note that different types of data are associated with different computer types and different configurations. data The data value identified with data_type is returned in the location specified by data. If a type of data is requested that does not exist on a particular computer type or configuration, slurm_get_select_jobinfo returns an error. See the slurm.h header file for identification of the data types associated with each value of data_type. end_time_ptr Specified a pointer to a storage location into which the expected termination time of a job is placed. job_info_msg_pptr Specifies the double pointer to the structure to be created and filled with the time of the last job update, a record count, and detailed information about each job. Detailed job information is written to fixed sized records and includes: ID number, name, user ID, state, assigned or requested node names, indexes into the node table, etc. In the case of indexes into the node table, this is an array of integers with pairs of start and end index number into the node information records and the data is terminated with a value of -1. See slurm.h for full details on the data structure's contents. job_id Specifies a slurm job id. If zero, use the SLURM_JOB_ID environment variable to get the jobid. job_id_ptr Specifies a pointer to a storage location into which a Slurm job id may be placed. job_info_msg_ptr Specifies the pointer to the structure created by slurm_load_job or slurm_load_jobs. jobinfo Job-specific information as constructed by Slurm's NodeSelect plugin. This data object is returned for each job by the slurm_load_job or slurm_load_jobs function. job_pid Specifies a process id of some process on the current node. job_ptr Specifies a pointer to a single job records from the job_info_msg_ptr data structure. job_resrcs_ptr Pointer to a job_resources_t structure previously using the function slurm_load_job with a show_flags value of SHOW_DETAIL. node_id Zero origin ID of a node allocated to a job. node_name Name of a node allocated to a job. one_liner Print one record per line if non-zero. out_file Specifies the file to print data to. show_flags Job filtering flags, may be ORed. Information about jobs in partitions that are configured as hidden and partitions that the user's group is unable to utilize are not reported by default. The SHOW_ALL flag will cause information about jobs in all partitions to be displayed. The SHOW_DETAIL flag will cause detailed resource allocation information to be reported (e.g. the could of CPUs allo- cated to a job on each node). update_time For all of the following informational calls, if update_time is equal to or greater than the last time changes where made to that information, new information is not returned. Otherwise all the configuration. job, node, or partition records are returned. DESCRIPTION
slurm_free_resource_allocation_response_msg Free slurm resource allocation response message. slurm_free_job_info_msg Release the storage generated by the slurm_load_jobs function. slurm_get_end_time Returns the expected termination time of a specified SLURM job. The time corresponds to the exhaustion of the job's or partition's time limit. NOTE: The data is cached locally and only retrieved from the SLURM controller once per minute. slurm_get_rem_time Returns the number of seconds remaining before the expected termination time of a specified SLURM job id. The time cor- responds to the exhaustion of the job's or partition's time limit. NOTE: The data is cached locally and only retrieved from the SLURM con- troller once per minute. slurm_job_cpus_allocated_on_node and slurm_job_cpus_allocated_on_node_id return the number of CPUs allocated to a job on a specific node allocated to a job. slurm_load_job Returns a job_info_msg_t that contains an update time, record count, and array of job_table records for some specific job ID. slurm_load_jobs Returns a job_info_msg_t that contains an update time, record count, and array of job_table records for all jobs. slurm_notify_job Sends the specified message to standard output of the specified job ID. slurm_pid2jobid Returns a Slurm job id corresponding to the supplied local process id. This only works for processes which Slurm spawns and their descendants. slurm_print_job_info Prints the contents of the data structure describing a single job records from the data loaded by the slurm_load_node function. slurm_print_job_info_msg Prints the contents of the data structure describing all job records loaded by the slurm_load_node function. RETURN VALUE
For slurm_get_rem_time on success a number of seconds is returned. For all other functions zero is returned on success. On error, -1 is returned, and Slurm error code is set appropriately. ERRORS
SLURM_NO_CHANGE_IN_DATA Data has not changed since update_time. SLURM_PROTOCOL_VERSION_ERROR Protocol version has changed, re-link your code. ESLURM_INVALID_JOB_ID Request for information about a non-existent job. SLURM_PROTOCOL_SOCKET_IMPL_TIMEOUT Timeout in communicating with SLURM controller. INVAL Invalid function argument. EXAMPLE
#include <stdio.h> #include <stdlib.h> #include <slurm/slurm.h> #include <slurm/slurm_errno.h> #include <sys/types.h> int main (int argc, char *argv[]) { int i; job_info_msg_t * job_buffer_ptr = NULL; job_info_t * job_ptr; uint32_t job_id; /* get and dump some job information */ if ( slurm_load_jobs ((time_t) NULL, &job_buffer_ptr, SHOW_ALL) ) { slurm_perror ("slurm_load_jobs error"); exit (1); } /* The easy way to print... */ slurm_print_job_info_msg (stdout, job_buffer_ptr, 0); /* A harder way.. */ for (i = 0; i < job_buffer_ptr->record_count; i++) { job_ptr = &job_buffer_ptr->job_array[i]; slurm_print_job_info(stdout, job_ptr, 1); } /* The hardest way. */ printf ("Jobs updated at %lx, record count %d ", job_buffer_ptr->last_update, job_buffer_ptr->record_count); for (i = 0; i < job_buffer_ptr->record_count; i++) { printf ("JobId=%u UserId=%u ", job_buffer_ptr->job_array[i].job_id, job_buffer_ptr->job_array[i].user_id); } if (job_buffer_ptr->record_count >= 1) { uint16_t nodes; if (slurm_get_select_jobinfo( job_buffer_ptr->job_array[0].select_jobinfo, SELECT_JOBDATA_NODE_CNT, &nodes) == SLURM_SUCCESS) printf("JobId=%u Nodes=%u ", job_buffer_ptr->job_array[0].job_id, nodes); } slurm_free_job_info_msg (job_buffer_ptr); if (slurm_pid2jobid (getpid(), &job_id)) slurm_perror ("slurm_load_jobs error"); else printf ("Slurm job id = %u ", job_id); exit (0); } NOTES
These functions are included in the libslurm library, which must be linked to your process for use (e.g. "cc -lslurm myprog.c"). Some data structures contain index values to cross-reference each other. If the show_flags argument is not set to SHOW_ALL when getting this data, these index values will be invalid. The slurm_hostlist_ functions can be used to convert SLURM node list expressions into a collection of individual node names. COPYING
Copyright (C) 2002-2006 The Regents of the University of California. Copyright (C) 2008-2010 Lawrence Livermore National Security. Pro- duced at Lawrence Livermore National Laboratory (cf, DISCLAIMER). CODE-OCEC-09-009. All rights reserved. This file is part of SLURM, a resource management program. For details, see <http://www.schedmd.com/slurmdocs/>. SLURM is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. SLURM is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. SEE ALSO
scontrol(1), squeue(1), slurm_hostlist_create(3), slurm_hostlist_shift(3), slurm_hostlist_destroy(3), slurm_allocation_lookup(3), slurm_get_errno(3), slurm_perror(3), slurm_strerror(3) Morris Jette September 2006 Slurm API(3)
All times are GMT -4. The time now is 10:59 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy