Sponsored Content
Top Forums Shell Programming and Scripting awk - fetch multiple data from huge dump Post 302884002 by navkanwal on Friday 17th of January 2014 06:56:41 AM
Old 01-17-2014
Hi Akshay

Pattern:
Code:
110010000000206
110010000000210
110010000000211
110010000000214

Dump:
Code:
567810008161509|aav|xac|aab|aav|xac|aab|567810008161509|110010000000206|aav|xac|aab|aav|xac|aab|
567810072227627|aav|xac|aab|aav|xac|aab|567810008161509|110010000000207|aav|xac|aab|aav|xac|aab|
567811368851555|aav|xac|aab|aav|xac|aab|567810008161509|110010000000208|aav|xac|aab|aav|xac|aab|
567811422513652|aav|xac|aab|aav|xac|aab|567810008161509|110010000000209|aav|xac|aab|aav|xac|aab|
567812130217683|aav|xac|aab|aav|xac|aab|567810008161509|xac|dud|110010000000210|xac|aab|aav|xac|aab|
567813220211182|aav|xac|aab|aav|xac|aab|567810008161509|110010000000211|aav|xac|aab|aav|xac|aab|
567813449322589|aav|xac|aab|aav|xac|aab|567810008161509|110010000000212|aav|xac|aab|aav|xac|aab|
567813741319623|aav|xac|aab|aav|xac|aab|567810008161509|110010000000213|aav|xac|aab|aav|xac|aab|
567816323171591|aav|xac|aab|aav|xac|aab|567810008161509|mum|does|110010000000214|aav|xac|aab|aav|xac|aab|
567816660521463|aav|xac|aab|aav|xac|aab|567810008161509|110010000000215|aav|xac|aab|aav|xac|aab|
567818208711973|aav|xac|aab|aav|xac|aab|567810008161509|110010000000216|aav|xac|aab|aav|xac|aab|

In the Dump files, the field is not constant so it could be in any field number.
Regards
Navkanwal

Last edited by Scott; 01-17-2014 at 08:10 AM.. Reason: Use code tags, please...
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Fetch selected data from webpage

Hi All, Can anybody tell me the command used for extracting some selected lines from a web-page. I guess we'll have to do this using wget or Curl to achieve this.... If anbody has any idea abt it, kindly post your reply ASAP. Thanks. (1 Reply)
Discussion started by: sunnydynamic15
1 Replies

2. Shell Programming and Scripting

fetch data between two timestamp using script

Hi Guys, I have the data in below format. 25 Dec 2011 03:00:01 : aaaaaaaaaaaaaaa 25 Dec 2011 04:23:23 : bbbbbbbbbbbbbbb 25 Dec 2011 16:12:45 : ccccccccccccccc 26 Dec 2011 04:45:34 : ddddddddddddddd 26 Dec 2011 17:01:22 : eeeeeeeeeeeeeee 27 Dec 2011 12:33:45 : ffffffffffffffffffffffff 28... (13 Replies)
Discussion started by: jaituteja
13 Replies

3. Shell Programming and Scripting

Awk to Count Multiple patterns in a huge file

Hi, I have a file that is 430K lines long. It has records like below |site1|MAP |site2|MAP |site1|MODAL |site2|MAP |site2|MODAL |site2|LINK |site1|LINK My task is to count the number of time MAP, MODAL, LINK occurs for a single site and write new records like below to a new file ... (5 Replies)
Discussion started by: reach.sree@gmai
5 Replies

4. Shell Programming and Scripting

Fetch data from a particular location

I want to fetch value from a particular location from a file but in each line in the file it appears at a different position so i tried using variable with cut command but it is not working properly. The code i have written is #!/bin/sh cat Sri1.log | while read d2 do grep -w... (9 Replies)
Discussion started by: Prachi Gupta
9 Replies

5. UNIX for Dummies Questions & Answers

how to fetch data in unix

Hi All, I have a file with the below data as shown. A|2|20120430 B|EMP|NAME|DEPT C|12|SARC|01 C|23||ASDD|02 D|END OF FILE I want to fetch only the records that contains C|, what is unix command to fetch this data. Thanks (5 Replies)
Discussion started by: halpavan2
5 Replies

6. Shell Programming and Scripting

Help Need to fetch the required data

Hi Guys, Am in need of your help one more time on my real data. I have a file which contains more than thousand lines of data Live data shown for 4 iterations. We have more than thousand lines of data:- -------------------------------------------------------------------------- ... (4 Replies)
Discussion started by: rocky2013
4 Replies

7. Shell Programming and Scripting

awk does not work well with huge data?

Dear all , I found that if we work with thousands line of data, awk does not work perfectly. It will cut hundreds line (others are deleted) and works only on the remain data. I used this command : awk '$1==1{$1="Si"}{print>FILENAME}' coba.xyz to change value of first column whose value is 1... (4 Replies)
Discussion started by: ariesto
4 Replies

8. Shell Programming and Scripting

Need to fetch only selected data in CSV

Hi Team, I m getting my script commands output like given below GETA-TILL-INF; U-UU-YRYT-NOD-6002 2015-05-14 THU 19:44:10 C2221 RETRIEVE TILL INFORMATION : COMPLD ---------------------------------------------------------------------- CONNECT_CARD_ID ... (9 Replies)
Discussion started by: Ganesh Mankar
9 Replies

9. Shell Programming and Scripting

Fetch data from file

Hi, I am new to scripting. I have a log file and need to fetch specific logs and copy to another file. A copy of the log is like this: =============================================================== = JOB : server123#jobs1.jobstream1 = USER : andyc = Tue 08/01/17... (3 Replies)
Discussion started by: Prngp
3 Replies

10. UNIX for Advanced & Expert Users

Need Optimization shell/awk script to aggreagte (sum) for all the columns of Huge data file

Optimization shell/awk script to aggregate (sum) for all the columns of Huge data file File delimiter "|" Need to have Sum of all columns, with column number : aggregation (summation) for each column File not having the header Like below - Column 1 "Total Column 2 : "Total ... ...... (2 Replies)
Discussion started by: kartikirans
2 Replies
SPLIT(1P)						     POSIX Programmer's Manual							 SPLIT(1P)

PROLOG
This manual page is part of the POSIX Programmer's Manual. The Linux implementation of this interface may differ (consult the correspond- ing Linux manual page for details of Linux behavior), or the interface may not be implemented on Linux. NAME
split -- split files into pieces SYNOPSIS
split [-l line_count] [-a suffix_length] [file[name]] split -b n[k|m] [-a suffix_length] [file[name]] DESCRIPTION
The split utility shall read an input file and write one or more output files. The default size of each output file shall be 1000 lines. The size of the output files can be modified by specification of the -b or -l options. Each output file shall be created with a unique suf- fix. The suffix shall consist of exactly suffix_length lowercase letters from the POSIX locale. The letters of the suffix shall be used as if they were a base-26 digit system, with the first suffix to be created consisting of all 'a' characters, the second with a 'b' replacing the last 'a', and so on, until a name of all 'z' characters is created. By default, the names of the output files shall be 'x', followed by a two-character suffix from the character set as described above, starting with "aa", "ab", "ac", and so on, and continuing until the suf- fix "zz", for a maximum of 676 files. If the number of files required exceeds the maximum allowed by the suffix length provided, such that the last allowable file would be larger than the requested size, the split utility shall fail after creating the last file with a valid suffix; split shall not delete the files it created with valid suffixes. If the file limit is not exceeded, the last file created shall contain the remainder of the input file, and may be smaller than the requested size. OPTIONS
The split utility shall conform to the Base Definitions volume of POSIX.1-2008, Section 12.2, Utility Syntax Guidelines. The following options shall be supported: -a suffix_length Use suffix_length letters to form the suffix portion of the filenames of the split file. If -a is not specified, the default suf- fix length shall be two. If the sum of the name operand and the suffix_length option-argument would create a filename exceeding {NAME_MAX} bytes, an error shall result; split shall exit with a diagnostic message and no files shall be created. -b n Split a file into pieces n bytes in size. -b nk Split a file into pieces n*1024 bytes in size. -b nm Split a file into pieces n*1048576 bytes in size. -l line_count Specify the number of lines in each resulting file piece. The line_count argument is an unsigned decimal integer. The default is 1000. If the input does not end with a <newline>, the partial line shall be included in the last output file. OPERANDS
The following operands shall be supported: file The pathname of the ordinary file to be split. If no input file is given or file is '-', the standard input shall be used. name The prefix to be used for each of the files resulting from the split operation. If no name argument is given, 'x' shall be used as the prefix of the output files. The combined length of the basename of prefix and suffix_length cannot exceed {NAME_MAX} bytes. See the OPTIONS section. STDIN
See the INPUT FILES section. INPUT FILES
Any file can be used as input. ENVIRONMENT VARIABLES
The following environment variables shall affect the execution of split: LANG Provide a default value for the internationalization variables that are unset or null. (See the Base Definitions volume of POSIX.1-2008, Section 8.2, Internationalization Variables for the precedence of internationalization variables used to determine the values of locale categories.) LC_ALL If set to a non-empty string value, override the values of all the other internationalization variables. LC_CTYPE Determine the locale for the interpretation of sequences of bytes of text data as characters (for example, single-byte as opposed to multi-byte characters in arguments and input files). LC_MESSAGES Determine the locale that should be used to affect the format and contents of diagnostic messages written to standard error. NLSPATH Determine the location of message catalogs for the processing of LC_MESSAGES. ASYNCHRONOUS EVENTS
Default. STDOUT
Not used. STDERR
The standard error shall be used only for diagnostic messages. OUTPUT FILES
The output files contain portions of the original input file; otherwise, unchanged. EXTENDED DESCRIPTION
None. EXIT STATUS
The following exit values shall be returned: 0 Successful completion. >0 An error occurred. CONSEQUENCES OF ERRORS
Default. The following sections are informative. APPLICATION USAGE
None. EXAMPLES
In the following examples foo is a text file that contains 5000 lines. 1. Create five files, xaa, xab, xac, xad, and xae: split foo 2. Create five files, but the suffixed portion of the created files consists of three letters, xaaa, xaab, xaac, xaad, and xaae: split -a 3 foo 3. Create three files with four-letter suffixes and a supplied prefix, bar_aaaa, bar_aaab, and bar_aaac: split -a 4 -l 2000 foo bar_ 4. Create as many files as are necessary to contain at most 20*1024 bytes, each with the default prefix of x and a five-letter suffix: split -a 5 -b 20k foo RATIONALE
The -b option was added to provide a mechanism for splitting files other than by lines. While most uses of the -b option are for transmit- ting files over networks, some believed it would have additional uses. The -a option was added to overcome the limitation of being able to create only 676 files. Consideration was given to deleting this utility, using the rationale that the functionality provided by this utility is available via the csplit utility (see csplit). Upon reconsideration of the purpose of the User Portability Utilities option, it was decided to retain both this utility and the csplit utility because users use both utilities and have historical expectations of their behavior. Furthermore, the splitting on byte boundaries in split cannot be duplicated with the historical csplit. The text ``split shall not delete the files it created with valid suffixes'' would normally be assumed, but since the related utility, csplit, does delete files under some circumstances, the historical behavior of split is made explicit to avoid misinterpretation. Earlier versions of this standard allowed a -line_count option. This form is no longer specified by POSIX.1-2008 but may be present in some implementations. FUTURE DIRECTIONS
None. SEE ALSO
csplit The Base Definitions volume of POSIX.1-2008, Chapter 8, Environment Variables, Section 12.2, Utility Syntax Guidelines COPYRIGHT
Portions of this text are reprinted and reproduced in electronic form from IEEE Std 1003.1, 2013 Edition, Standard for Information Technol- ogy -- Portable Operating System Interface (POSIX), The Open Group Base Specifications Issue 7, Copyright (C) 2013 by the Institute of Electrical and Electronics Engineers, Inc and The Open Group. (This is POSIX.1-2008 with the 2013 Technical Corrigendum 1 applied.) In the event of any discrepancy between this version and the original IEEE and The Open Group Standard, the original IEEE and The Open Group Stan- dard is the referee document. The original Standard can be obtained online at http://www.unix.org/online.html . Any typographical or formatting errors that appear in this page are most likely to have been introduced during the conversion of the source files to man page format. To report such errors, see https://www.kernel.org/doc/man-pages/reporting_bugs.html . IEEE
/The Open Group 2013 SPLIT(1P)
All times are GMT -4. The time now is 05:51 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy