Sponsored Content
Top Forums UNIX for Advanced & Expert Users Need help for faster file read and grep in big files Post 303018518 by RudiC on Friday 8th of June 2018 03:58:04 PM
Old 06-08-2018
Try also (tackling it from the other end)
Code:
awk '
FNR == 1        {FILE++
                }
FILE < 3        {FIN[FILE,$4] = FIN[FILE,$4] $8 FS
                }
FILE == 3       {n = split(FIN[2,$1], T1)
                 for (i=1; i<=n; i++)   {m = split(FIN[1,T1[i]], T2)
                                         for (j=1; j<=m; j++) print $1, T1[i], T2[j]
                                        }
                }
' file3 file2 file1
3434343 myout1 FinalOut1
3434343 myout2 FinalOut2
3434343 myout2 FinalOut3

This User Gave Thanks to RudiC For This Post:
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

How to grep faster ?

Hi I have to grep for 2000 strings in a file one after the other.Say the file name is Snxx.out which has these strings. I have to search for all the strings in the file Snxx.out one after the other. What is the fastest way to do it ?? Note:The current grep process is taking lot of time per... (7 Replies)
Discussion started by: preethgideon
7 Replies

2. Shell Programming and Scripting

Big data file - sed/grep/awk?

Morning guys. Another day another question. :rolleyes: I am knocking up a script to pull some data from a file. The problem is the file is very big (up to 1 gig in size), so this solution: for results in `grep "^\ ... works, but takes ages (we're talking minutes) to run. The data is held... (8 Replies)
Discussion started by: dlam
8 Replies

3. UNIX for Advanced & Expert Users

Split a big file into two others files

Hello, i have a very big file that has more then 80 MBytes (100MBytes). So with my CVS Application I cannot commit this file (too Big) because it must have < 80 MBytes. How can I split this file into two others files, i think the AIX Unix command : split -b can do that, buit how is the right... (2 Replies)
Discussion started by: steiner
2 Replies

4. UNIX for Dummies Questions & Answers

Faster than nested while read loops?

Hi experts, I just want to know if there is a better solution to my nested while read loops below: while read line; do while read line2; do while read line3; do echo "$line $line2 $line3" done < file3.txt done < file2.txt done < file1.txt >... (4 Replies)
Discussion started by: chstr_14
4 Replies

5. Shell Programming and Scripting

Read a file with n records as one big string using linux

Hello! Is there a way i can read a file with n records as one big string using linux shell script? I have a file in the below format - REC1 REC2 REC3 . . . REC4 Record length is 3000 bytes per record and with a newline char at the end. What i need to do is - read this file as one... (5 Replies)
Discussion started by: mailme0205
5 Replies

6. Shell Programming and Scripting

Grep -v -f and sort|diff which way is faster

Hi Gurus, I have two big files. I need to compare the different. currently, I am using sort file1 > file1_temp; sort file2 > file2_tmp diff file1_tmp file2_tmp I can use command grep -v -f file1 file2 just wondering which way is fast to compare two big files. Thanks... (4 Replies)
Discussion started by: ken6503
4 Replies

7. UNIX for Dummies Questions & Answers

What is the faster way to grep from huge file?

Hi All, I am new to this forum and this is my first post. My requirement is like to optimize the time taken to grep the file with 40000 lines. There are two files FILEA(40000 lines) FILEB(40000 lines). The requirement is like this, both the file will be in the format below... (11 Replies)
Discussion started by: mad man
11 Replies

8. Shell Programming and Scripting

A faster way to read and search

I have a simple script that reads in data from fileA.txt and searches line by line for that data in multiple files (*multfiles.txt). It only prints the data when there is more than 1 instance of it. The problem is that its really slow (3+ hours) to complete the entire process. There are nearly 1500... (10 Replies)
Discussion started by: ncwxpanther
10 Replies

9. UNIX for Beginners Questions & Answers

Grep -f for big files

ok guys. this isnt homework or anything. i have been using grep -f all my life but i am trying this for a huge file and it doesnt work. can someone give me a replacement for grep -f pattern file for big files? thanks (6 Replies)
Discussion started by: ahfze
6 Replies

10. Solaris

Split a big file system to several files

Gents Actually I have question and i need your support. I have this NAS file system mounted as /coresys has size of 7 TB I need to Split this file system into several file systems as mount points I mean how to can I Split it professionally to different NAS mount points how to can I decide... (2 Replies)
Discussion started by: AbuAliiiiiiiiii
2 Replies
UNLOCKED_STDIO(3)					     Linux Programmer's Manual						 UNLOCKED_STDIO(3)

NAME
*_unlocked - non-locking stdio functions SYNOPSIS
#include <stdio.h> int getc_unlocked(FILE *stream); int getchar_unlocked(void); int putc_unlocked(int c, FILE *stream); int putchar_unlocked(int c); #define _BSD_SOURCE /* or _SVID_SOURCE or _GNU_SOURCE */ #include <stdio.h> void clearerr_unlocked(FILE *stream); int feof_unlocked(FILE *stream); int ferror_unlocked(FILE *stream); int fileno_unlocked(FILE *stream); int fflush_unlocked(FILE *stream); int fgetc_unlocked(FILE *stream); int fputc_unlocked(int c, FILE *stream); size_t fread_unlocked(void *ptr, size_t size, size_t n, FILE *stream); size_t fwrite_unlocked(const void *ptr, size_t size, size_t n, FILE *stream); #define _GNU_SOURCE #include <stdio.h> char *fgets_unlocked(char *s, int n, FILE *stream); int fputs_unlocked(const char *s, FILE *stream); #define _GNU_SOURCE #include <wchar.h> wint_t getwc_unlocked(FILE *stream); wint_t getwchar_unlocked(void); wint_t fgetwc_unlocked(FILE *stream); wint_t fputwc_unlocked(wchar_t wc, FILE *stream); wint_t putwc_unlocked(wchar_t wc, FILE *stream); wint_t putwchar_unlocked(wchar_t wc); wchar_t *fgetws_unlocked(wchar_t *ws, int n, FILE *stream); int fputws_unlocked(const wchar_t *ws, FILE *stream); DESCRIPTION
Each of these functions has the same behaviour as its counterpart without the `_unlocked' suffix, except that they do not use locking (they do not set locks themselves, and do not test for the presence of locks set by others) and hence are thread-unsafe. See flockfile(3). CONFORMING TO
The four functions getc_unlocked(), getchar_unlocked(), putc_unlocked(), putchar_unlocked() are in POSIX.1. The nonstandard *_unlocked() variants occur on a few Unix systems, and are available in recent glibc. They should probably not be used. SEE ALSO
flockfile(3) 2001-10-18 UNLOCKED_STDIO(3)
All times are GMT -4. The time now is 03:32 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy