Sponsored Content
Top Forums UNIX for Dummies Questions & Answers Finding and Extracting uniq data in multiple files Post 302489302 by ctsgnb on Thursday 20th of January 2011 06:22:42 AM
Old 01-20-2011
not sur about the grep -f <( ) syntax (may need some syntax fix)
but maybe something like

Code:
for i in File*.txt; do grep -f <(cat File*.txt | sort | uniq -u | sed 's/.*/\^&\$/') $i | head -1 >$i.out; done

or maybe something like (i didn't test it so it may require some syntax adjustment)

Code:
cat File*.txt | sort | uniq -u | sed 's/.*/\^&\$/' | xargs -i grep {} File*.txt | while IFS=: read a b; do [[ ! -f $a.out ]] && echo "$b" >"$a".out; done

the sed part has been added to ensure that grep Data2 would grep ^Data2$ so if there are also occurrence of Data22 or Data23 they won't be displayed

Last edited by ctsgnb; 01-20-2011 at 07:35 AM..
This User Gave Thanks to ctsgnb For This Post:
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

extracting data from files..

frnds, I m having prob woth doing some 2-3 task simultaneously... what I want is... I have lots ( lacs ) of files in a dir... I want.. these info from arround 2-3 months files filename convention is - abc20080403sdas.xyz ( for todays files ) I want 1. total no of files for 1 dec... (1 Reply)
Discussion started by: clx
1 Replies

2. Shell Programming and Scripting

Modify log files to get uniq data

Hello, I have a log file that has following output as below. LAP.sun5 CC LAP.sun5 CQ perl.sun5 CC perl.sun5 CQ TSLogger.sun5 CC TSLogger.sun5 CQ TSLogger.sun5 KR WAS.sun5 CC WAS.sun5 MT WAS.sun5 CQ I want to output to be in the way below, i tried using awk but could not do it. ... (12 Replies)
Discussion started by: asirohi
12 Replies

3. UNIX for Dummies Questions & Answers

Extracting data from many compressed files

I have a large number (50,000) of pretty large compressed files and I need only certain lines of data from them (each relevant line contains a certain key word). Each file contains 300 such lines. The individual file names are indexed by file number (file_name.1, file_name.2, ... ,... (1 Reply)
Discussion started by: Boltzmann
1 Replies

4. Shell Programming and Scripting

Extracting/condensing text from multiple files to multiples files

Hi Everyone, I'm really new to all this so I'm really hoping someone can help. I have a directory with ~1000 lists from which I want to extract lines from and write to new files. For simplicity lets say they are shopping lists and I want to write out the lines corresponding to apples to a new... (2 Replies)
Discussion started by: born2phase
2 Replies

5. UNIX for Dummies Questions & Answers

Extracting data between specific lines, multiple times

I need help extracting specific lines in a text file. The file looks like this: POSITION TOTAL-FORCE (eV/Angst) ----------------------------------------------------------------------------------- 1.86126 1.86973 1.86972 ... (14 Replies)
Discussion started by: captainalright
14 Replies

6. Shell Programming and Scripting

Extracting specific files from multiple .tgz files

Hey, I have number of .tgz files and want to extract the file with the ending *results.txt from each one. I have tried for file in *.tgz; do tar --wildcards -zxf $file *results.txt; doneas well as list=$(ls *.tgz) for i in $list; do tar --wildcards -zxvf $i *.results.txt; done... (1 Reply)
Discussion started by: jfern
1 Replies

7. Shell Programming and Scripting

Combine data from two files base on uniq data

File 1 ID Name Po1 Po2 DD134 DD134_4A_1 NN-1 L_0_1 DD134 DD134_4B_1 NN-2 L_1_1 DD134 DD134_4C_1 NN-3 L_2_1 DD142 DD142_4A_1 NN-1 L_0_1 DD142 DD142_4B_1 NN-2 L_1_1 DD142 DD142_4C_1 NN-3 L_2_1 DD142 DD142_3A_1 NN-41 L_3_1 DD142 DD142_3A_1 NN-42 L_3_2 File 2 ( Combination of... (1 Reply)
Discussion started by: pareshkp
1 Replies

8. Shell Programming and Scripting

Extracting data from multiple lines

Hi All, I am stuck in one step.. I have one file named file.txt having content: And SGMT.perd_id = (SELECT cal.fiscal_perd_id FROM $ODS_TARGT.TIM_DT_CAL_D CAL FROM $ODS_TARGT.GL_COA_SEGMNT_XREF_A SGMT SGMT.COA_XREF_TYP_IDN In (SEL COA_XREF_TYP_IDN From... (4 Replies)
Discussion started by: Shilpi Gupta
4 Replies

9. Shell Programming and Scripting

Extracting data from specific rows and columns from multiple csv files

I have a series of csv files in the following format eg file1 Experiment Name,XYZ_07/28/15, Specimen Name,Specimen_001, Tube Name, Control, Record Date,7/28/2015 14:50, $OP,XYZYZ, GUID,abc, Population,#Events,%Parent All Events,10500, P1,10071,95.9 Early Apoptosis,1113,11.1 Late... (6 Replies)
Discussion started by: pawannoel
6 Replies

10. Shell Programming and Scripting

Extracting part of data from files

Hi All, I have log files as below. log1.txt <table name="content_analyzer" primary-key="id"> <type="global" /> </table> <table name="content_analyzer2" primary-key="id"> <type="global" /> </table> Time taken: 1.008 seconds ID = gd54321bbvbvbcvb <table name="content_analyzer"... (7 Replies)
Discussion started by: ROCK_PLSQL
7 Replies
SUR(1)																	    SUR(1)

NAME
sur - subtle user repository SYNOPSIS
sur COMMAND OPTIONS DESCIPTION
sur is the subtle user repository and a manager for sublets in a rubygems like fashion. The main goal of sur is to help the user managing sublets and to have a central point where new sublets can be found. Addionally sur has some capabilities to create and test sublets. COMMANDS
o sur annotate NAME [-v VERSION|-h] Mark a sublet as to be reviewed o sur build SPEC Create a sublet package o sur config NAME Show available config settings of a sublet o sur fetch NAME Download sublet to current directory o sur help Show this help and exit o sur grabs NAME Show available grabs provided by a sublet o sur info NAME Show info about an installed sublet o sur install NAME [-R|-t|-v VERSION|-h] Install a sublet o sur list [-l|-r|-h] List local/remote sublets o sur notes NAME Show notes about a sublet o sur query NAME [-e|-l|-r|-t|-v VERSION|-h] Query for a sublet (e.g clock, clock -v 0.3) o sur reorder Reorder installed sublets for loading order o sur server [-p PORT|-h] Serve sublets (default: http://localhost:4567) o sur submit FILE Submit a sublet to SUR o sur template FILE Create a new sublet template in current dir o sur test NAME [-C VALUE|-h] Test given sublets for syntax and functionality o sur uninstall NAME [-R|-t|-v VERSION|-h] Uninstall a sublet o sur unpack NAME [-t|-v VERSION|-h] Unpack a sublet in current directory o sur update [-l|-r|-h] Update local/remote sublet cache o sur upgrade [-R|-y|-h] Upgrade all installed sublets o sur version Show version info and exit o sur yank NAME Delete sublet from server OPTIONS
o annotate NAME [-v VERSION|-h] -v, --version VERSION Annotate a specific version -h, --help Show this help and exit o fetch NAME [-t|-v VERSION|-h] -t, --tags Include tags in search -v, --version VERSION Annotate a specific version -h, --help Show this help and exit o install NAME [-R|-t|-v VERSION|-h] -R, --reload Reload sublets after installing -t, --tags Include tags in search -v, --version VERSION Search for a specific version -h, --help Show this help and exit o list [-l|-r|-h] -l, --local Select local repository (default) -r, --remote Select remote repository -h, --help Show this help and exit o query NAME [-e|-l|-r|-t|-v VERSION|-h] -e, --regex Use regex for query -l, --local Select local repository (default) -r, --remote Select remote repository -t, --tags Include tags in search -v, --version VERSION Search for a specific version -h, --help Show this help and exit o server [-p PORT|-h] -p, --port Select a specific port -h, --help Show this help and exit o test NAME [-C VALUE|-h] -C, --config VALUE Add config value (can be used multiple times) -h, --help Show this help and exit o unpack NAME [-t|-v VERSION|-h] -t, --tag Search for a specific tag -v, --version VERSION Search for a specific version -h, --help Show this help and exit o uninstall NAME [-R|-t|-v VERSION|-h] -R, --reload Reload sublets after installing -t, --tag Search for a specific tag -v, --version VERSION Search for a specific version -h, --help Show this help and exit o update [-l|-r|-h] -l, --local Select local repository (default) -r, --remote Select remote repository -h, --help Show this help and exit o upgrade [-R|-y|-h] -R, --reload Reload sublets after upgrading -y, --yes Assume yes to questions -h, --help Show this help and exit EXAMPLES
sur install clock sur query -r clock sur uninstall -v 0.1 clock SPECIFICATION
In order to create a sublet, you need to create a specifiction which basically contains some information about your sublet. Here is a list of known attributes: o name Name of the sublet Example: spec.name = "Sublet" o version Version of the sublet Example: spec.version = "0.1" o tags List of tags to categorize the sublet Example: spec.tags = [ "Broken" ] o files List of files in the sublet Example: spec.files = [ "sublet.rb" ] o icons List of supplied icons Example: spec.icons = [ "icon.xbm" ] o description Description of the sublet Example: spec.description = "A shiny new sublet" o notes Longer description of the sublet Example: spec.notes = <<NOTES "This sublet is just a dummy, have fun with it!" NOTES o authors List of authors of the sublet in case there are more than one Example: spec.authors = [ "You" ] o contact Contact mail address Example: spec.contact = "your@mail.com" o date Date of creation Example: spec.date = "Sat Sep 13 19:00 CET 2008" o config Description of config settings Example: spec.config = [ { :name => "format_string", :type => "string", :description => "Format of the clock (man date)" }, :def_value => "Default value" } ] o grabs Description of grabs Example: spec.grabs = [ { :SubletTest => "Test grab", } ] o required_version Required version of subtle Example: spec.required_version = "0.9.10" o add_dependency(name, version) Add a gem dependency Example: spec.add_dependency("a_gem", "0.0") If you use the template command sur will create an empty template with a spec file. BUGS
Report bugs at http://subforge.org/projects/subtle/issues Homepage: http://subtle.subforge.org COPYRIGHT
Copyright (c) Christoph Kappel unexist@dorfelite.net SEE ALSO
surserver(1), subtle(1), subtler(1), subtlext(1) October 2011 SUR(1)
All times are GMT -4. The time now is 07:20 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy