Understanding Xargs


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Understanding Xargs
# 1  
Old 12-26-2017
Understanding Xargs

I'm struggling to understand the man page entry about xargs in conjunction with the option -I. It states:

Quote:
Execute utility for each input line, replacing one or more occur-
rences of replstr in up to replacements (or 5 if no -R flag is
specified) arguments to utility with the entire line of input
And the description reads:

Quote:
The xargs utility reads space, tab, newline and end-of-file delimited
strings from the standard input and executes utility with the strings as
arguments.
Am I correct interpreting that as the fact that if no specified otherwise xargs -I can pass to a command following it no more than 5 arguments (a.k.a "strings") each of them containing unlimited number of occurrences of the data to be replaced using that command?

Furthermore what is a string in UNIX?

Example:
pass a list of highest level objects of current directory to some command

Code:
ls . | xargs some_command

Does ls return one string comprising multiple lines? How many arguments that passes for? Or is one line = 1 string? 1 argument?

Moderator's Comments:
Mod Comment Thread moved. Please post your questions in the appropriate sub-forum. Thank you.
# 2  
Old 12-26-2017
A string in *nix is a sequence of (not necessarily printable) characters terminated by a null (0x00) character.

xargs usually collects lines read from stdin into one or more long (influenced by several options) parameter lists and executes the command / utility one or more times with the respective parameter list. With the -I option you can define where in that command execution the data from stdin show up. I'd recommend you do some exercises and testing with some innocuous commands and the -t (verbose) option.
# 3  
Old 12-26-2017
If you need to filter files based on information only provided by "ls -l" (long list), you can do so but then remove everything through the last space with a sed command. Its output can then be passed to xargs and any do whatever needs to be done to the files.

Last edited by wbport; 12-26-2017 at 12:43 PM..
# 4  
Old 12-27-2017
To the moderator

Quote:
Originally Posted by scrutinizerix
Moderator's Comments:
Mod Comment Thread moved. Please post your questions in the appropriate sub-forum. Thank you.
I'm sorry, I wasn't here for a very long time thus effectively mismemorizing many things and the forum's very abstract, disorganized cold interface confused me as to where did I find a relevant section to post to. Sorry, anyway.

---------- Post updated at 11:11 AM ---------- Previous update was at 11:02 AM ----------

Quote:
Originally Posted by RudiC
A string in *nix is a sequence of (not necessarily printable) characters terminated by a null (0x00) character.

xargs usually collects lines read from stdin into one or more long (influenced by several options) parameter lists and executes the command / utility one or more times with the respective parameter list. With the -I option you can define where in that command execution the data from stdin show up. I'd recommend you do some exercises and testing with some innocuous commands and the -t (verbose) option.
1.But do these lines form a string?

2. The problem lies for me considering distinction between the notions "string", "line", "argument", "occurrence" as per the cited manual page.

From xargs manual:

Quote:
Any arguments specified on the command line are given to utility upon
each invocation, followed by some number of the arguments read from the
standard input of xargs
That saying what are the "strings as arguments" exactly? Is that like a list containing items? If yes what those items are? Are they lines?

If yes, are occurrences contained in these lines?

I'd like to construct in my mind a sort of "object model" for any of the terms.

Last edited by scrutinizerix; 12-27-2017 at 05:13 AM.. Reason: Text added
# 5  
Old 12-27-2017
Now, these are very basic questions from quite broad a range of IT, and I'm not sure I can cover that to satisfaction. On top, there may be a language barrier, e.g with "occurrence". I'd advise to use a dictionary, as - to me - the meaning got immediately clear when seeing the translations. Please also consult introductory text books and / or man pages.

It might be worthwhile to internalize the concept of a string which you will encounter everywhere in IT (tools, databases, documents, files, ...) when needing to represent text. It can come in a variety of shapes, like fixed or varying length strings, zero terminated or with a leading length indicator, string constants, substrings, string concatenations, and what have you, and there are many tools, libraries, functions to handle them. Different digital items (numbers, logical values) can be output to screen as readable text representations only, not as the individual items themselves.

Then, there are text files, a loosely structured collection of (mostly) printable characters. In *nix systems, those consist of lines of characters terminated by a <new line> (\n, ^J, 0x0A) character. But this is not the only possible text representation. When reading a line from a file, you can put it into a single string variable, or split it into several substrings. If you do so by applying spaces and / or punctuation chars for separating, the substrings will be words. But any other separation is possible albeit not necessarily sensible. So, a line is sort of a superset of (a group of) strings.

A "command line" specifies a collection of a command name (perhaps including a path), zero or more options (with possible arguments), and zero or more parameters. Any of those is a (possibly one character) string, analysed by the command interpreter, and then supplied to the program being executed. Please be aware that the terms "argument" and "parameter" are not strictly distinguished between and both are loosely and interchangably used. (I neglected possible local variable assingments and redirections to avoid overcomplicaton.)

Last edited by RudiC; 12-27-2017 at 10:13 AM.. Reason: Some typos
# 6  
Old 12-28-2017
Quote:
Originally Posted by RudiC
On top, there may be a language barrier, e.g with "occurrence". I'd advise to use a dictionary, as - to me - the meaning got immediately clear when seeing the translations.
I'm not sure I got you right: there's absolutely no alleged "language barrier"; why the word "occurrence" caught your eye I didn't get either. That was not some kind of mistyping (which would be a common thing among both native and non-native English speakers even then and certainly would not be the worst one to consider it intolerable). I use dictionaries well enough when I deem it necessary. Thank you for your advise.

Quote:
Originally Posted by RudiC
Please also consult introductory text books and / or man pages.
I think I made it unambiguously clear when I referenced man pages in all of my earlier posts. I suppose you have a solid tech background so I guess for you the language of man pages is anything but non-perceptible. However for many people lacking that level of expertise some phrases, terms, word choices, word combinations and the overall jargon those pages are written in, feel very painful and so are those "introductory books" written by the extra-class specialists, apparently reckoned on individuals sharing the same level of knowledge, these specialists failing to render the material in a way more acceptable by general audience; on the other hand those few that attempt to do that often fall victims of incoherent, inconsistent, over-simplified, too rushed manner of unfolding of instructional material. I suppose this forums were not designed to serve concerns of a tiny stratum of élite expertise bearing IT-specialists only. Everyone can read any manual or watch a tutorial but not everyone is guaranteed to understand it. That's why forums exist: for these persons to get opportunity to raise their issues and expect their questions to be addressed. And if not then what's the point of the forums such as this one at all?

I thought (and I really meant to) I started my thread in "Beginners" section of this forum but for some reason it was moved to Shell and Programming though the very fact of someone asking about xargs by no means implies the person asking it is a shell programmer and neither am I. My level is basic as opposed to Advanced obviously suggested by that title.

Quote:
Originally Posted by RudiC
Now, these are very basic questions from quite broad a range of IT, and I'm not sure I can cover that to satisfaction.
That's the thing, what I asked was not to "cover to full satisfaction" but to give a short outline, a compact account of the notion of the string. I know what string is in AppleScript and the authors were smart enough to explain that, but in all the tutorials on Unix (applicable to bash shell on the Mac) I watched, their authors simply skip this step as overly obvious (for their taste) and thus redundant (for their taste).


Quote:
Originally Posted by RudiC

Then, there are text files, a loosely structured collection of (mostly) printable characters. In *nix systems, those consist of lines of characters terminated by a <new line> (\n, ^J, 0x0A) character. But this is not the only possible text representation. When reading a line from a file, you can put it into a single string variable, or split it into several substrings. If you do so by applying spaces and / or punctuation chars for separating, the substrings will be words. But any other separation is possible albeit not necessarily sensible. So, a line is sort of a superset of (a group of) strings.
Ok, that basically conforms to my understanding however some clarification needed regarding standard in-/out -put first and foremost:
1. Lines are delimited by using one of those new line characters.
2. Strings are determined by splitting one string of the read data (which to put it simply is an entire output of a read command - in this case standard output) to many subsets/substrings.
3. Any punctuation, whitespace char, save new line/zero, sets new string which is a word (because these delimiters are by default obviuosly).
4. Any other separator also sets a new string.
5.
Quote:
Originally Posted by RudiC
A line is sort of a superset of (a group of) strings
That's interesting for I could see it conversely, based on points 1-4: a string is a superset of lines since a line of characters delimited by an appropriate symbol constructs a string because every non-separated character or/and word is a part of a bigger string item (that could be turned into a substring), especially if reading a text file returns none other than string. No? I'm thinking in terms of stdin and stdout.


Returning to the main topic:

I understand that xargs only passes arguments to another command. What do they mean by replstr and replacement arguments (no more than 5 of replacement arguments if we use -I option without specifying that -R sub option)? What is replaced with what and where? Can I replace certain text items of input? Wouldn't it be wiser to use sed?

I saw even this (read a file which contents are a list of fruits and append a new words (constantly repeating like "I like") to an every line containing a name of a fruit):

Code:
cat fruits.txt | xargs -I {} echo "I like {}"

, where {} is some placeholder (and in maths and AppleScript "{}" actually means a multitude and a list respectively and these two are related notions).

The result (stdout):

I like banana
I like apple
I like pear
..etc (10 more fruit names)

What's the technique? I see that -R is not specified, what does that tell us? "Specify max number of arguments that -I will do replacements in": yeah, but what's the class of a value R takes?


In the way the description is put it's not immediately clear for me how to use these options.

P.S. The first 2 screenshots of xargs man pages as shown in Dash for Mac.
Understanding Xargs-man_refpng
Understanding Xargs-man_ref2png
Understanding Xargs-thesaurus_wordcheckpng

Last edited by scrutinizerix; 12-28-2017 at 07:02 PM..
# 7  
Old 12-28-2017
A single sentence in a man page doesn't stand alone. If you look at the SYNOPSIS section of the xargs man page on macOS you'll see:
Code:
SYNOPSIS
     xargs [-0opt] [-E eofstr] [-I replstr [-R replacements]] [-J replstr]
           [-L number] [-n number [-x]] [-P maxprocs] [-s size]
           [utility [argument ...]]

Every time you see "replstr" on this man page as underlined text, it will be referring to the option-argument given to a -I or a -J option. Every time you see "replacements" on this man page as underlined text, it will be referring to the option-argument to the -R option. As noted on the description of the -I option, if no -R option is present on the command line, the value of "replacements" defaults to 5.

In the command line:
Code:
cat fruits.txt | xargs -I {} echo "I like {}"

or the better command line (using fewer system resources, running faster, and producing the same output):
Code:
xargs -I {} echo "I like {}" < fruits.txt

there are two occurrences of the string {}: first occurrence is as the option-argument to the -I option and second is in the quoted argument to the echo command. With the -I {} option, each time {} appears (or occurs) in the command line arguments that xargs will be executing, it will be replaced by the contents of the current line of input read from standard input. So, if you get:
Code:
I like banana
I like apple
I like pear
...

from one of the above command lines, you'll get:
Code:
I like banana, I really like banana
I like apple, I really like apple
I like pear, I really like pear
...

from the command line:
Code:
xargs -I {} echo "I like {}, I really like {}" < fruits.txt

in other words, each occurrence of the -I option-argument in the command line arguments will be replaced by a copy of the current input line.

Does this help?
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Xargs

Hi, can anyone tell me in detail ? what the following do in detail ? I am trying to get a largest number in a list Thanks Tao LARGEST=$(echo $* | xargs -n1 | sort -nr | tail -1) (3 Replies)
Discussion started by: ccp
3 Replies

2. Shell Programming and Scripting

Xargs

Hello, I need some help with xargs $ ls aaa bbb ccc ddd$ ls | xargs -I{} ls -la {} -rw-rw-r--. 1 xxx xx 0 May 30 20:04 aaa -rw-rw-r--. 1 xxx xx 0 May 30 20:04 bbb -rw-rw-r--. 1 xxx xx 0 May 30 20:04 ccc -rw-rw-r--. 1 xxx xx 0 May 30 20:04 dddit's possible to have output like this with... (3 Replies)
Discussion started by: vikus
3 Replies

3. Shell Programming and Scripting

Help with xargs

Using the bash shell I'm trying to either create a command for the command line or a script that will show netstat info for a given process name. Here is an example of what I'm trying to do:$ ps aux |grep catalina |grep -v grep | awk '{print $2}' 5132 $ netstat -nlp |grep 5132 (Not all processes... (11 Replies)
Discussion started by: axiopisty
11 Replies

4. Shell Programming and Scripting

xargs

I have dir with many files ( close to 4M ) . $ ls -la total 76392 drwxr-xr-x 10 oracle dba 512 Jun 06 14:39 . drwxr-xr-x 11 oracle dba 512 Dec 20 13:21 .. drwxr-xr-x 2 oracle dba 39074816 Jun 15 14:07 ad I am trying to delete them using... (8 Replies)
Discussion started by: talashil
8 Replies

5. Shell Programming and Scripting

Help with xargs

hi Could any one please tell me the option using which we can run multiple commands using xargs I have list of files, I want to run dos2unix and chmod at one shot on them I tried google n searched man pages but couldnt really find the solution , please help right now im doing this ls... (4 Replies)
Discussion started by: sunilmenhdiratt
4 Replies

6. Shell Programming and Scripting

Xargs and

Hello there, Let me show you a simple example of what I am trying to achieve: 1) I have an input text file with some lines: 1 a 2 b 3 c 2) And I want to run a command with these lines as arguments (+ arbitrary extra arguments). For example: $ command "1 a" "2 b" "3 c" "bye" I... (7 Replies)
Discussion started by: tokland
7 Replies

7. Shell Programming and Scripting

Help in using xargs

Hi, I have a requirement to RCP the files from remote server to local server. Also the RCP has to run in parallel. However using 'xargs' retrives 2 file names during each loop. How do we restrict to only one file name using xargs and loop till remaining files. I use the below code for... (2 Replies)
Discussion started by: senthil3d
2 Replies

8. Shell Programming and Scripting

Using xargs

hi i just want to know that how do we use xargs command to find files which are greater than specified memory in a given directory (6 Replies)
Discussion started by: sumit the cool
6 Replies

9. UNIX for Advanced & Expert Users

xargs -P

I discovered that GNU's xargs has a -P option to allow its processes to run in parallel. Great! Is this a GNU thing, or is it supported by other platforms as well? (4 Replies)
Discussion started by: otheus
4 Replies

10. UNIX for Dummies Questions & Answers

xargs

Can I use xargs to send a list of commands to a process, to be acted upon individually? Here's what I have: a file that contains numbers, one per line. The desired outcome it to send each number to a DB2 query. I thought xargs would work, but it doesn't. I tried it like this: cat file | xargs |... (4 Replies)
Discussion started by: jpprial
4 Replies
Login or Register to Ask a Question