Home Man
Today's Posts

Linux & Unix Commands - Search Man Pages

CentOS 7.0 - man page for trace-cmd-record (centos section 1)


       trace-cmd-record - record a trace from the Ftrace Linux internal tracer

       trace-cmd record [OPTIONS] [command]

       The trace-cmd(1) record command will set up the Ftrace Linux kernel tracer to record the
       specified plugins or events that happen while the command executes. If no command is
       given, then it will record until the user hits Ctrl-C.

       The record command of trace-cmd will set up the Ftrace tracer to start tracing the various
       events or plugins that are given on the command line. It will then create a number of
       tracing processes (one per CPU) that will start recording from the kernel ring buffer
       straight into temporary files. When the command is complete (or Ctrl-C is hit) all the
       files will be combined into a trace.dat file that can later be read (see

       -p plugin
	   Specify a trace plugin. Plugins are special Ftrace tracers that usually do more than
	   just trace an event. Common plugins are function, function_graph, preemptirqsoff,
	   irqsoff, preemptoff, and wakeup. A plugin must be supported by the running kernel. To
	   see a list of available plugins, see trace-cmd-list(1).

       -e event
	   Specify an event to trace. Various static trace points have been added to the Linux
	   kernel. They are grouped by subsystem where you can enable all events of a given
	   subsystem or specify specific events to be enabled. The event is of the format
	   "subsystem:event-name". You can also just specify the subsystem without the
	   :event-name or the event-name without the "subsystem:". Using "-e sched_switch" will
	   enable the "sched_switch" event where as, "-e sched" will enable all events under the
	   "sched" subsystem.

	       The 'event' can also contain glob expressions. That is, "*stat*" will
	       select all events (or subsystems) that have the characters "stat" in their

	       The keyword 'all' can be used to enable all events.

	   Every event that is being recorded has its output format file saved in the output file
	   to be able to display it later. But if other events are enabled in the trace without
	   trace-cmd's knowledge, the formats of those events will not be recorded and trace-cmd
	   report will not be able to display them. If this is the case, then specify the -a
	   option and the format for all events in the system will be saved.

	   Enable a stacktrace on each event. For example:

			 <idle>-0     [003] 58549.289091: sched_switch: 	kworker/0:1:0 [120] R ==> trace-cmd:2603 [120]
			 <idle>-0     [003] 58549.289092: kernel_stack: 	<stack trace>
	       => schedule (ffffffff814b260e)
	       => cpu_idle (ffffffff8100a38c)
	       => start_secondary (ffffffff814ab828)

	   Enable a stack trace on all functions. Note this is only applicable for the "function"
	   plugin tracer, and will only take effect if the -l option is used and succeeds in
	   limiting functions. If the function tracer is not filtered, and the stack trace is
	   enabled, you can live lock the machine.

       -f filter
	   Specify a filter for the previous event. This must come after a -e. This will filter
	   what events get recorded based on the content of the event. Filtering is passed to the
	   kernel directly so what filtering is allowed may depend on what version of the kernel
	   you have. Basically, it will let you use C notation to check if an event should be
	   processed or not.

	       ==, >=, <=, >, <, &, |, && and ||

	   The above are usually safe to use to compare fields.

	   This will cause all events specified after it on the command line to not be traced.
	   This is useful for selecting a subsystem to be traced but to leave out various events.
	   For Example: "-e sched -v -e "*stat\*"" will enable all events in the sched subsystem
	   except those that have "stat" in their names.

	       Note: the *-v* option was taken from the way grep(1) inverts the following

	   This will filter only the executable that is given on the command line. If no command
	   is given, then it will filter itself (pretty pointless). Using -F will let you trace
	   only events that are caused by the given command.

       -P pid
	   Similar to -F but lets you specify a process ID to trace.

	   Used with either -F to trace the process' children too.

       -o output-file
	   By default, trace-cmd report will create a trace.dat file. You can specify a different
	   file to write to with the -o option.

       -l function-name
	   This will limit the function and function_graph tracers to only trace the given
	   function name. More than one -l may be specified on the command line to trace more
	   than one function. The limited use of glob expressions are also allowed. These are
	   match* to only filter functions that start with match.  *match to only filter
	   functions that end with match.  *match\* to only filter on functions that contain

       -g function-name
	   This option is for the function_graph plugin. It will graph the given function. That
	   is, it will only trace the function and all functions that it calls. You can have more
	   than one -g on the command line.

       -n function-name
	   This has the opposite effect of -l. The function given with the -n option will not be
	   traced. This takes precedence, that is, if you include the same function for both -n
	   and -l, it will not be traced.

	   Some tracer plugins enable the function tracer by default. Like the latency tracers.
	   This option prevents the function tracer from being enabled at start up.

       -O option
	   Ftrace has various options that can be enabled or disabled. This allows you to set
	   them. Appending the text no to an option disables it. For example: "-O nograph-time"
	   will disable the "graph-time" Ftrace option.

       -s interval
	   The processes that trace-cmd creates to record from the ring buffer need to wake up to
	   do the recording. Setting the interval to zero will cause the processes to wakeup
	   every time new data is written into the buffer. But since Ftrace is recording kernel
	   activity, the act of this processes going back to sleep may cause new events into the
	   ring buffer which will wake the process back up. This will needlessly add extra data
	   into the ring buffer.

	       The 'interval' metric is microseconds. The default is set to 1000 (1 ms).
	       This is the time each recording process will sleep before waking up to
	       record any new data that was written to the ring buffer.

       -r priority
	   The priority to run the capture threads at. In a busy system the trace capturing
	   threads may be staved and events can be lost. This increases the priority of those
	   threads to the real time (FIFO) priority. But use this option with care, it can also
	   change the behaviour of the system being traced.

       -b size
	   This sets the ring buffer size to size kilobytes. Because the Ftrace ring buffer is
	   per CPU, this size is the size of each per CPU ring buffer inside the kernel. Using
	   "-b 10000" on a machine with 4 CPUs will make Ftrace have a total buffer size of 40

	   By default, when trace-cmd is finished tracing, it will reset the buffers and disable
	   all the tracing that it enabled. This option keeps trace-cmd from disabling the tracer
	   and reseting the buffer. This option is useful for debugging trace-cmd.

	       Note: usually trace-cmd will set the "tracing_on" file back to what it
	       was before it was called. This option will leave that file set to zero.

	   By default, if an event is listed that trace-cmd does not find, it will exit with an
	   error. This option will just ignore events that are listed on the command line but are
	   not found on the system.

       -N host:port
	   If another machine is running "trace-cmd listen", this option is used to have the data
	   sent to that machine with UDP packets. Instead of writing to an output file, the data
	   is sent off to a remote box. This is ideal for embedded machines with little storage,
	   or having a single machine that will keep all the data in a single repository.

	       Note: This option is not supported with latency tracer plugins:
		 wakeup, wakeup_rt, irqsoff, preemptoff and preemptirqsoff

	   This option is used with -N, when there's a need to send the live data with TCP
	   packets instead of UDP. Although TCP is not nearly as fast as sending the UDP packets,
	   but it may be needed if the network is not that reliable, the amount of data is not
	   that intensive, and a guarantee is needed that all traced information is transfered

	   With the --date option, "trace-cmd" will write timestamps into the trace buffer after
	   it has finished recording. It will then map the timestamp to gettimeofday which will
	   allow wall time output from the timestamps reading the created trace.dat file.

       The basic way to trace all events:

	    # trace-cmd record -e all ls > /dev/null
	    # trace-cmd report
		  trace-cmd-13541 [003] 106260.693809: filemap_fault: address=0x128122 offset=0xce
		  trace-cmd-13543 [001] 106260.693809: kmalloc: call_site=81128dd4 ptr=0xffff88003dd83800 bytes_req=768 bytes_alloc=1024 gfp_flags=GFP_KERNEL|GFP_ZERO
			 ls-13545 [002] 106260.693809: kfree: call_site=810a7abb ptr=0x0
			 ls-13545 [002] 106260.693818: sys_exit_write:	     0x1

       To use the function tracer with sched switch tracing:

	    # trace-cmd record -p function -e sched_switch ls > /dev/null
	    # trace-cmd report
			 ls-13587 [002] 106467.860310: function: hrtick_start_fair <-- pick_next_task_fair
			 ls-13587 [002] 106467.860313: sched_switch: prev_comm=trace-cmd prev_pid=13587 prev_prio=120 prev_state=R ==> next_comm=trace-cmd next_pid=13583 next_prio=120
		  trace-cmd-13585 [001] 106467.860314: function: native_set_pte_at <-- __do_fault
		  trace-cmd-13586 [003] 106467.860314: function:	     up_read <-- do_page_fault
			 ls-13587 [002] 106467.860317: function:	     __phys_addr <-- schedule
		  trace-cmd-13585 [001] 106467.860318: function: _raw_spin_unlock <-- __do_fault
			 ls-13587 [002] 106467.860320: function: native_load_sp0 <-- __switch_to
		  trace-cmd-13586 [003] 106467.860322: function: down_read_trylock <-- do_page_fault

       Here is a nice way to find what interrupts have the highest latency:

	    # trace-cmd record -p function_graph -e irq_handler_entry  -l do_IRQ sleep 10
	    # trace-cmd report
		     <idle>-0	  [000] 157412.933969: funcgraph_entry: 		 |  do_IRQ() {
		     <idle>-0	  [000] 157412.933974: irq_handler_entry:    irq=48 name=eth0
		     <idle>-0	  [000] 157412.934004: funcgraph_exit:	     + 36.358 us |  }
		     <idle>-0	  [000] 157413.895004: funcgraph_entry: 		 |  do_IRQ() {
		     <idle>-0	  [000] 157413.895011: irq_handler_entry:    irq=48 name=eth0
		     <idle>-0	  [000] 157413.895026: funcgraph_exit:			      + 24.014 us |  }
		     <idle>-0	  [000] 157415.891762: funcgraph_entry: 		 |  do_IRQ() {
		     <idle>-0	  [000] 157415.891769: irq_handler_entry:    irq=48 name=eth0
		     <idle>-0	  [000] 157415.891784: funcgraph_exit:	     + 22.928 us |  }
		     <idle>-0	  [000] 157415.934869: funcgraph_entry: 		 |  do_IRQ() {
		     <idle>-0	  [000] 157415.934874: irq_handler_entry:    irq=48 name=eth0
		     <idle>-0	  [000] 157415.934906: funcgraph_exit:	     + 37.512 us |  }
		     <idle>-0	  [000] 157417.888373: funcgraph_entry: 		 |  do_IRQ() {
		     <idle>-0	  [000] 157417.888381: irq_handler_entry:    irq=48 name=eth0
		     <idle>-0	  [000] 157417.888398: funcgraph_exit:	     + 25.943 us |  }

       trace-cmd(1), trace-cmd-report(1), trace-cmd-start(1), trace-cmd-stop(1),
       trace-cmd-extract(1), trace-cmd-reset(1), trace-cmd-split(1), trace-cmd-list(1),

       Written by Steven Rostedt, <rostedt@goodmis.org[1]>


       Copyright (C) 2010 Red Hat, Inc. Free use of this software is granted under the terms of
       the GNU Public License (GPL).

	1. rostedt@goodmis.org

					    06/11/2014			      TRACE-CMD-RECORD(1)

All times are GMT -4. The time now is 03:24 AM.

Unix & Linux Forums Content Copyrightę1993-2018. All Rights Reserved.
Show Password