Presentor's BLOG
dtrace.org/blog/brandon
JOYENT
@brendangregg
System metrix
-- iostat
UPtime -- Load avg. 1/5 / 15mins
Only a clue, don't stress about load svgs, use other tools
TOP
1. Top can consume cpu
-- Does top show all cpu consumers?
-- Can miss short lived processes may not list kernel threads unless included
-- A process has high cpu - what are next steps
-- identify why -- profile code path
-- identify what -- execution or stall cycles
Htop - SUper top
MPSTAT --- Check for hot threads, on multi core/processors
iostat -- disk/io
vmstat - virtual memory statistics (started from bad days)
-- vmstat 1
summary since boot values - first line
free
-buffer cache for filesystem
ping
-- network might behave differently for ICMP packets
hping - try using TCP
TCP
nicstat - network statistics tool
-- nicstat -z 1
The presenter created this tool - Tim Cook ported and enhanced it on LINUX
dstat - better vmstat tool (Does coloring) - FWIW
-- dstat 1
-== INTERMEDIATE TOOLS =---
-- SAR
- PRETTY GOOD ON LINUX BUT HAS impression of being buggy.
- configure to archive stats from cron
- -d block device stats
- -q run queue statistics
-- NETSTAT
-- netstat -s
Active connection openings
passive connection openings
failed connection attempts
- Calculate the retransmission rates
- high retransmission rates for mobile phones
-- PIDSTAT
- Very useful process breakdowns:
-- pidstat 1
-- pidstat -d 1
- shows disk i/o
-- STRACE
- system call tracer (attach to process or a command)
-- strace -tttT -p 12670
- resource i/o
- memory mapped i/o can't be done through strace as there is no sys call
-- strace -c dd if=/dev/zero of=/dev/null bs=512 count =1024k
- worst case
-- time strace -c dd if=/dev/zero of=/dev/null bs=512 count =1024k
-- TCPDUMP
-- tcpdump -i eth4 -w /tmp/out.tcpdump
does have overhead in terms of cpu and storage
should use socket ring buffers
use filter expressions to reduce overhead
still be problematic for busy interfaces
tricky for 10gb ethernet
BLKTRACE - block device i/o event tracing.
-- btrace /dev/sdb
IOTOP - for disk i/o *** authors preferred**
-- iotop -bod5
kbytes is not universal currenty, time is.
SLABTOP -
-- kernel slap allocator usage
-- slabtop -sc
- shows where kernel memory is consumed (tool used by kernel developers)
- gives good reason why the memory is used,
SYSCTL - static performance tuning: Check the config of the system (even without workload)
-- sysctl -a
- check this even if no load, cause there might be something you would overlook.
-=== TOOLS ADVANCED
* PERF
- performance counters for linux (PCL) focusing on CPU performance counters (Programable registers)
- now collection of profiling and tracing tools, which numerous subcommand, including:
kmem - reace meause kernel mem
kvm trace kvm gues os
list lis available events (targets of instrumentation)
- key performance counter summary
#perf stat gzip file1
- low IPC (<0.2) means stall cycles (likely memory); look for ways to reduce memory i/o and improve locality (NUMA)
#perf lis |grep Hardware
# perf stat -e instructions, cycles, L1-dcache
(BIOS developer /kernel developer user guide)
( PROFILES CPU ACTIVITIES ALSO)
- profileing (sampling)
# perf record -a -g -F 997 sleep 10
- focusing on non-interactive mode
# perf report --stdio
- profilling/graphs (FLAME GRAPHS)
- generates .svg file interactive in browser
- (as performance engineer its very important - exonerate the areas that are culprit)
- Static Tracing
# perf list | grep block:
- shows block i/o trace points
- where the trace points instruments (pic in slide)
- Dynamic Tracing
# perf probe --add='tcp_sendmsg'
# perf record -e probe:tcp_sendmsg -aR -g sleep 5
# perf report --stdio
active traced call stacks from arbitrary kernel locations
= fills in kernel observability gaps
- awesome capability
- takes some effort to use
** DTRACE **
- programmable real-time - dynamic and static tracing
- pert analysis and troubleshooting without restarting anything
- used on solaris, illumos/SmartOS, mac OS X, freeBSD
- two ports in development for LINUX
1. dtrace4linux - paul fox (ubuntu, fedora, centos)
2. Oracle Enterprise Linux DTRACE
- steady progress
- dtrace 4 linux version
- github.com/dtrace4linux/dtrace
-- make load
- warning: still a prototype, can panic/
- programming cabapilities
# dtrace -n 'fbt::tcp_sendmsg:entry /execname =="sshd"/ { @["bytes"] = quantize(arg3); }'
- Multiple GUIs use Dtrace for real-time statistics.
Eg, JOYENET CLOUD ANALYTICS, showing real-time cloud-wide sys call latency:
- advanced capabilities, not difficult
- use one-liners (google "DTrace one-liners")
- use scripts (DTRace Toolkit; Dtrace book; google)
- tweak one-liners or scripts a little
- ask someone else to write
EXAMPLES
-fbt::vfs_read:entry
{ self->start=tmestamp; }
CLI Syntax
dtrace.org/guide
-Providers IO,CPC,VMINFO,IP,FBT-function boundary tracing(kernel dynamic tracing)
the presenter has a script for dtrace in github for this
dtrace:ext4slower.d 10 (list process thawith i/o latency more than given parameter ), clearly exonerates or blames the filesystem (ext4)
Show me
- TCP retransmits (only trace retransmits and not full dump)
- dtrace:tcpretransmit.d (in github soon)
illumos/SmartOS (is the on-going fork of opensolaris kernel)
IOSNOOP - shows disk/ i/o (mac ships it with osx)
SYSTEM TAP
- created when there wasn't Dtrace for linux ports
- can't use in LInux prod systems because of un-reliability
METHODOLOGIES
- SELECTED FOUR
- PICK OBSERVABILITY TOOLS
- ITS INEFFICIENT
- WORKLOAD CHARACTERIZATION METHOD
- WHO WHAT WHY AND WHEN - IDENTIFIES ISSUES OF LOAD
- best performance wins are from eliminating
- DRILL-Down Analysis
- start from top and get down and find root cause
- USE Method
- for every roursore find
- utilization
- saturation
- Errors
- mutex locks
- threads pools
Sunday, February 24, 2013
Saturday, February 9, 2013
Fwd: Open Source Term Apps
15 Greatest Open Source Terminal Applications Of 2012
by NIXCRAFT on DECEMBER 11, 2012 · 61 COMMENTS· LAST UPDATED JANUARY 7, 2013
Linux on the desktop is making great progress. However, the real beauty of Linux and Unix like operating system lies beneath the surface at the command prompt. nixCraft picks his best open source terminal applications of 2012.
Most of the following tools are packaged by all major Linux distributions and can be installed on *BSD or Apple OS X.#1: siege - An HTTP/HTTPS stress load tester
Siege is a multi-threaded http or https load testing and benchmarking utility. This tool allows me to measure the performance of web apps under duress. I often use this tool test a web server and apps. I have had very good results with this tool. It can stress a single url such as example.com/foo.php or multiple urls. At the end of each test you will get all data about the web server performance, total data transferred, latency, server response time, concurrency and much more.#2: abcde - A better CD encoder
Usually, the process of grabbing the data off a CD and encoding it, then tagging or commenting it, is very involved. abcde is designed to automate this. It will take an entire CD and convert it into a compressed audio format - Ogg/Vorbis, MPEG Audio Layer III, Free Lossless Audio Codec (FLAC), Ogg/Speex, MPP/MP+(Musepack) and/or M4A (AAC) format(s). It will do a CDDB query over the Internet to look up your CD or use a locally stored CDDB entry.
#3: ngrep - Network grep
Ngrep is a network packet analyzer. It follows most of GNU grep's common features, applying them to the network layer. Ngrep is not related to tcpdump. It is just an easy to use tool. You can run queries such as:## grep all HTTP GET or POST requests from network traffic on eth0 interface ## sudo ngrep -l -q -d eth0 "^GET |^POST " tcp and port 80I often use this tool to find out security related problems and tracking down other network and server related problems.
#4: pv
The pv command allows you to see the progress of data through a pipeline. It provides the following info:
- Time elapsed
- Percentage completed (with progress bar)
- Current throughput rate
- Total data transferred
- ETA
See how to install and use pv command under Linux. Or download pv by visiting this page.
#5: dtrx
dtrx is an acronmy for "Do The Right Extraction." It's a tool for Unix-like systems that take all the hassle out of extracting archives. As a sysadmin, I download source code and tar balls. This tool saves lots of time.
- You only need to remember one simple command to extract tar, zip, cpio, deb, rpm, gem, 7z, cab, lzh, rar, gz, bz2, lzma, xz, and many kinds of exe files, including Microsoft Cabinet archives, InstallShield archives, and self-extracting zip files. If they have any extra compression, like tar.bz2 files, dtrx will take care of that for you, too.
- dtrx will make sure that archives are extracted into their own dedicated directories.
- dtrx makes sure you can read and write all the files you just extracted, while leaving the rest of the permissions intact.
- Recursive extraction: dtrx can find archives inside the archive and extract those too.
- Download dtrx
#6:dstat - Versatile resource statistics tool
As a sysadmin, I heavily depends upon tools such as vmstat, iostat and friends for troubleshooting server issues. Dstat overcomes some of the limitations provided by vmstat and friends. It adds some extra features. It allows me to view all of my system resources instantly. I can compare disk usage in combination with interrupts from hard disk controller, or compare the network bandwidth numbers directly with the disk throughput and much more.#7:ffmpeg - Record, convert, stream and play multimedia content
Recently, I started a youtube channel for nixCraft. I need to convert video and audio in various format such as Youtube HD web streaming format. This tool saves lots of my time. I often use this tool for audio/video conversion. This is the best tool for converting Audio, AVI, MP4, Ipod, Mobile phone, PSP, Quicktime, Rockbox, Web (Flash), WMV and much more.#8:mtr - Traceroute+ping in a single network diagnostic tool
The mtr command combines the functionality of the traceroute and ping programs in a single network diagnostic tool. Use mtr to monitor outgoing bandwidth, latency and jitter in your network. A great little app to solve network problems. If you see a sudden increase in packetloss or response time is often an indication of a bad or simply overloaded link.#9:multitail - Tail command on steroids
MultiTail is a program for monitoring multiple log files, in the fashion of the original tail program. This program lets you view one or multiple files like the original tail program. The difference is that it creates multiple windows on your console (with ncurses). I often use this tool when I am monitoring logs on my server.#10: curl - Transfer data and see behind the scenes
Curl is a command line tool to transfer data from or to a server, using one of the supported protocols. The command is designed to work without user interaction. curl offers a busload of useful tricks like proxy support, user authentication, FTP upload, and much more. I often usecurl command to:
- Troubleshoot http/ftp/cdn server problems.
- Check or pass HTTP/HTTPS headers.
- Upload / download files using ftp protocol or to cloud account.
- Debug HTTP responses and find out exactly what an Apache/Nginx/Lighttpd/IIS server is sending to you without using any browser add-ons or 3rd party applications.
- Download curl
#11: netcat - TCP/IP swiss army knife
Netcat or nc is a simple Linux or Unix command which reads and writes data across network connections, using TCP or UDP protocol. I often use this tool to open up a network pipe to test network connectivity, make backups, bind to sockets to handle incoming / outgoing requests and much more. In this example, I tell nc to listen to a port # 3005 and execute /usr/bin/w command when client connects and send data back to the client:$ nc -l -p 3005 -e /usr/bin/w
From a different system try to connect to port # 3005:$ telnet server1.cyberciti.biz.lan 3005#12: nmap - Offensive and defensive network security scanner
Nmap is short for Network Mapper. It is an open source security tool for network exploration, security scanning and auditing. However, nmap command comes with lots of options that can make the utility more robust and difficult to follow for new users.#13: openssl command line tool
The openssl command is used for the various cryptography functions of OpenSSL's crypto library from the shell. I often use this tool to encrypt files, test/verify ssl connections, and check the integrity of downloaded files. Further, openssl can be used for:
- Creation of RSA, DH and DSA key parameters
- Creation of X.509 certificates, CSRs and CRLs
- Calculation of Message Digests
- Handling of S/MIME signed or encrypted mail
The following few examples demonstrate the power of openssl command:
File integrity verification (cryptographic hashing function)
Verify that a file called financial-records-fy-2011-12.dbx.aes has not been tampered with:
openssl dgst -sha1 -c financial-records-fy-2011-12.dbx.aes openssl dgst -ripemd160 -c financial-records-fy-2011-12.dbx.aes openssl dgst -md5 -c financial-records-fy-2011-12.dbx.aesSample outputs from the last command:
MD5(financial-records-fy-2011-12.dbx.aes)= d4:1d:8c:d9:8f:00:b2:04:e9:80:09:98:ec:f8:42:7eEncryption and Decryption with Ciphers (files)
## encrypt file ## openssl aes-256-cbc -salt -in financial-records-fy-2011-12.dbx -out financial-records-fy-2011-12.dbx.aes ## decrypt file ## openssl aes-256-cbc -d -in financial-records-fy-2011-12.dbx.aes -out financial-records-fy-2011-12.dbxSSL/TLS client and server tests
## connect to gmail mail server for testing purpose ## openssl s_client -connect smtp.gmail.com:995 openssl s_client -connect smtp.gmail.com:995 -CApath /etc/ssl#14: lftp: A better command-line ftp/http/sftp client
This is the best and most sophisticated sftp/ftp/http download and upload client program. I often use this tool to:
- Recursively mirroring entire directory trees from a ftp server
- Accelerate ftp / http download speed
- Location bookmarks and resuming downloads.
- Backup files to a remote ftp servers.
- Transfers can be scheduled for execution at a later time.
- Bandwidth can be throttled and transfer queues can be set up.
- Lftp has shell-like command syntax allowing you to launch several commands in parallel in background (&).
- Segmented file transfer, that allows more than one connection for the same file.
- And much more.
- Download lftp
#15: Irssi - IRC client
Irssi is a modular Internet Relay Chat client. It is highly extensible and very secure. Being a fullscreen, termcap based client with many features, Irssi is easily extensible through scripts and modules. I often use this client to get help about certain problmes from IRC rooms or just to hang out with old buddies.#16: Rest...
- Mutt - Email client and I often use mutt to send email attachments from my shell scripts.
- bittorrent - Command line torrent client.
- screen - A full-screen window manager and must have tool for all *nix admins.
- rsync - Sync files and save bandwidth.
- sar - Old good system activity collector and reporter.
- lsof - List open files.
- vim - Best text editor ever.
- elinks or lynx - I use this browse remotely when some sites (such as RHN or Novell or Sun/Oracle) require registration/login before making downloads.
- wget - Best download tool ever. I use wget all the time, even with Gnome desktop.
- mplayer - Best console mp3 player that can play any audio file format.
- newsbeuter - Text mode rss feed reader with podcast support.
- parallel - Build and execute shell command lines from standard input in parallel.
- iftop - Display bandwidth usage on network interface by host.
- iotop - Find out what's stressing and increasing load on your hard disks.
Conclusion
This is my personal FOSS terminal apps list and it is not absolutely definitive, so if you've got your own terminal apps, share in the comments below.
Subscribe to:
Posts (Atom)