Sysdig

Sysdig Features:

It is fast, stable and easy-to-use with comprehensively well documented.
Comes with native support for container technologies, including Docker, LXC.
It is scriptable in Lua; offers chisels (lightweight Lua scripts) for processing captured system events.
Supports useful filtering of output.
Supports system and application tracing.
It can be integrated with Ansible, Puppet, and Logstash.
Enable sample advanced log analysis.
It also offers Linux server attack (forensics) analysis features for ethical hackers and lot’s more.

How To Install Sysdig in Linux

# yum install dkms
# curl -s https://s3.amazonaws.com/download.draios.com/stable/install-sysdig | bash

Sysdig support:-

Capture system events, filter them, run the useful script.
Strace + tcpdump + lsof + htop + Lua
Open source
Native support for containers.

Capture every event and write it to standard output

[root@feenixdv ~]# sysdig

7 02:18:23.758037421 0 sysdig (50839) > switch next=9 pgft_maj=0 pgft_min=4531 vm_size=74000 vm_rss=8412 vm_swap=0

8 02:18:23.758042728 0 <NA> (9) > switch next=373 pgft_maj=0 pgft_min=0 vm_size=0 vm_rss=0 vm_swap=0

9 02:18:23.758047151 0 <NA> (373) > switch next=0 pgft_maj=0 pgft_min=0 vm_size=0 vm_rss=0 vm_swap=0

10 02:18:23.759045764 0 <NA> (0) > switch next=3 pgft_maj=0 pgft_min=0 vm_size=0 vm_rss=0 vm_swap=0

11 02:18:23.759088619 0 <NA> (3) > switch next=9 pgft_maj=0 pgft_min=0 vm_size=0 vm_rss=0 vm_swap=0

12 02:18:23.759092985 0 <NA> (9) > switch next=0 pgft_maj=0 pgft_min=0 vm_size=0 vm_rss=0 vm_swap=0

13 02:18:23.760184013 0 <NA> (0) > switch next=9 pgft_maj=0 pgft_min=0 vm_size=0 vm_rss=0 vm_swap=0

14 02:18:23.760196200 0 <NA> (9) > switch next=3 pgft_maj=0 pgft_min=0 vm_size=0 vm_rss=0 vm_swap=0

15 02:18:23.760252153 0 <NA> (3) > switch next=0 pgft_maj=0 pgft_min=0 vm_size=0 vm_rss=0 vm_swap=0

16 02:18:23.765874541 0 <NA> (0) > switch next=3717(python) pgft_maj=0 pgft_min=0 vm_size=0 vm_rss=0 vm_swap=0

17 02:18:23.765884914 0 python (3717) < poll res=0 fds=

By default, sysdig prints the information for each event on a single line, with the following format:

*%evt.num %evt.time %evt.cpu %proc.name (%thread.tid) %evt.dir %evt.type %evt.args

Where:

evt.num is the incremental event number
evt.time is the event timestamp
evt.cpu is the CPU number where the event was captured
proc.name is the name of the process that generated the event
thread.tid is the TID that generated the event, which corresponds to the PID for single thread processes
evt.dir is the event direction, > for enter events and < for exit events
evt.type is the name of the event, e.g. 'open' or 'read'
evt.args is the list of event arguments. In case of system calls, these tend to correspond to the system call arguments, but that’s not always the case: some system call arguments are excluded for simplicity or performance reasons.

Top 10 files, where data are written in files max.

[root@feenixdv ~]# sysdig -pc -c topfiles_bytes
Bytes               container.name      Filename
--------------------------------------------------------------------------------
24.05KB             host                /var/lib/grafana/grafana.db
1.33KB              cadvisor            /rootfs/proc/1/net/dev
1.27KB              host                /dev/ptmx
1.17KB              host                /proc/989/status
665B                cadvisor            /sys/fs/cgroup/memory/system.slice/grafana-server.service/memory.stat
661B                cadvisor            /sys/fs/cgroup/memory/docker/1830625ea4d56e2ef008929317f/memory.stat
659B                cadvisor            /sys/fs/cgroup/memory/memory.stat
655B                cadvisor            /sys/fs/cgroup/memory/docker/07496d62108d1ee479bc742e257/memory.stat
655B                cadvisor            /sys/fs/cgroup/memory/system.slice/prometheus.service/memory.stat
651B                cadvisor            /sys/fs/cgroup/memory/system.slice/docker.service/memory.stat

Dig by process name

[root@feenixdv ~]# sysdig proc.name=cadvisor

323613 02:38:14.544503024 0 cadvisor (4049) > epoll_ctl

323614 02:38:14.544503453 0 cadvisor (4049) < epoll_ctl

323615 02:38:14.544505114 0 cadvisor (4049) > epoll_ctl

323616 02:38:14.544505489 0 cadvisor (4049) < epoll_ctl

323617 02:38:14.544507992 0 cadvisor (4049) > fstat fd=10(<f>/sys/fs/cgroup/memory/system.slice/grafana-server.service/memory.kmem.tcp.max_usage_in_bytes)

323618 02:38:14.544508585 0 cadvisor (4049) < fstat res=0

323619 02:38:14.544510297 0 cadvisor (4049) > read fd=10(<f>/sys/fs/cgroup/memory/system.slice/grafana-server.service/memory.kmem.tcp.max_usage_in_bytes) size=512

323620 02:38:14.544511269 0 cadvisor (4049) < read res=2 data=0.

csysdig the ncurses user interface for sysdig

Not the same as “top” command. Details with container level.

Press “F2” for more option to select monitoring.

Identify user data from the SSH session.

In this lab, I am going to log in with “feenixdv” user and see on screen all action with the help of “sysdig”.

Press “ctrl+F” then type the user name. In my case its “feenixdv”

Here we can see one process with “feenixdv” user. Now press “Enter” and then press “F5” for echo or see the output.

Here we can see, from user “feenixdv” user fire command “cat > test” on his session and we have the same output on echo screen.