3 Step to keep track OS memory usage by process

Tutorial

So many people asking how to keep track on the OS memory usage by process, as you know a bit about Physical Memory, Virtual Memory (VSZ), Resident Set Size (RSS), Shared Memory, Paging Memory…etc, counting RSS to determine memory usage is very common way and easier to understand for dummies like me, although it’s not a precise figure while you’re ignoring shared memory, fs cache..etc.
But anyway this tutorial is starting point to help you monitor the process RSS usage and keep tracking the usage that allows to identify peak/min. 
Alright, you need to create one database or table, and one script and one cron job.

Step 1 – Create a database for storing stat data

CREATE DATABASE sysstat DEFAULT CHARACTER SET latin1; GRANT INSERT, UPDATE, DELETE, SELECT on sysstat.* to sysstat@'localhost' identified by 'passw0rd'; USE sysstat; CREATE TABLE 'memory' (  'id' int(11) NOT NULL auto_increment,  'date' datetime NOT NULL,  'memory'int(20) NOT NULL,  'process' varchar(300) NOT NULL,  PRIMARY KEY  ('id')) ENGINE=MyISAM AUTO_INCREMENT=21794 DEFAULT CHARSET=latin1;

Step 2 – Script

Usage:
To show all memory usage by process 

./memory_stat.sh usage

[root@vm1 sbin]# ./memory_stat.sh usage 2011-05-01 01:55:39 16576 /usr/lib/courier-imap/bin/couriertls 2011-05-01 01:55:39 35576 /usr/lib/courier-imap/bin/imapd 2011-05-01 01:55:39 38832 /usr/sbin/httpd.worker 2011-05-01 01:55:39 73100 MailScanner: 2011-05-01 01:55:39 125632 clamd 2011-05-01 01:55:39 208964 /usr/libexec/mysqld 2011-05-01 01:55:39 429720 /usr/bin/php-cgi

To record into database

./memory_stat.sh record

To retrieve the usage report

./memory_stat.sh report {process} {date - optional}

[root@vm1 sbin]# ./memory_stat.sh report php "2011-05-01 01" php usage report on vm1.dreamerworks.net date    memory 2011-05-01 00:00:01 471588 2011-05-01 00:05:02 468772 2011-05-01 00:10:02 470788 2011-05-01 00:15:01 471392 2011-05-01 00:20:01 472140 2011-05-01 00:25:01 469016 2011-05-01 00:30:02 469016 2011-05-01 00:35:01 469016 2011-05-01 00:40:01 472376

To housekeep the database

./memory_stat.sh housekeep

memory_stat.sh

#!/bin/bash   process=`ps aux|awk '{print $11}'|sort|uniq|grep -v -e grep -e awk -e ps -e sort -e uniq` date=`date +%Y-%m-%d\ %H:%M:%S` database="sysstat" dbuser="sysstat" password="passw0rd" housekeepday="30"   usage(){ for PS in $process; do   echo "$date `ps aux | grep -- $PS |grep -v grep | awk '{sum +=$6}; END {print sum}'` $PS"   done }   record() { usage > /tmp/usage.log mysql -u$dbuser -p$password <<EOF use $database; `while read date time memory process; do echo "insert into memory (date,memory,process) values ('$date $time','$memory','$process');" done < /tmp/usage.log ` EOF   rm -f /tmp/usage.log   }   housekeep() { mysql -u$dbuser -p$password <<EOF use $database; delete from memory where date < CURRENT_DATE - $housekeepday; EOF }   report() { if [ x$1 = "x" ];then         echo "Usage: $0 [process]"         exit fi process=$1 datetime_arg="AND date like '$2%'" echo "$process usage report on `hostname`" mysql -u$dbuser -p$password <<EOF use $database; ` echo "select date, memory from memory where process like '%$1%' $datetime_arg order by date"; ` EOF   }   case $1 in     usage)     usage | sort -k 3n     ;;         housekeep)         housekeep         ;;         record)         record         ;;     report)     report $2 $3     ;;         *)         echo "Usage: $1 (usage|record|housekeep|report [process] [date])"         ;; esac

Step 3 – Cron Job

0,5,10,15,20,25,30,35,40,45,50,55 * * * *       /usr/local/sbin/memory_stat.sh record

That’s it

At least, you may tell a bit more about which process is huger one and then you’ll able to narrow down for investigation.