Monitoring UNIX Application Performance

Valgrind

Application Monitoring is a very important aspect of a project. Unfortunately, not much attention is being is paid to develop an effective monitoring while the project is not live. Once the project is live, the lack of proper monitoring factors in terms of downtime when the support person is not aware if application is having some problems or application not working at all.

In my line of work, we are being constantly bug by the application team why the system is slow or what is the reason of the system crash. Most of the time the cause is related to memory-management and performance problems, leaks/corruption in the program being run.

Memory management is prone to errors that are too hard to detect. Common errors may be listed as:

  1. Use of uninitialized memory
  2. Reading/writing memory after it has been freed
  3. Reading/writing off the end of malloc’d blocks
  4. Reading/writing inappropriate areas on the stack
  5. Memory leaks — where pointers to malloc’d blocks are lost forever
  6. Mismatched use of malloc/new/new[] vs free/delete/delete[]
  7. Some misuses of the POSIX pthreads API

In trying to monitor the health of a server, the usual suspects can be used: iostat, vmstat and netstat but when we need to dig down to memory-management like the above issues, we need a serious tool.

This is a situation where we need Valgrind. Valgrind works directly with the executables, with no need to recompile, relink or modify the program to be checked. Valgrind decides whether the program should be modified to avoid memory leak, and also points out the spots of “leak.”

Valgrind simulates every single instruction as the program executes. For this reason, Valgrind finds errors not only in the application but also in all supporting dynamically-linked (.so-format) libraries, including the GNU C library, the X client libraries, Qt if you work with KDE, and so on. That often includes libraries, for example the GNU C library, which may contain memory access violations.

With Valgrind’s tool suite you can automatically detect many memory management and threading bugs, avoiding hours of frustrating bug-hunting, making your programs more stable. You can also perform detailed profiling to help speed up your programs.

The Valgrind distribution Tool Suite includes four useful debugging and profiling tools:

Memcheck

Memcheck detects memory-management problems, and is aimed primarily at C and C++ programs. When a program is run under Memcheck’s supervision, all reads and writes of memory are checked, and calls to malloc/new/free/delete are intercepted. As a result, Memcheck can detect if the program:

  • Accesses memory it shouldn’t (areas not yet allocated, areas that have been freed, areas past the end of heap blocks, inaccessible areas of the stack).
  • Uses uninitialised values in dangerous ways.
  • Leaks memory.
  • Does bad frees of heap blocks (double frees, mismatched frees).
  • Passes overlapping source and destination memory blocks to memcpy() and related functions.

Memcheck reports these errors as soon as they occur, giving the source line number at which it occurred, and also a stack trace of the functions called to reach that line. Memcheck tracks addressability at the byte-level, and initialisation of values at the bit-level. As a result, it can detect the use of single uninitialised bits, and does not report spurious errors on bitfield operations. Memcheck runs programs about 10–30x slower than normal.

Cachegrind

Cachegrind is a cache profiler. It performs detailed simulation of the I1, D1 and L2 caches in the CPU and so can accurately pinpoint the sources of cache misses in your code. It identifies the number of cache misses, memory references and instructions executed for each line of source code, with per-function, per-module and whole-program summaries. It is useful with programs written in any language. Cachegrind runs programs about 20–100x slower than normal.

Callgrind

Callgrind, by Josef Weidendorfer, is an extension to Cachegrind. It provides all the information that Cachegrind does, plus extra information about callgraphs. It was folded into the main Valgrind distribution in version 3.2.0. Available separately is an amazing visualisation tool, KCachegrind, which gives a much better overview of the data that Callgrind collects; it can also be used to visualise Cachegrind’s output.

Massif

Massif is a heap profiler. It performs detailed heap profiling by taking regular snapshots of a program’s heap. It produces a graph showing heap usage over time, including information about which parts of the program are responsible for the most memory allocations. The graph is supplemented by a text or HTML file that includes more information for determining where the most memory is being allocated. Massif runs programs about 20x slower than normal.

Helgrind

Helgrind is a thread debugger which finds data races in multithreaded programs. It looks for memory locations which are accessed by more than one (POSIX p-)thread, but for which no consistently used (pthread_mutex_) lock can be found. Such locations are indicative of missing synchronisation between threads, and could cause hard-to-find timing-dependent problems. It is useful for any program that uses pthreads.

Some of the big name software projects using or have used Valgrind are Firefox, OpenOffice, StarOffice, AbiWord, Opera, KDE, GNOME, Qt, libstdc++, MySQL, PostgreSQL, Perl, Python, PHP, Samba, RenderMan, Nasa Mars Lander software, SAS, The GIMP, Ogg Vorbis, Unreal Tournament, Medal of Honour…..

If you’re on ubuntu, you can install it with the usual:
sudo apt-get install valgrind

or for Fedora/RedHat
sudo yum -y install valgrind

Playing around, the checking may be performed by simply placing the word valgrind just before the normal command used to invoke the program. For example:

mobile1:~$ valgrind ps -ef
==6752== Memcheck, a memory error detector.
==6752== Copyright (C) 2002-2007, and GNU GPL'd, by Julian Seward et al.
==6752== Using LibVEX rev 1804, a library for dynamic binary translation.
==6752== Copyright (C) 2004-2007, and GNU GPL'd, by OpenWorks LLP.
==6752== Using valgrind-3.3.0-Debian, a dynamic binary instrumentation framework.
==6752== Copyright (C) 2000-2007, and GNU GPL'd, by Julian Seward et al.
==6752== For more details, rerun with: -v
.............................
.............................
==6752==
==6752== ERROR SUMMARY: 4 errors from 4 contexts (suppressed: 21 from 1)
==6752== malloc/free: in use at exit: 1,102 bytes in 48 blocks.
==6752== malloc/free: 961 allocs, 913 frees, 42,627 bytes allocated.
==6752== For counts of detected errors, rerun with: -v
==6752== searching for pointers to 48 not-freed blocks.
==6752== checked 407,936 bytes.
==6752==
==6752== LEAK SUMMARY:
==6752==    definitely lost: 156 bytes in 11 blocks.
==6752==      possibly lost: 0 bytes in 0 blocks.
==6752==    still reachable: 946 bytes in 37 blocks.
==6752==         suppressed: 0 bytes in 0 blocks.
==6752== Rerun with --leak-check=full to see details of leaked memory.

Any error (memory related) is pointed out in the error report. With our example, there are 4 errors encountered and 156 bytes was lost due to memory leaked.

2,026 Comments

  1. Donnie

    I used to be able to find good info from your blog articles.

    Reply
  2. arsenal soccer jerseys

    I will right away take hold of your rss as I can’t in finding your e-mail subscription link or e-newsletter service.
    Do you have any? Kindly permit me recognise so that I may
    just subscribe. Thanks.

    Reply
  3. soccer sweat shirts

    Hello there! This post couldn’t be written much better!
    Looking through this post reminds me of my previous roommate!
    He continually kept preaching about this.
    I am going to forward this article to him. Fairly certain he will have a good read.
    Thanks for sharing!

    Reply
  4. holiday travel around

    I’ve read a few just right stuff here. Certainly value bookmarking
    for revisiting. I wonder how a lot attempt you set to create such a wonderful informative web site.

    Reply
  5. Lelio Vieira Carneiro Junior

    Hi there, i read your blog occasionally and i own a similar one and i was just curious if
    you get a lot of spam remarks? If so how do you
    reduce it, any plugin or anything you can suggest?

    I get so much lately it’s driving me insane so any support is
    very much appreciated.

    Reply
  6. Silvia Odete Morani Massad

    Hi there are using WordPress for your site platform?
    I’m new to the blog world but I’m trying to get started and create my own. Do you require any html coding expertise to make your own blog?
    Any help would be really appreciated!

    Reply
  7. Neva

    I am curious to find out what blog system you have been using?
    I’m having some small security issues with my
    latest website and I’d like to find something more
    safeguarded. Do you have any suggestions?

    Reply
  8. travel tips

    I’m really loving the theme/design of your website. Do you ever run into any internet browser compatibility problems?
    A handful of my blog visitors have complained
    about my blog not working correctly in Explorer but looks great
    in Chrome. Do you have any solutions to help fix this issue?

    Reply
  9. compatible code

    Ridiculous story there. What happened after? Good luck!

    Reply
  10. podcast listeners

    Hello, I enjoy reading all of your article. I like to write a little comment to support you.

    Reply
  11. James Frazer Mann

    Today, I went to the beach with my kids. I found a sea shell and gave it to
    my 4 year old daughter and said “You can hear the ocean if you put this to your ear.”
    She put the shell to her ear and screamed. There was a hermit crab inside and it pinched her ear.
    She never wants to go back! LoL I know this is completely off topic but I had to tell someone!

    Reply
  12. car fuel saving products

    Hey there would you mind stating which blog platform
    you’re using? I’m looking to start my own blog in the near
    future but I’m having a tough time choosing between BlogEngine/Wordpress/B2evolution and Drupal.

    The reason I ask is because your layout seems different
    then most blogs and I’m looking for something completely unique.
    P.S Sorry for being off-topic but I had to ask!

    Reply
  13. car gas saving system

    When someone writes an post he/she keeps the plan of a
    user in his/her brain that how a user can understand it.
    Therefore that’s why this article is perfect.
    Thanks!

    Reply
  14. two exhaust systems

    Write more, thats all I have to say. Literally, it seems as though you relied on the video to make your point.
    You obviously know what youre talking about, why throw away your intelligence on just
    posting videos to your weblog when you could be
    giving us something enlightening to read?

    Reply
  15. computer protection

    I think the admin of this web page is really working hard for his web site, since here
    every information is quality based data.

    Reply
  16. security products

    I read this piece of writing completely concerning the resemblance of latest
    and earlier technologies, it’s amazing article.

    Reply
  17. good business

    Thanks a lot for sharing this with all of us you actually recognize what you’re talking about!
    Bookmarked. Kindly also seek advice from my website =).
    We will have a hyperlink alternate agreement between us

    Reply
  18. cars value

    Hello! Do you use Twitter? I’d like to follow you if that would be okay.
    I’m absolutely enjoying your blog and look forward to new posts.

    Reply
  19. car hydrogen generator

    You are so cool! I do not think I have read through a single thing like this before.
    So wonderful to find someone with some unique thoughts on this issue.
    Seriously.. thank you for starting this up. This website is something that is needed on the web, someone with some
    originality!

    Reply
  20. car remotes

    Hi there to all, how is the whole thing, I think every one is getting more from this web
    site, and your views are good for new users.

    Reply
  21. run car water

    This post will assist the internet visitors for setting up new blog or even a blog from start to end.

    Reply
  22. podcast listeners

    Nice post. I learn something new and challenging on websites
    I stumbleupon every day. It’s always interesting to read content from other writers and use a little something from other web sites.

    Reply
  23. podcast uses

    Oh my goodness! Impressive article dude! Many thanks, However I am going through difficulties with your RSS.
    I don’t know why I cannot join it. Is there anybody else having identical RSS
    problems? Anybody who knows the answer will you kindly respond?

    Thanks!!

    Reply
  24. Great blog you have got here.. It’s difficult
    to find excellent writing like yours these days.
    I seriously appreciate individuals like you!
    Take care!!

    Reply
  25. Dalene

    hello there and thank you for your information – I have definitely picked up something new from right here.
    I did however expertise a few technical points using this website, since I experienced to reload the website a lot of times previous to I could get it to
    load properly. I had been wondering if your hosting is OK?
    Not that I’m complaining, but sluggish loading instances times will sometimes affect your placement in google and could damage your quality score if ads and
    marketing with Adwords. Well I’m adding this RSS to my e-mail and could look out for much more of your respective intriguing content.
    Ensure that you update this again soon.

    Reply
  26. range cars

    Very soon this website will be famous among all
    blog people, due to it’s nice content

    Reply

Leave a Comment

Your email address will not be published. Required fields are marked *