Monitoring UNIX Application Performance

Valgrind

Application Monitoring is a very important aspect of a project. Unfortunately, not much attention is being is paid to develop an effective monitoring while the project is not live. Once the project is live, the lack of proper monitoring factors in terms of downtime when the support person is not aware if application is having some problems or application not working at all.

In my line of work, we are being constantly bug by the application team why the system is slow or what is the reason of the system crash. Most of the time the cause is related to memory-management and performance problems, leaks/corruption in the program being run.

Memory management is prone to errors that are too hard to detect. Common errors may be listed as:

  1. Use of uninitialized memory
  2. Reading/writing memory after it has been freed
  3. Reading/writing off the end of malloc’d blocks
  4. Reading/writing inappropriate areas on the stack
  5. Memory leaks — where pointers to malloc’d blocks are lost forever
  6. Mismatched use of malloc/new/new[] vs free/delete/delete[]
  7. Some misuses of the POSIX pthreads API

In trying to monitor the health of a server, the usual suspects can be used: iostat, vmstat and netstat but when we need to dig down to memory-management like the above issues, we need a serious tool.

This is a situation where we need Valgrind. Valgrind works directly with the executables, with no need to recompile, relink or modify the program to be checked. Valgrind decides whether the program should be modified to avoid memory leak, and also points out the spots of “leak.”

Valgrind simulates every single instruction as the program executes. For this reason, Valgrind finds errors not only in the application but also in all supporting dynamically-linked (.so-format) libraries, including the GNU C library, the X client libraries, Qt if you work with KDE, and so on. That often includes libraries, for example the GNU C library, which may contain memory access violations.

With Valgrind’s tool suite you can automatically detect many memory management and threading bugs, avoiding hours of frustrating bug-hunting, making your programs more stable. You can also perform detailed profiling to help speed up your programs.

The Valgrind distribution Tool Suite includes four useful debugging and profiling tools:

Memcheck

Memcheck detects memory-management problems, and is aimed primarily at C and C++ programs. When a program is run under Memcheck’s supervision, all reads and writes of memory are checked, and calls to malloc/new/free/delete are intercepted. As a result, Memcheck can detect if the program:

  • Accesses memory it shouldn’t (areas not yet allocated, areas that have been freed, areas past the end of heap blocks, inaccessible areas of the stack).
  • Uses uninitialised values in dangerous ways.
  • Leaks memory.
  • Does bad frees of heap blocks (double frees, mismatched frees).
  • Passes overlapping source and destination memory blocks to memcpy() and related functions.

Memcheck reports these errors as soon as they occur, giving the source line number at which it occurred, and also a stack trace of the functions called to reach that line. Memcheck tracks addressability at the byte-level, and initialisation of values at the bit-level. As a result, it can detect the use of single uninitialised bits, and does not report spurious errors on bitfield operations. Memcheck runs programs about 10–30x slower than normal.

Cachegrind

Cachegrind is a cache profiler. It performs detailed simulation of the I1, D1 and L2 caches in the CPU and so can accurately pinpoint the sources of cache misses in your code. It identifies the number of cache misses, memory references and instructions executed for each line of source code, with per-function, per-module and whole-program summaries. It is useful with programs written in any language. Cachegrind runs programs about 20–100x slower than normal.

Callgrind

Callgrind, by Josef Weidendorfer, is an extension to Cachegrind. It provides all the information that Cachegrind does, plus extra information about callgraphs. It was folded into the main Valgrind distribution in version 3.2.0. Available separately is an amazing visualisation tool, KCachegrind, which gives a much better overview of the data that Callgrind collects; it can also be used to visualise Cachegrind’s output.

Massif

Massif is a heap profiler. It performs detailed heap profiling by taking regular snapshots of a program’s heap. It produces a graph showing heap usage over time, including information about which parts of the program are responsible for the most memory allocations. The graph is supplemented by a text or HTML file that includes more information for determining where the most memory is being allocated. Massif runs programs about 20x slower than normal.

Helgrind

Helgrind is a thread debugger which finds data races in multithreaded programs. It looks for memory locations which are accessed by more than one (POSIX p-)thread, but for which no consistently used (pthread_mutex_) lock can be found. Such locations are indicative of missing synchronisation between threads, and could cause hard-to-find timing-dependent problems. It is useful for any program that uses pthreads.

Some of the big name software projects using or have used Valgrind are Firefox, OpenOffice, StarOffice, AbiWord, Opera, KDE, GNOME, Qt, libstdc++, MySQL, PostgreSQL, Perl, Python, PHP, Samba, RenderMan, Nasa Mars Lander software, SAS, The GIMP, Ogg Vorbis, Unreal Tournament, Medal of Honour…..

If you’re on ubuntu, you can install it with the usual:
sudo apt-get install valgrind

or for Fedora/RedHat
sudo yum -y install valgrind

Playing around, the checking may be performed by simply placing the word valgrind just before the normal command used to invoke the program. For example:

mobile1:~$ valgrind ps -ef
==6752== Memcheck, a memory error detector.
==6752== Copyright (C) 2002-2007, and GNU GPL'd, by Julian Seward et al.
==6752== Using LibVEX rev 1804, a library for dynamic binary translation.
==6752== Copyright (C) 2004-2007, and GNU GPL'd, by OpenWorks LLP.
==6752== Using valgrind-3.3.0-Debian, a dynamic binary instrumentation framework.
==6752== Copyright (C) 2000-2007, and GNU GPL'd, by Julian Seward et al.
==6752== For more details, rerun with: -v
.............................
.............................
==6752==
==6752== ERROR SUMMARY: 4 errors from 4 contexts (suppressed: 21 from 1)
==6752== malloc/free: in use at exit: 1,102 bytes in 48 blocks.
==6752== malloc/free: 961 allocs, 913 frees, 42,627 bytes allocated.
==6752== For counts of detected errors, rerun with: -v
==6752== searching for pointers to 48 not-freed blocks.
==6752== checked 407,936 bytes.
==6752==
==6752== LEAK SUMMARY:
==6752==    definitely lost: 156 bytes in 11 blocks.
==6752==      possibly lost: 0 bytes in 0 blocks.
==6752==    still reachable: 946 bytes in 37 blocks.
==6752==         suppressed: 0 bytes in 0 blocks.
==6752== Rerun with --leak-check=full to see details of leaked memory.

Any error (memory related) is pointed out in the error report. With our example, there are 4 errors encountered and 156 bytes was lost due to memory leaked.

2,360 Comments

  1. besting timex

    Hello! Someone in my Myspace group shared this site with us so I came to look it over.
    I’m definitely enjoying the information. I’m bookmarking and
    will be tweeting this to my followers! Terrific blog
    and outstanding design.

    Reply
  2. Itamar Serp

    I’m not that much of a online reader to be honest but your blogs really nice, keep it up! I’ll go ahead and bookmark your website to come back later. Cheers|

    Reply
  3. Earle

    It is in point of fact a great and useful piece of information. I’m happy that
    you simply shared this helpful info with us. Please stay us informed like this.
    Thanks for sharing.

    Reply
  4. phones allow

    My partner and I absolutely love your blog and find a
    lot of your post’s to be just what I’m looking for.
    Do you offer guest writers to write content for yourself?
    I wouldn’t mind composing a post or elaborating on a lot of the subjects you write regarding here.
    Again, awesome blog!

    Reply
  5. Luiz Fernando Monteiro Bittencourt

    Thank you for every other informative website. The place else may just I am getting that type of info written in such an ideal method? I’ve a challenge that I’m simply now operating on, and I’ve been on the look out for such information.|

    Reply
  6. Umanizzare presídios

    These are genuinely great ideas in concerning blogging.
    You have touched some pleasant factors here. Any way keep up
    wrinting.

    Reply
  7. domain names

    This is my first time pay a visit at here and i am actually
    pleassant to read all at one place.

    Reply
  8. Ophelia

    I loved as much as you’ll receive carried out right here. The sketch is attractive,
    your authored subject matter stylish. nonetheless, you command
    get bought an impatience over that you wish be delivering the following.
    unwell unquestionably come more formerly again as exactly the same nearly
    a lot often inside case you shield this hike.

    Reply
  9. Jose Wilame Araujo Rodrigues

    In fact no matter if someone doesn’t understand afterward its up to other users that they will assist, so here it occurs.|

    Reply
  10. Umanizzare presidios

    Do you have a spam problem on this site; I also am a blogger, and I was wanting to know your situation; we have developed some nice methods and we are looking to swap solutions with other folks, why not shoot me an e-mail if interested.|

    Reply
  11. Fred

    What’s Taking place i’m new to this, I stumbled
    upon this I’ve found It positively helpful and it has helped me out loads.
    I hope to give a contribution & assist different users like its aided
    me. Great job.

    Reply
  12. Itamar Serpa Fernandes

    It is appropriate time to make some plans for the future and it is time to be happy. I have read this post and if I could I desire to suggest you few interesting things or suggestions. Perhaps you can write next articles referring to this article. I desire to read even more things about it!|

    Reply
  13. Luiz Gastao Bittencourt da Silva

    I don’t know if it’s just me or if perhaps everybody else experiencing issues with your blog. It appears like some of the text in your content are running off the screen. Can somebody else please comment and let me know if this is happening to them too? This could be a problem with my internet browser because I’ve had this happen previously. Appreciate it|

    Reply
  14. Itamar Serp

    This is the perfect website for everyone who really wants to find out about
    this topic. You understand so much its almost tough to
    argue with you (not that I personally will need to…HaHa).
    You definitely put a brand new spin on a topic
    that has been written about for many years.
    Wonderful stuff, just great!

    Reply
  15. Hello all, here every person is sharing such know-how, so it’s good to read this web site, and I used to
    visit this website everyday.

    Reply
  16. Luiz Gastão Bittencourt da Silva

    It is the best time to make some plans for the longer term and it is time to be happy. I have learn this post and if I may I wish to counsel you few interesting issues or advice. Maybe you could write subsequent articles relating to this article. I desire to read more issues approximately it!|

    Reply
  17. smart phone

    I think the admin of this web page is really
    working hard for his website, as here every data
    is quality based material.

    Reply
  18. uch.zncai.com

    You’re so cool! I do not think I have read anything like this before.
    So wonderful to discover another person with original thoughts on this issue.
    Really.. thanks for starting this up. This website is one
    thing that’s needed on the web, someone with a little originality!

    Reply
  19. Itamar Serpa Fernandes

    This is my first time pay a quick visit at here and i am actually impressed to read everthing at single place.|

    Reply
  20. Ardis

    What i don’t understood is in fact how you’re not actually
    a lot more neatly-appreciated than you might be right now.
    You are so intelligent. You realize thus considerably on the subject of this matter, made me for my part imagine it from numerous various angles.

    Its like men and women are not fascinated except it’s something
    to accomplish with Girl gaga! Your individual stuffs outstanding.
    Always handle it up!

    Reply
  21. Itamar Serp

    Hello would you mind stating which blog platform you’re working with?
    I’m planning to start my own blog in the near future but I’m having
    a difficult time selecting between BlogEngine/Wordpress/B2evolution and Drupal.
    The reason I ask is because your design and style seems different then most blogs and I’m looking for something unique.

    P.S Sorry for getting off-topic but I had to ask!

    Reply
  22. Lelio Junior

    I really love your site.. Very nice colors & theme. Did you create this web site yourself? Please reply back as I’m trying to create my very own website and would like to know where you got this from or just what the theme is named. Kudos!|

    Reply
  23. Grupo Serval presidios

    I was able to find good info from your blog articles.

    Reply
  24. Dadado Veig

    I believe that is one of the such a lot important information for me. And i am happy reading your article. However should remark on few common issues, The website taste is wonderful, the articles is truly nice : D. Good process, cheers|

    Reply
  25. Luiz Gastão Bittencourt

    Hurrah, that’s what I was exploring for, what a information! existing here at this website, thanks admin of this website.|

    Reply
  26. Luiz Gastão Bittencourt da Silva

    It’s the best time to make a few plans for the future and it is time to be happy. I’ve read this publish and if I could I wish to recommend you some interesting things or tips. Maybe you can write subsequent articles referring to this article. I wish to learn more issues about it!|

    Reply
  27. Itamar Serp

    Fantastic beat ! I wish to apprentice whilst you amend your web site, how can i subscribe for a weblog website? The account helped me a acceptable deal. I have been a little bit acquainted of this your broadcast offered shiny clear idea|

    Reply
  28. Carlos Eduardo Correa da Veig

    When I initially left a comment I seem to have clicked on the -Notify me when new comments are added- checkbox and now whenever a comment is added I receive four emails with the exact same comment. There has to be a means you are able to remove me from that service? Thanks a lot!|

    Reply
  29. Egli Diana Pinto

    This is a topic which is close to my heart… Many thanks! Exactly where are your contact details though?|

    Reply
  30. smart phone applications

    Hi! I’ve been following your blog for a while now and finally
    got the bravery to go ahead and give you a shout out from Dallas Tx!

    Just wanted to say keep up the good job!

    Reply
  31. Luiz Gastão Bittencourt da Silva

    Good answers in return of this question with real arguments and describing everything concerning that.|

    Reply
  32. Itamar Serp

    Hmm is anyone else experiencing problems with the images on this blog loading? I’m trying to find out if its a problem on my end or if it’s the blog. Any feed-back would be greatly appreciated.|

    Reply
  33. samsung home security cameras amazon

    Shield your home from deterioration?

    Reply
  34. Umanizzare Gestão Prisional Ltda

    Hi friends, how is everything, and what you wish for to say concerning this post, in my view its genuinely amazing for me.|

    Reply
  35. home security systems consumer reports 2015

    Do you experience feeling they can are often more find in your house?

    Reply
  36. Grupo Serval presídios

    }

    Reply
  37. Lelio Junior

    I simply couldn’t leave your website before suggesting that I extremely enjoyed the usual information a person supply to your visitors? Is gonna be again often to investigate cross-check new posts|

    Reply
  38. domain name registrar

    I’m gone to inform my little brother, that he should also visit this weblog on regular basis to obtain updated
    from newest gossip.

    Reply
  39. Umanizzare

    Hi there, just became alert to your blog through Google, and found that it’s truly informative. I’m gonna watch out for brussels. I will be grateful if you continue this in future. A lot of people will be benefited from your writing. Cheers!|

    Reply
  40. reverse phone directory

    This is my first time pay a quick visit at here and i am truly happy to read everthing at one place.

    Reply
  41. Lelio Vieira Carneiro Junior

    }

    Reply
  42. Luiz Fernando Monteiro Bittencourt

    It’s the best time to make a few plans for the long run and it is time to be happy. I have read this submit and if I may I wish to recommend you some attention-grabbing issues or tips. Maybe you could write subsequent articles referring to this article. I desire to learn more things approximately it!|

    Reply
  43. Jose Wilame Araujo Rodrigues

    I do not even know how I stopped up right here, but I thought this submit was great. I don’t know who you’re however definitely you’re going to a well-known blogger for those who aren’t already. Cheers!|

    Reply
  44. Umanizzare Gestão Prisional Ltda

    Great information. Lucky me I came across your site by accident (stumbleupon). I’ve book marked it for later!|

    Reply
  45. Luiz Gastão Bittencourt

    Good post. I learn something new and challenging on websites I stumbleupon everyday. It will always be exciting to read content from other writers and use something from their web sites. |

    Reply
  46. Itamar Serpa Fernandes

    Hi there colleagues, nice article and fastidious urging commented at this place, I am truly enjoying by these.|

    Reply
  47. tercio borlenghi jr

    You should be a part of a contest for one of the best sites on the net. I most certainly will recommend this web site!|

    Reply
  48. Luiz Fernando Monteiro Bittencourt

    This paragraph offers clear idea for the new users of blogging, that really how to do blogging.|

    Reply
  49. Grupo Serval presidios

    Nice post. I learn something totally new and challenging on websites I stumbleupon everyday. It will always be helpful to read articles from other writers and practice something from their websites. |

    Reply
  50. Carlos Eduardo Veiga

    I do not even understand how I ended up here, but I assumed this post was once great. I do not recognise who you might be but certainly you’re going to a well-known blogger when you are not already. Cheers!|

    Reply

Leave a Comment

Your email address will not be published. Required fields are marked *