Monitoring UNIX Application Performance

Valgrind

Application Monitoring is a very important aspect of a project. Unfortunately, not much attention is being is paid to develop an effective monitoring while the project is not live. Once the project is live, the lack of proper monitoring factors in terms of downtime when the support person is not aware if application is having some problems or application not working at all.

In my line of work, we are being constantly bug by the application team why the system is slow or what is the reason of the system crash. Most of the time the cause is related to memory-management and performance problems, leaks/corruption in the program being run.

Memory management is prone to errors that are too hard to detect. Common errors may be listed as:

  1. Use of uninitialized memory
  2. Reading/writing memory after it has been freed
  3. Reading/writing off the end of malloc’d blocks
  4. Reading/writing inappropriate areas on the stack
  5. Memory leaks — where pointers to malloc’d blocks are lost forever
  6. Mismatched use of malloc/new/new[] vs free/delete/delete[]
  7. Some misuses of the POSIX pthreads API

In trying to monitor the health of a server, the usual suspects can be used: iostat, vmstat and netstat but when we need to dig down to memory-management like the above issues, we need a serious tool.

This is a situation where we need Valgrind. Valgrind works directly with the executables, with no need to recompile, relink or modify the program to be checked. Valgrind decides whether the program should be modified to avoid memory leak, and also points out the spots of “leak.”

Valgrind simulates every single instruction as the program executes. For this reason, Valgrind finds errors not only in the application but also in all supporting dynamically-linked (.so-format) libraries, including the GNU C library, the X client libraries, Qt if you work with KDE, and so on. That often includes libraries, for example the GNU C library, which may contain memory access violations.

With Valgrind’s tool suite you can automatically detect many memory management and threading bugs, avoiding hours of frustrating bug-hunting, making your programs more stable. You can also perform detailed profiling to help speed up your programs.

The Valgrind distribution Tool Suite includes four useful debugging and profiling tools:

Memcheck

Memcheck detects memory-management problems, and is aimed primarily at C and C++ programs. When a program is run under Memcheck’s supervision, all reads and writes of memory are checked, and calls to malloc/new/free/delete are intercepted. As a result, Memcheck can detect if the program:

  • Accesses memory it shouldn’t (areas not yet allocated, areas that have been freed, areas past the end of heap blocks, inaccessible areas of the stack).
  • Uses uninitialised values in dangerous ways.
  • Leaks memory.
  • Does bad frees of heap blocks (double frees, mismatched frees).
  • Passes overlapping source and destination memory blocks to memcpy() and related functions.

Memcheck reports these errors as soon as they occur, giving the source line number at which it occurred, and also a stack trace of the functions called to reach that line. Memcheck tracks addressability at the byte-level, and initialisation of values at the bit-level. As a result, it can detect the use of single uninitialised bits, and does not report spurious errors on bitfield operations. Memcheck runs programs about 10–30x slower than normal.

Cachegrind

Cachegrind is a cache profiler. It performs detailed simulation of the I1, D1 and L2 caches in the CPU and so can accurately pinpoint the sources of cache misses in your code. It identifies the number of cache misses, memory references and instructions executed for each line of source code, with per-function, per-module and whole-program summaries. It is useful with programs written in any language. Cachegrind runs programs about 20–100x slower than normal.

Callgrind

Callgrind, by Josef Weidendorfer, is an extension to Cachegrind. It provides all the information that Cachegrind does, plus extra information about callgraphs. It was folded into the main Valgrind distribution in version 3.2.0. Available separately is an amazing visualisation tool, KCachegrind, which gives a much better overview of the data that Callgrind collects; it can also be used to visualise Cachegrind’s output.

Massif

Massif is a heap profiler. It performs detailed heap profiling by taking regular snapshots of a program’s heap. It produces a graph showing heap usage over time, including information about which parts of the program are responsible for the most memory allocations. The graph is supplemented by a text or HTML file that includes more information for determining where the most memory is being allocated. Massif runs programs about 20x slower than normal.

Helgrind

Helgrind is a thread debugger which finds data races in multithreaded programs. It looks for memory locations which are accessed by more than one (POSIX p-)thread, but for which no consistently used (pthread_mutex_) lock can be found. Such locations are indicative of missing synchronisation between threads, and could cause hard-to-find timing-dependent problems. It is useful for any program that uses pthreads.

Some of the big name software projects using or have used Valgrind are Firefox, OpenOffice, StarOffice, AbiWord, Opera, KDE, GNOME, Qt, libstdc++, MySQL, PostgreSQL, Perl, Python, PHP, Samba, RenderMan, Nasa Mars Lander software, SAS, The GIMP, Ogg Vorbis, Unreal Tournament, Medal of Honour…..

If you’re on ubuntu, you can install it with the usual:
sudo apt-get install valgrind

or for Fedora/RedHat
sudo yum -y install valgrind

Playing around, the checking may be performed by simply placing the word valgrind just before the normal command used to invoke the program. For example:

mobile1:~$ valgrind ps -ef
==6752== Memcheck, a memory error detector.
==6752== Copyright (C) 2002-2007, and GNU GPL'd, by Julian Seward et al.
==6752== Using LibVEX rev 1804, a library for dynamic binary translation.
==6752== Copyright (C) 2004-2007, and GNU GPL'd, by OpenWorks LLP.
==6752== Using valgrind-3.3.0-Debian, a dynamic binary instrumentation framework.
==6752== Copyright (C) 2000-2007, and GNU GPL'd, by Julian Seward et al.
==6752== For more details, rerun with: -v
.............................
.............................
==6752==
==6752== ERROR SUMMARY: 4 errors from 4 contexts (suppressed: 21 from 1)
==6752== malloc/free: in use at exit: 1,102 bytes in 48 blocks.
==6752== malloc/free: 961 allocs, 913 frees, 42,627 bytes allocated.
==6752== For counts of detected errors, rerun with: -v
==6752== searching for pointers to 48 not-freed blocks.
==6752== checked 407,936 bytes.
==6752==
==6752== LEAK SUMMARY:
==6752==    definitely lost: 156 bytes in 11 blocks.
==6752==      possibly lost: 0 bytes in 0 blocks.
==6752==    still reachable: 946 bytes in 37 blocks.
==6752==         suppressed: 0 bytes in 0 blocks.
==6752== Rerun with --leak-check=full to see details of leaked memory.

Any error (memory related) is pointed out in the error report. With our example, there are 4 errors encountered and 156 bytes was lost due to memory leaked.

2,121 Comments

  1. Donnie

    I used to be able to find good info from your blog articles.

    Reply
  2. arsenal soccer jerseys

    I will right away take hold of your rss as I can’t in finding your e-mail subscription link or e-newsletter service.
    Do you have any? Kindly permit me recognise so that I may
    just subscribe. Thanks.

    Reply
  3. soccer sweat shirts

    Hello there! This post couldn’t be written much better!
    Looking through this post reminds me of my previous roommate!
    He continually kept preaching about this.
    I am going to forward this article to him. Fairly certain he will have a good read.
    Thanks for sharing!

    Reply
  4. holiday travel around

    I’ve read a few just right stuff here. Certainly value bookmarking
    for revisiting. I wonder how a lot attempt you set to create such a wonderful informative web site.

    Reply
  5. Lelio Vieira Carneiro Junior

    Hi there, i read your blog occasionally and i own a similar one and i was just curious if
    you get a lot of spam remarks? If so how do you
    reduce it, any plugin or anything you can suggest?

    I get so much lately it’s driving me insane so any support is
    very much appreciated.

    Reply
  6. Silvia Odete Morani Massad

    Hi there are using WordPress for your site platform?
    I’m new to the blog world but I’m trying to get started and create my own. Do you require any html coding expertise to make your own blog?
    Any help would be really appreciated!

    Reply
  7. Neva

    I am curious to find out what blog system you have been using?
    I’m having some small security issues with my
    latest website and I’d like to find something more
    safeguarded. Do you have any suggestions?

    Reply
  8. travel tips

    I’m really loving the theme/design of your website. Do you ever run into any internet browser compatibility problems?
    A handful of my blog visitors have complained
    about my blog not working correctly in Explorer but looks great
    in Chrome. Do you have any solutions to help fix this issue?

    Reply
  9. compatible code

    Ridiculous story there. What happened after? Good luck!

    Reply
  10. podcast listeners

    Hello, I enjoy reading all of your article. I like to write a little comment to support you.

    Reply
  11. James Frazer Mann

    Today, I went to the beach with my kids. I found a sea shell and gave it to
    my 4 year old daughter and said “You can hear the ocean if you put this to your ear.”
    She put the shell to her ear and screamed. There was a hermit crab inside and it pinched her ear.
    She never wants to go back! LoL I know this is completely off topic but I had to tell someone!

    Reply
  12. car fuel saving products

    Hey there would you mind stating which blog platform
    you’re using? I’m looking to start my own blog in the near
    future but I’m having a tough time choosing between BlogEngine/Wordpress/B2evolution and Drupal.

    The reason I ask is because your layout seems different
    then most blogs and I’m looking for something completely unique.
    P.S Sorry for being off-topic but I had to ask!

    Reply
  13. car gas saving system

    When someone writes an post he/she keeps the plan of a
    user in his/her brain that how a user can understand it.
    Therefore that’s why this article is perfect.
    Thanks!

    Reply
  14. two exhaust systems

    Write more, thats all I have to say. Literally, it seems as though you relied on the video to make your point.
    You obviously know what youre talking about, why throw away your intelligence on just
    posting videos to your weblog when you could be
    giving us something enlightening to read?

    Reply
  15. computer protection

    I think the admin of this web page is really working hard for his web site, since here
    every information is quality based data.

    Reply
  16. security products

    I read this piece of writing completely concerning the resemblance of latest
    and earlier technologies, it’s amazing article.

    Reply
  17. good business

    Thanks a lot for sharing this with all of us you actually recognize what you’re talking about!
    Bookmarked. Kindly also seek advice from my website =).
    We will have a hyperlink alternate agreement between us

    Reply
  18. cars value

    Hello! Do you use Twitter? I’d like to follow you if that would be okay.
    I’m absolutely enjoying your blog and look forward to new posts.

    Reply
  19. car hydrogen generator

    You are so cool! I do not think I have read through a single thing like this before.
    So wonderful to find someone with some unique thoughts on this issue.
    Seriously.. thank you for starting this up. This website is something that is needed on the web, someone with some
    originality!

    Reply
  20. car remotes

    Hi there to all, how is the whole thing, I think every one is getting more from this web
    site, and your views are good for new users.

    Reply
  21. run car water

    This post will assist the internet visitors for setting up new blog or even a blog from start to end.

    Reply
  22. podcast listeners

    Nice post. I learn something new and challenging on websites
    I stumbleupon every day. It’s always interesting to read content from other writers and use a little something from other web sites.

    Reply
  23. podcast uses

    Oh my goodness! Impressive article dude! Many thanks, However I am going through difficulties with your RSS.
    I don’t know why I cannot join it. Is there anybody else having identical RSS
    problems? Anybody who knows the answer will you kindly respond?

    Thanks!!

    Reply
  24. Great blog you have got here.. It’s difficult
    to find excellent writing like yours these days.
    I seriously appreciate individuals like you!
    Take care!!

    Reply
  25. Dalene

    hello there and thank you for your information – I have definitely picked up something new from right here.
    I did however expertise a few technical points using this website, since I experienced to reload the website a lot of times previous to I could get it to
    load properly. I had been wondering if your hosting is OK?
    Not that I’m complaining, but sluggish loading instances times will sometimes affect your placement in google and could damage your quality score if ads and
    marketing with Adwords. Well I’m adding this RSS to my e-mail and could look out for much more of your respective intriguing content.
    Ensure that you update this again soon.

    Reply
  26. range cars

    Very soon this website will be famous among all
    blog people, due to it’s nice content

    Reply
  27. http://www.globo.com/

    Do you mind if I quote a couple of your posts as long as I provide credit
    and sources back to your weblog? My website is in the very same area of interest as yours and my
    users would certainly benefit from some of the information you provide here.

    Please let me know if this okay with you. Many thanks!

    Reply
  28. Tilly

    Unquestionably believe that which you said. Your favorite justification appeared to
    be on the internet the easiest thing to be aware of.
    I say to you, I certainly get irked while people consider
    worries that they plainly don’t know about. You managed to hit the
    nail upon the top as well as defined out the whole thing without having side effect ,
    people could take a signal. Will probably be back to get more.
    Thanks

    Reply
  29. Jonathan

    I like the valuable info you provide in your articles. I will bookmark
    your weblog and check again here regularly. I am quite sure I will learn lots of new stuff right here!

    Good luck for the next!

    Reply
  30. makanan bayi usia 5 bulan

    Hi there, I found your web site by way of Google whilst searching for a comparable
    topic, your web site got here up, it appears great.
    I have bookmarked it in my google bookmarks.
    Hi there, simply was alert to your blog via Google, and found that it
    is really informative. I’m gonna be careful for brussels.
    I will appreciate when you continue this in future.
    Numerous other folks shall be benefited out of your writing.

    Cheers!

    Reply
  31. menu makanan bayi 9 bulan

    I’m really enjoying the design and layout
    of your site. It’s a very easy on the eyes which makes it much more pleasant for me to come here and visit more often. Did you hire out a
    designer to create your theme? Fantastic work!

    Reply
  32. augusto de arruda botelho

    Whats up are using WordPress for your blog platform?
    I’m new to the blog world but I’m trying to get started and set up
    my own. Do you require any html coding expertise to make your
    own blog? Any help would be really appreciated!

    Reply
  33. augusto de arruda botelho

    Heya i am for the first time here. I came across this board
    and I find It really useful & it helped me out a lot.
    I hope to give something back and aid others like you helped me.

    Reply
  34. Silvia Odete Morani Massad

    Hi friends, fastidious post and nice urging commented at this place, I am truly enjoying by these.

    Reply
  35. Silvia Odete Morani Massad

    You could certainly see your skills in the work you write.
    The arena hopes for more passionate writers like
    you who are not afraid to say how they believe.
    Always follow your heart.

    Reply
  36. I am really thankful to the owner of this website who has shared this enormous article at at this place.

    Reply
  37. mengatasi bayi yang susah makan

    What a data of un-ambiguity and preserveness of precious know-how concerning
    unpredicted emotions.

    Reply
  38. Josh

    An impressive share! I’ve just forwarded this onto a co-worker
    who had been doing a little research on this. And he actually ordered me dinner due to
    the fact that I stumbled upon it for him… lol. So let me reword this….
    Thanks for the meal!! But yeah, thanx for spending some time
    to discuss this issue here on your web page.

    Reply
  39. augusto de arruda botelho

    Hey there! I know this is kinda off topic but I was wondering which
    blog platform are you using for this site? I’m getting tired
    of WordPress because I’ve had problems with hackers and I’m looking at
    options for another platform. I would be awesome if you could point me in the direction of
    a good platform.

    Reply
  40. resep makanan bayi 11 bulan susah makan

    Aw, this was a really nice post. Taking the time and actual effort to make a great
    article… but what can I say… I hesitate a whole lot and never seem to get nearly anything done.

    Reply
  41. see my site

    Your place is valueble for me. Thanks!?

    Reply
  42. Lelio Vieira Carneiro Junior

    I have to thank you for the efforts you’ve put in writing
    this blog. I’m hoping to see the same high-grade blog
    posts from you later on as well. In truth, your creative writing abilities has inspired me to get
    my very own blog now 😉

    Reply
  43. jerrystory.com

    This design is wicked! You certainly know how to keep a reader amused.
    Between your wit and your videos, I was almost moved to start my own blog (well, almost…HaHa!) Great job.

    I really enjoyed what you had to say, and more than that, how you presented it.

    Too cool!

    Reply
  44. ocbc sibu contact us

    you’re ?nn po?nt ?f fact a good webmaster.
    The web siute loading se?d is ama?ing. It seems that you’re doing aany distinctive trick.
    Moreover, The cont?nts are masterpi?ce. you’ve
    performed a magnificent job oon this matter!

    Reply
  45. see my site

    When I originally commented I clicked the -Notify me when new comments are added- checkbox and now each time a comment is added I get four emails with the same comment. Is there any way you can remove me from that service? Thanks!

    Reply
  46. click on my website

    It?s hard to find knowledgeable people on this topic, but you sound like you know what you?re talking about! Thanks

    Reply
  47. check this out

    I?m impressed, I must say. Really rarely do I encounter a blog that?s both educative and entertaining, and let me tell you, you have hit the nail on the head. Your idea is outstanding; the issue is something that not enough people are speaking intelligently about. I am very happy that I stumbled across this in my search for something relating to this.

    Reply
  48. anak susah makan usia 1 tahun

    Thank you, I’ve just been searching for info about this topic
    for a while and yours is the best I have found out so far.
    But, what about the conclusion? Are you positive concerning the supply?

    Reply
  49. resep camilan sehat untuk anak

    Hey, I think your blog might be having browser compatibility issues.
    When I look at your website in Chrome, it looks fine but when opening
    in Internet Explorer, it has some overlapping.
    I just wanted to give you a quick heads up! Other then that, very good blog!

    Reply
  50. Selene

    Hello! I could have sworn I’ve been to this site before but after checking
    through some of the post I realized it’s new to me. Nonetheless, I’m definitely delighted I found it and I’ll be
    bookmarking and checking back often!

    Reply

Leave a Comment

Your email address will not be published. Required fields are marked *