Monitoring UNIX Application Performance

Valgrind

Application Monitoring is a very important aspect of a project. Unfortunately, not much attention is being is paid to develop an effective monitoring while the project is not live. Once the project is live, the lack of proper monitoring factors in terms of downtime when the support person is not aware if application is having some problems or application not working at all.

In my line of work, we are being constantly bug by the application team why the system is slow or what is the reason of the system crash. Most of the time the cause is related to memory-management and performance problems, leaks/corruption in the program being run.

Memory management is prone to errors that are too hard to detect. Common errors may be listed as:

  1. Use of uninitialized memory
  2. Reading/writing memory after it has been freed
  3. Reading/writing off the end of malloc’d blocks
  4. Reading/writing inappropriate areas on the stack
  5. Memory leaks — where pointers to malloc’d blocks are lost forever
  6. Mismatched use of malloc/new/new[] vs free/delete/delete[]
  7. Some misuses of the POSIX pthreads API

In trying to monitor the health of a server, the usual suspects can be used: iostat, vmstat and netstat but when we need to dig down to memory-management like the above issues, we need a serious tool.

This is a situation where we need Valgrind. Valgrind works directly with the executables, with no need to recompile, relink or modify the program to be checked. Valgrind decides whether the program should be modified to avoid memory leak, and also points out the spots of “leak.”

Valgrind simulates every single instruction as the program executes. For this reason, Valgrind finds errors not only in the application but also in all supporting dynamically-linked (.so-format) libraries, including the GNU C library, the X client libraries, Qt if you work with KDE, and so on. That often includes libraries, for example the GNU C library, which may contain memory access violations.

With Valgrind’s tool suite you can automatically detect many memory management and threading bugs, avoiding hours of frustrating bug-hunting, making your programs more stable. You can also perform detailed profiling to help speed up your programs.

The Valgrind distribution Tool Suite includes four useful debugging and profiling tools:

Memcheck

Memcheck detects memory-management problems, and is aimed primarily at C and C++ programs. When a program is run under Memcheck’s supervision, all reads and writes of memory are checked, and calls to malloc/new/free/delete are intercepted. As a result, Memcheck can detect if the program:

  • Accesses memory it shouldn’t (areas not yet allocated, areas that have been freed, areas past the end of heap blocks, inaccessible areas of the stack).
  • Uses uninitialised values in dangerous ways.
  • Leaks memory.
  • Does bad frees of heap blocks (double frees, mismatched frees).
  • Passes overlapping source and destination memory blocks to memcpy() and related functions.

Memcheck reports these errors as soon as they occur, giving the source line number at which it occurred, and also a stack trace of the functions called to reach that line. Memcheck tracks addressability at the byte-level, and initialisation of values at the bit-level. As a result, it can detect the use of single uninitialised bits, and does not report spurious errors on bitfield operations. Memcheck runs programs about 10–30x slower than normal.

Cachegrind

Cachegrind is a cache profiler. It performs detailed simulation of the I1, D1 and L2 caches in the CPU and so can accurately pinpoint the sources of cache misses in your code. It identifies the number of cache misses, memory references and instructions executed for each line of source code, with per-function, per-module and whole-program summaries. It is useful with programs written in any language. Cachegrind runs programs about 20–100x slower than normal.

Callgrind

Callgrind, by Josef Weidendorfer, is an extension to Cachegrind. It provides all the information that Cachegrind does, plus extra information about callgraphs. It was folded into the main Valgrind distribution in version 3.2.0. Available separately is an amazing visualisation tool, KCachegrind, which gives a much better overview of the data that Callgrind collects; it can also be used to visualise Cachegrind’s output.

Massif

Massif is a heap profiler. It performs detailed heap profiling by taking regular snapshots of a program’s heap. It produces a graph showing heap usage over time, including information about which parts of the program are responsible for the most memory allocations. The graph is supplemented by a text or HTML file that includes more information for determining where the most memory is being allocated. Massif runs programs about 20x slower than normal.

Helgrind

Helgrind is a thread debugger which finds data races in multithreaded programs. It looks for memory locations which are accessed by more than one (POSIX p-)thread, but for which no consistently used (pthread_mutex_) lock can be found. Such locations are indicative of missing synchronisation between threads, and could cause hard-to-find timing-dependent problems. It is useful for any program that uses pthreads.

Some of the big name software projects using or have used Valgrind are Firefox, OpenOffice, StarOffice, AbiWord, Opera, KDE, GNOME, Qt, libstdc++, MySQL, PostgreSQL, Perl, Python, PHP, Samba, RenderMan, Nasa Mars Lander software, SAS, The GIMP, Ogg Vorbis, Unreal Tournament, Medal of Honour…..

If you’re on ubuntu, you can install it with the usual:
sudo apt-get install valgrind

or for Fedora/RedHat
sudo yum -y install valgrind

Playing around, the checking may be performed by simply placing the word valgrind just before the normal command used to invoke the program. For example:

mobile1:~$ valgrind ps -ef
==6752== Memcheck, a memory error detector.
==6752== Copyright (C) 2002-2007, and GNU GPL'd, by Julian Seward et al.
==6752== Using LibVEX rev 1804, a library for dynamic binary translation.
==6752== Copyright (C) 2004-2007, and GNU GPL'd, by OpenWorks LLP.
==6752== Using valgrind-3.3.0-Debian, a dynamic binary instrumentation framework.
==6752== Copyright (C) 2000-2007, and GNU GPL'd, by Julian Seward et al.
==6752== For more details, rerun with: -v
.............................
.............................
==6752==
==6752== ERROR SUMMARY: 4 errors from 4 contexts (suppressed: 21 from 1)
==6752== malloc/free: in use at exit: 1,102 bytes in 48 blocks.
==6752== malloc/free: 961 allocs, 913 frees, 42,627 bytes allocated.
==6752== For counts of detected errors, rerun with: -v
==6752== searching for pointers to 48 not-freed blocks.
==6752== checked 407,936 bytes.
==6752==
==6752== LEAK SUMMARY:
==6752==    definitely lost: 156 bytes in 11 blocks.
==6752==      possibly lost: 0 bytes in 0 blocks.
==6752==    still reachable: 946 bytes in 37 blocks.
==6752==         suppressed: 0 bytes in 0 blocks.
==6752== Rerun with --leak-check=full to see details of leaked memory.

Any error (memory related) is pointed out in the error report. With our example, there are 4 errors encountered and 156 bytes was lost due to memory leaked.

2,081 Comments

  1. iplayhavocph.com

    Thanks for some other wonderful post. The place else may anyone get
    that type of info in such an ideal means of writing?

    I have a presentation next week, and I am on the search for
    such info.

    Reply
  2. decisions regarding real

    I used to be recommended this blog by my cousin. I’m not sure whether or
    not this submit is written through him as no one else recognize
    such detailed about my difficulty. You’re incredible! Thank you!

    Reply
  3. door pops back

    Excellent, what a webpage it is! This blog gives
    helpful information to us, keep it up.

    Reply
  4. www.german-soldiers.de

    We’re a group of volunteers and starting a new scheme in our community.
    Your web site provided us with valuable info to work on. You’ve done a formidable
    job and our entire community will be thankful to you.

    Reply
  5. James Frazer-Mann

    If some one needs expert view on the topic of blogging afterward i
    recommend him/her to pay a visit this weblog, Keep up
    the fastidious job.

    Reply
  6. superlative real estate

    Fine way of explaining, and fastidious article to take facts about
    my presentation subject, which i am going to present
    in academy.

    Reply
  7. 360 frontal

    360 frontal https://www.youtube.com/watch?v=ny8rUpI_98I Hyper Agile et conforme, à recommander très vivement

    Reply
  8. car loan

    Ahaa, its good discussion regarding this piece of writing here at this blog, I have
    read all that, so at this time me also commenting here.

    Reply
  9. compound interest problems

    Hi there i am kavin, its my first time to commenting anyplace, when i read this paragraph i thought i could also make comment due to this brilliant paragraph.

    Reply
  10. www.fibbeadvocaten.nl

    Generally I do not read article on blogs, however I would like to
    say that this write-up very pressured me to take
    a look at and do it! Your writing taste has been amazed me.
    Thank you, quite nice article.

    Reply
  11. garage doors

    Hi there to every body, it’s my first pay a visit of
    this weblog; this website consists of amazing and truly excellent information for visitors.

    Reply
  12. http://web.enerjiuzmanlari.org.tr

    Magnificent beat ! I would like to apprentice whilst you amend your web site, how can i subscribe for a blog web site?
    The account aided me a applicable deal. I have been tiny bit familiar of this your broadcast
    provided vivid clear concept

    Reply
  13. Melvina

    I do not even know how I ended up here, but I thought this post was good.
    I don’t know who you are but certainly you are going
    to a famous blogger if you are not already 😉 Cheers!

    Reply
  14. garage door installers

    hello there and thank you for your information – I have definitely picked up
    something new from right here. I did however expertise a
    few technical issues using this web site, since I experienced to reload the site
    many times previous to I could get it to load correctly. I had
    been wondering if your hosting is OK? Not that I’m complaining, but sluggish
    loading instances times will often affect your placement in google and
    could damage your quality score if ads and
    marketing with Adwords. Well I am adding this RSS to
    my email and can look out for a lot more of your respective exciting content.
    Make sure you update this again soon.

    Reply
  15. Ollie

    I used to be recommended this blog by means of my cousin. I’m now not positive whether or not this
    submit is written by means of him as no one else recognize
    such specific about my problem. You are amazing! Thank you!

    Reply
  16. tourism presents

    Thanks for sharing your thoughts about air travel.
    Regards

    Reply
  17. Judson

    I’m really impressed along with your writing abilities
    as smartly as with the structure for your blog. Is that
    this a paid topic or did you customize it yourself?

    Anyway keep up the excellent quality writing, it’s rare to
    see a great blog like this one today..

    Reply
  18. travel companions

    Pretty! This has been a really wonderful post.

    Many thanks for providing these details.

    Reply
  19. good golf course

    Incredible quest there. What happened after? Thanks!

    Reply
  20. accounting jobs klang

    Thanks ffor your marvel?us posting! I definitely enjoyed reading
    ?t, you are ? ??eat a?thor. I will be sure to bookmark
    your blog and will oft?n come back from now on. I want to
    encourage you to def?itely cont?nue your g?eat writing, have a
    nixe morning!

    Reply
  21. cause fatal heart

    Simply desire to say your article is as astounding.
    The clarity in your post is just spectacular and i could assume you’re an expert
    on this subject. Well with your permission allow me to grab your feed to keep updated with forthcoming post.
    Thanks a million and please carry on the gratifying work.

    Reply
  22. Jose

    Pretty! This was an incredibly wonderful post. Thanks for supplying
    these details.

    Reply
  23. eyesin-tours-travels.com

    I got this web site from my buddy who shared with
    me regarding this website and at the moment this time I am browsing this website and reading very informative articles or reviews at this place.

    Reply
  24. hnbbs.mh.gy

    Thanks for finally talking about >Monitoring UNIX Application Performance
    – Kramfs Tech Chronicles <Loved it!

    Reply
  25. irregular heart rhythms

    Wow that was unusual. I just wrote an very long comment but after I clicked submit my comment didn’t appear.
    Grrrr… well I’m not writing all that over again. Anyhow, just wanted to say excellent
    blog!

    Reply
  26. Its such as you read my mind! You seem to know so much about this, such as you wrote the
    book in it or something. I feel that you can do with some %
    to power the message house a little bit, but instead of that,
    this is wonderful blog. An excellent read. I
    will certainly be back.

    Reply
  27. drjuandiegovalverde.com

    Have you ever thought about including a little bit more than just your articles?
    I mean, what you say is valuable and everything.

    But think of if you added some great visuals or videos to give your posts more, “pop”!

    Your content is excellent but with images and video clips, this blog could certainly be one
    of the most beneficial in its niche. Great blog!

    Reply
  28. Wilbur

    I absolutely love your blog and find a lot of your post’s to be just what
    I’m looking for. Do you offer guest writers to write content for you?
    I wouldn’t mind writing a post or elaborating on many of the subjects you write related to here.
    Again, awesome weblog!

    Reply
  29. arsenal soccer

    Wonderful goods from you, man. I have understand your stuff previous to and
    you’re just extremely great. I actually like
    what you have acquired here, really like what you are saying and the way in which you
    say it. You make it enjoyable and you still take care of to keep it sensible.
    I can’t wait to read much more from you. This is really a
    tremendous site.

    Reply
  30. fatal heart

    I was wondering if you ever considered changing the page layout of your website?

    Its very well written; I love what youve got to say. But maybe you could a little more
    in the way of content so people could connect
    with it better. Youve got an awful lot of text
    for only having 1 or two pictures. Maybe you could space it out better?

    Reply
  31. www.kaixinmaogou.com

    I believe this is among the such a lot important information for
    me. And i’m satisfied studying your article.
    However want to remark on some normal things, The site style is
    perfect, the articles is actually great : D. Just right task, cheers

    Reply

Leave a Comment

Your email address will not be published. Required fields are marked *