Apache hangs on Digest Secret generation

I have a machine on my network that is very special. It’s a rather old Quad Xeon, an HP LH4 that I scavenged out of the proverbial dumpster of a buisness that didn’t want it anymore. In fact, they were about to trash six of them.

I decided it was a crime to shitcan such beautiful machines, so with the help of my friend Mike “I mangle french words” Le Blanc, we drove there, armed with a large truck and patience. Carrying them down the three stories with no elevator was an interesting experience. I scavenged lots and lots of interesting hardware there.

But i’m getting ahead of myself here. The point is, lots of screwy things start occuring when you have a Quad SMP machine, such a timers and clock drift. I started having a lot of problems with Apache recently… once in a blue moon, on restart, it would spawn a single process with no PID file, and hang there. Checking out error_log pointed out that apache would apparently hang while generating the Secret seed for Digest authentication (mod_digest). Disabling mod_digest would have worked, but sadly, I use it. This could be found in error_log:


[notice] Digest: generating secret for digest authentication ...
[notice] Digest: done
[notice] Apache configured -- resuming normal operations
[notice] caught SIGTERM, shutting down
[notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec2)
[notice] Digest: generating secret for digest authentication ...

Fortunately, I cobbled up a fix. Updated! Read on for a more elegant fix.

So it hangs while generating the secret for digest authentication. My gut feeling was that it had to do with the system running out of entropy (random data, that is). Probably waits for apr_generate_random_bytes() to complete but that never happens. I immediatly checked out the kernel for available entropy from /dev/random and friends.


# cat /proc/sys/kernel/random/entropy_avail
16

Now, I have no clue why, but entropy ran out (not quite, but close). Probably another whacky quad SMP issue. I’m starting to drive the Gentoo devs and maintainers crazy with my strange hardware.

Note: Before proceeding, please read the updated section below.

So, to remedy it, I fetched and installed rngd from the rng-tools package in Gentoo Portage. Check out your distro’s package repository for it or something similar. This tool allegedy gathers random data from I/O transactions and hardware stuff around. I compiled it, installed it and launched it. The Gentoo way:


# emerge -va rng-tools
# rc-update add rngd default && /etc/init.d/rngd start

Once that was taken care of, I checked the available entropy by querying the kernel once again.


# cat /proc/sys/kernel/random/entropy_avail
6854

That’s much better. To my surpise, Apache finished loading instantly the split second rngd started up. Schweet. Sounds like that fixes the problem. I have no idea why it occurs right now, but this works around it elegantly.

UPDATE! I figured out what the greasy poop was going on. Installing rngd will gather entropy using /dev/urandom if no hardware RNG is found by default. I didn’t pay much attention to that detail at the time. However, you will notice that when installing the Apache Portable Runtime on Gentoo (dev-libs/apr) the use flag urandom is available. Using euse (from gentoolkit) to get info about it returns the following:


valkyrie ~ # euse -i urandom
global use flags (searching: urandom)
************************************************************
no matching entries found

local use flags (searching: urandom)
************************************************************
[-    ] urandom (dev-libs/apr):
Use /dev/urandom instead of /dev/random

This will make apache fetch randomness from /dev/urandom directly, therefore giving the same result as rng-tools. The proper solution would be to just re-emerge dev-libs/apr with the urandom use flag enabled.

I tried it and it works beautifully, and looks a lot more elegant to me.

29 Responses to “Apache hangs on Digest Secret generation”

  1. eddyod says:

    Well, i don’t have any quad xeon machines around but i was having the same problem, apache would take almost 5 minutes to do a restart.
    i installed that rngd (i’m on gentoo as well) and bang zoom, working nicely,
    thanks,

    eddyod

  2. mr_daemon says:

    Hey, glad this could be useful to at least someone :)

    I’ve seen a lot of posts on the subject around message boards but with no answer…

    Thanks for the comment!

  3. Shadow aok says:

    Same here. My apache takes 5 minutes and 30 secondes to restart (gentoo on a Celeron 1,7 with 512 MB DDR) and not it tooks 3s !!!
    I searched a fix for a long time and thanks to you, it works fine now :) Santé !

  4. Paul says:

    I had the same problem just now. I could strace the apache2 pid and it would be stuck reading file handle 23, and “ls -l /proc/[apache2pid]/fd” showed that fd 23 was /dev/random. Adding rngd immediately fixed that problem. Thanks!

  5. Peter says:

    Thanks, that seem to solve my hanging problem also.

  6. Seth says:

    You’re a god!! I’ve been having this issue for a while now with no luck. I’m also running a Gentoo machine but with very standard hardware. ! proc and 512MB of memory but when I cat’d /proc/sys/kernel/random/entropy_avail I only had 32, after the emerge I was up to 2200 :).

    Cheers,

    Seth

  7. Bastian says:

    Our single CPU p3 ML370 G2 with 1 gb has the same problem. I also run Gentoo. This solved it. Thanks for the help :-)

  8. Graham King says:

    Worked perfectly on my UML host. Apache startup time is down from 1.5 minutes to about 1 second. Thanks very much.

  9. adlerweb says:

    Nice Tipp

  10. the says:

    you made a happy man :D Thk’s a lot Apache now startup take less than 1 second on our dual xeon. :D

  11. mr_daemon says:

    Thanks for the comments everyone :) I’m glad I could help!

  12. der_flo says:

    you’re the man!
    damn!

  13. Jim says:

    Wow. This was just what I needed, thanx!!! :)

  14. Brett says:

    Thank you!

    You have sorted about 8 server of mine out!

    Yes you are the MAN!

  15. as0t0 says:

    Freakin’ awesome!
    Thank you so much for this! I’m new to Apache, but I’m running LAMPP on a Gentoo box and it has been doing this. It’s running an AMD 64 3200+ with 2GB of RAM, but Apache just refused to start up with any kind of speed. Just like you said… as soon as the emerge finished, Apache kicked off all of my queued requests.

    Thank you again! You rock!

  16. Kristaps Kaupe says:

    Blogiem.lv 2 serveri…

    Taks, beidzot ir palaists Blogiem.lv atsevišķs MySQL serveris, bet uz vecā tīmekļa servera paralēli griežas MySQL slave, kurš gan šobrīd kalpo nu jau izmanto.

    Turpinājumā pāris publiskas piezīmes ar šovakar piefiksētām lietām……

  17. Peter says:

    You’re the man.. thank..

  18. Mike Rosile says:

    I gotta agree, you’re the man! Seems a lot of us have this problem after all? Wonder what it is? Kernel problem? Timer issue on an SMP machine? I’m just glad its working!!

  19. raptorized.com » Blog Archive » Update on the Apache-Hangs-On-Digest-Generation topic says:

    […] Check it out. […]

  20. vbali blogja »  Digest: generating secret for digest authentication… says:

    […] Egészen mostanáig, mikoris belefutottam ugyanebbe a jelenségbe egy sokkal kisebb teljesítményű gépen, ahol a titok generálása néha még fél órába is beletelt. Ezt már nem tartottam megengedhetőnek így ismét keresgéltem a megoldás után, így találtam rá erre a postra: Apache hangs on Digest Secret generation. Örömmel nyugtáztam a hozzászólásokat olvasva, hogy nem csak nekem okozott fejfájást a hiba elhárítása. […]

  21. Ash says:

    same problem as everyone, glad that you ranked top 5 in google :p
    (NB. http://forums.gentoo.org/ currently down for maintenance)

    Now it seems that everyone that have this problem is running a Gentoo box (shouldn’t the “urandom” USE flag be turned on by default ?)

  22. Scott says:

    Thanks for this, the slow starts were driving me nuts!

  23. Alan J Castonguay says:

    Apache slow to start, same issue as everyone else. Re-emerged dev-libs/apr with urandom, works like a charm. Thanks for the writeup.

  24. dn says:

    nice dude! thank you very much for sharing your experience ;-)

  25. Aron Kovacs says:

    Hey
    I am so glad i found this post :) I have re emerged dev-libs/apr with urandom USEflag and now im doing a revdep-rebuild (its needed) and voilá
    cat /proc/sys/kernel/random/entropy_avail shows 225 now (very old / slow machine) it had 8 before :)

  26. Aron Kovacs says:

    Ill have to rephrase that entropy_avail is up to 1027 right now.
    Thanx alot

  27. Deep says:

    you rock mate… Damm

  28. Moonwick says:

    Wow, what a strange thing to run out of. We ran into this problem; thanks for solving it and posting about it so I wouldn’t have to figure it out for myself.

    Crazy, though. I’ve got enough things to worry about without having my computers run out of entropy, of all things…

  29. rsvanda says:

    you saved my weekend (on sunday evening :-) ). all works fine now. THANKS!

Leave a Reply

Line and paragraph breaks automatic.
XHTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <em> <i> <strike> <strong>