It is my intention to make this a series of short postings, each looking at one or two specific tunables. Hopefully, this “Part 1” will be followed by a “Part 2” and a “Part 3” and so on — until I run out of interesting things to say on the subject. But I’m not the most prolific blogger, so don’t hold your breath!

Read The Paper

I suffer from blogger’s block. Part of my problem is gaugeing the amount of context which needs filling out before I can get to the meat of what I want to say. For this short series of posts I am not going to go into lots of detail about Solaris multithreading architecture. In Solaris 8 we began a U-turn on our famous MxN application threads architecture when we introduced the alternate 1-1 implementation. In Solaris 9 we dropped the MxN implementation altogether, and the thankless task of explaining why this was a good thing fell to me. I’m going to assume that you’ve read Multithreading in the Solaris Operating Environment, most of which still applies in Solaris 10 and <a
href=”http://opensolaris.org”>OpenSolaris (we’ve just made it even better).

Heed The Warnings

“It betrayed Isildur to his death. And some things that should not have been forgotten were lost. History became legend, legend became myth, and for two and half thousand years the Ring passed out of all knowledge. Until when chance came, it ensnared a new bearer. The Ring came to the creature Gollum, who took it deep into the tunnels of the Misty Mountains. And there, it consumed him. The Ring brought to Gollum unnatural long life. For five hundred years it poisoned his mind. And in the gloom of Gollum’s cave, it waited. Darkness crept back into the forest of the world. Rumor grew of a shadow in the East, whispers of a nameless fear, and the Ring of Power perceived. Its time had now come.” — J R R Tolkien, The Lord Of The Rings

The stuff I’m going to mention used to be hidden and was intended solely for our own use — not yours. We believe Solaris should perform well out of the box, and that the provision of tunables amount to an admission of failure. We have never formally published or documented these things before because: we don’t think you really need them; we don’t want them to become de facto standards; and because we need the freedom to remove or change them in future. In many cases tweaking these tunables will actually hurt application performance. And they are a very blunt instrument, affecting all the threads and synchronisation objects within a process. So although some regions of your code may speed up, other regions may suffer, with no net gain overall.

Please use this information responsibly. It is unlikely ISVs have tested their applications with anything other than the default values — I know we don’t. Be up-front with your suppliers (including us) when dealing with support issues, and make sure you eliminate these tunables early from their inquiries.

So Why Bother?

With the advent of OpenSolaris some things which should have been forgotten are going to be found. Indeed, you will find the names of all the tunables I’m going to mention here.
We can’t hide this stuff any more — nor do we want to. With OpenSolaris we say “what’s ours is yours”.

Of course, there may be cases where fiddling with these tunables actually helps application performance. Of course, this may be because the applications themselves are poorly written, but it also may indicate that we have further work to do in improving Solaris performance out of the box. Please do share your data! And sharing this information with your suppliers — especially some of the debugging features — may actually speed the resolution of some support issues.

The Secret’s Out

For now, I’m just going to tell you what the tunables are. You’ll have to wait for subsequent posts (or go, grok the source) for an understanding of what they actually do and how they may be useful to you.

In
/usr/src/lib/libc/port/threads/thr.c we find these names:

QUEUE_SPIN
ADAPITVE_SPIN
RELEASE_SPIN
MAX_SPINNERS
QUEUE_FIFO
QUEUE_VERIFY
QUEUE_DUMP
STACK_CACHE
COND_WAIT_DEFER
ERROR_DETECTION
ASYNC_SAFE
DOOR_NORESERVE

The function set_thread_vars()
scans the environment variable list looking for these names prefixed by “_THREAD_” or “LIBTHREAD_”. The latter is deprecated now that libthread has been folded into libc, but provides some compatability back as far as the alternate thread library implementation in Solaris 8 (although there have been some changes along the way which I am not going to document in these postings).
The function etest()
specifies a maximum value for each variable and ensures that any user-defined value falls between zero and this limit. The default values are defined elsewhere in the code. In OpenSolaris the tunables are defined as follows:

envvar                          limit           default
_THREAD_QUEUE_SPIN              1000000         1000
_THREAD_ADAPITVE_SPIN           1000000         1000
_THREAD_RELEASE_SPIN*           1000000         500
_THREAD_MAX_SPINNERS            100             100
_THREAD_QUEUE_FIFO              8               4
_THREAD_QUEUE_VERIFY**          1               0
_THREAD_QUEUE_DUMP              1               0
_THREAD_STACK_CACHE             10000           10
_THREAD_COND_WAIT_DEFER         1               0
_THREAD_ERROR_DETECTION         1               0
_THREAD_ASYNC_SAFE              1               0
_THREAD_DOOR_NORESERVE          1               0

All but the last one also exist in Solaris 10 (and for now, the LIBTHREAD_ variants will also work). Making a change is as simple as setting an environment variable (just remember that all envvars are generally inherited by child processes).

I hope this has whetted your appetite for next next post, but in the meantime “hey, be careful out there”!

Tag: Solaris, OpenSolaris