published on 2009-11-20 21:41:00 in the "cuddletech" category
Role models aren't something we have few of; sad that perhaps the most recent one comes from a beer commercial:
I mean, come on... his advice on careers "Find what you don't do well.... and don't do that thing." Classic!
Need something more expansive? Learn Chinese! If you find it difficult, try to learn Japanesse... and then you'll go back and appreciate how much easier Chinese languages are.
Not intellectual enough? Need to stretch those brain cells a bit more? Then, I ask, what is justice? As a Christian I have all those answers, laid down thousands of years ago, but since apparently folks like to re-invent the wheel (something King Solomon explained to us about 1,000 BC... "There is nothing new under the sun"), try Harvard's Michael Sandel discussion on Justice. A fun and engaging discussion in one of Harvard's beautiful facilities, exploring the "Moral Side of Murder". Its an enjoyable metal excersize and well expressed.
If your reading this post on an aggregator or via RSS and don't see the embedded video, just come here to cuddletech to see it properly.
published on 2009-11-19 18:57:00 in the "cuddletech" category
Lots of folks have switched to Mac, its the most commonly used laptop in the Bay Area now. Sometimes people give me flack for using it, but I'll tell you why I use a Mac laptop:
It just works! When going to a client site, a conference, or just a cafe, there is nothing more embarrassing than spending 20 minutes trying to get your l337 *NIX laptop to connect to wireless or properly DHCP or work with a printer. This isn't as big a problem as it once was but it can still happen. This is especially the case if you ever do a presentation where your fiddling with things in front of 30+ people. Mac's just work, period.
The Apps are high quality! Thanks to the Linux desktop invasion we have a lot of great apps for *NIX; however Mac apps have a very high standard for quality, all work more or less similarly, and there are lots of great apps. The problem I have on Windows these days is that there aren't as many great apps for Windows as there are for OS X.
Its UNIX! This is the most important fact for me, its a real desktop OS with a real UNIX underneath. I was a Mac hater prior to OS X, but developed a love affair with NeXT... when the two converged in OS X I was a happy camper indeed.
The Apple Laptops are the best on the market! I can not find a PC Laptop with the same build quality and durability of the Apple's. Most PC's use cheap plastics, are too thick, too flimsy, etc. The MacBook Pro 15" Aluminum is what I still use and love. The size is absolutely perfect, the thing is solid, and very comfortable to use. The power adapters are even better. Even if I wanted a machine just to run Solaris on metal, I'd want a MacBook Pro over any PC laptop available. In terms of hardware you really do seem to get what you pay for.
Now, please note that I do not have nor do I ever plan to have a Mac desktop! For my daily work I need a real UNIX Workstation. I prefer to work with Enlightenment, Eterm, and have a real Solaris system on which to work. Without my desktop I can't accomplish real work, but for the road I need my MacBook Pro.
So here are some of my "must have apps" for OS X:
iTerm: It once was that OS X's terminal was pretty basic and pathetic, glTerm and iTerm filled the void. Since that time the default terminal application has improved significantly making iTerm unnecessary, but I continue to be faithful to it.
Adium: Adium is the best multi-protocol IM client available for Mac. While iChat AV is fantastic for voice and video "chat", I want to keep my desktop tidy which means I want IRC style chat in multiple tabs, not windows. I just can't stand having a real discussion in those iChat balloons.
NewsFire: Best RSS reader, imho. The primary advantage to Newsfire is that it doesn't make RSS look like email! Email feels like work, I just want to flip through RSS and see whats news. Newfire is free and really spiffy.
TrueCrypt: I'm not a really big crypto freak, I wish I were, but I'm lazy. Never the less, at some point you'll go on the road and Sysadmins are bound to have text files containing sensitive information. TrueCrypt makes it easy to create a small encrypted drives on which to store that data. Plus, the virtual drives it creates are cross-platform, so your not locked into only retrieving the data on Mac like other encrypting archive apps.
Things: I think its the best todo application available. Its light-weight and easy to use. OmniFocus is a much more structured application and I think is good for people who need rigorous structure to keep them honest, but Things can be made to do almost everything OmniFocus can do, if you choose to, or be used much more casually.
RealVNC: The most popular VNC Viewer application for OS X is "Chicken of the VNC". I love the name, love the icon, but a lot of times it doesn't work for me. RealVNC isn't so sexxy but works every time without a problem.
Colloquy: Great IRC application. Many *NIX folks will prefer a more traditional terminal based IRC client, but if your an Xchat users who's looking for a nicely integrated IRC client for OS X Colloquy is the best imho.
VirtualBox: Very powerful and free to boot. I use both VirtualBox and VMware Fusion. Honestly, VMware is slightly faster, but VirtualBox is still fantastic and the additional portability is handy.
Apache Directory Studio: If there is one nifty app the Windows boys have its Softerra LDAP Administrator. Apache Directory Studio is the best alternative I've seen, and I think will ultimately surpass Softerra's capabilities.
iShowU: Best screen recording app period. Very easy to use, very flexable and lightweight. When creating screencasts I recommend using the Quicktime Animation CODEC; you'll be happy with it.
globalSAN iSCSI initiator for OS X: Its sad that even in Snow Leopard we don't have an Apple supplied iSCSI Initiator, but thankfully globalSAN has us covered. Its free and works very well with COMSTAR.
Cornerstone: I didn't think Subversion needed a GUI... but Zennaware Cornerstone changed my mind. Its expensive, but if you do a lot of SVN work you won't want to miss it.
I'll add some more to the honorable mention list...
Textmate
iWork '09
iLife '09
Skitch
iStumbler
Netbeans
Navicat Lite
OmniGraffle
...
On the hardware side, every UNIX Admin must be able to access an RS-232 serial console. This fact kept me away from Mac laptops for a long time. Which is why you need this:
The Keyspan Serial-USB Adapter. Buy one, download the Keyspan Assistant software and install Zterm. Good to go!
Finally let me point out 2 things which are already in Leopard that you may not be aware of:
First, with the OS on the Install disk is the Apple Xcode IDE. Along with Xcode is the koolest GUI for DTrace you'll ever see: Instruments Its really amazingly awesome and a must see.
Secondly, OS X includes native Kerberos support and a ticket management GUI which is sort of buried: /System/Library/CoreServices/Kerberos. If you use Kerberos at all drag that binary onto your doc for quick access. Several other hidden gems can be found in the same directory.
This is a really exciting thing for us. This is the first time we've taken funding. We've really been proud of the fact that we haven't needed funding, but the benefits that come along with an investment from Intel are fantastic and just that relationship alone is exciting.
This is a big announcement not only for Joyent, but for OpenSolaris as well. We're thrilled that Intel supports not only what we're doing, but also how we're doing it. Combined with our recent expansion into China, we have a lot to be happy about.
published on 2009-11-17 21:54:00 in the "cuddletech" category
My talk at LISA is now available. This is a 1 hour version of the ZFS in the Trenches talk. As always I hope that you find it informative and at least a little entertaining. Slides are here).
Because of Deirdre countless people around the globe can participate and learn from important events. Not only does she spend a mind-boggling amount of time going to these events, but she has done a fantastic job producing very high quality content, and I think is setting the bar in community video presentation. We just don't get this kind of content from other top tier vendors and I really hope they take notice of her efforts and the benefit to Sun's current and prospective customer bases.
So please join me in extending your support and appreciation to Deirdre and everyone at Sun that makes these events accessible to the whole world!
published on 2009-11-13 02:47:00 in the "cuddletech" category
For sometime now I've gone back and forth on what is my personally preferred (LDAP) directory server; in particular between Sun Directory Server Enterprise Edition, OpenDS, and OpenLDAP. Each has advantages and trade-offs:
DSEE: Not free, complex, but well trusted, exceptional scalability
OpenDS: Free, super simple install and management GUI included, best starter directory for sure, but relatively new to the scene and thus needs to build more cred.
OpenLDAP: Not the best scalability, not the best replication or feature list, but very extensible, extremely well known and supported, free. Advanced features much more straight forward than competitors due to flat config file (especially ACLs, TLS, etc)
So I put it to my loyal and educated readers... which is your directory of choice?
published on 2009-11-08 07:09:00 in the "cuddletech" category
One of the many facinating things I discovered at LISA was that almost no one has heard of (no one at my sessions at LISA anyway) Dr. G: Medical Examiner
Jan C. Garavaglia, M.D., (aka "Dr. G") is the chief medical examiner for the District Nine (Orange-Osceola) Medical Examiner's Office in Florida. An assortment of her cases are strung together to create the weekly show on Discovery Health "Dr. G: Medical Examiner."
Another...
I started watching the show because Tamarah is a Discovery Health channel junky. She loves the medical detective shows such as Dr. G and Mystery Diagnosis. I am particularly drawn to the show when I do a lot of postmortem work on systems (aka: "core dump analysis"). Medical practice is a great model for how to approach problems systematically and to follow the story to its conclusion. I suspect many geeks (at least those who don't pontificate about not owning a TV) would also enjoy it.
published on 2009-10-29 22:13:00 in the "cuddletech" category
News is popping up that will interest those interested in the Sun/Oracle deal.
I've made peace with Jonathan Schwartz but those who haven't will no doubt love to bash on his pay. The first data I've seen in a while comes fresh from the AP: Sun CEO's pay package cut by a third in '09. According to the article, his 2008FY compensation was $11.1 Million, and it looks like 2009FY will come in at only $7 Million, information which came from the Sun Proxy filing with the SEC on Wednesday. One thing I have always wondered is what his personal driver costs... apparently he is company provided and costs over $45,000 per year (figured by the $55,000 spend on both driver and 401K match).
So, hey, even guys who make millions of dollars per year try to max out their 401K.... what does that tell ya? :)
Based on the same proxy filings, El Reg reports yet more on the compensation front. They report that Scott McNealy owns approx 2.3% of Sun. They estimate that if he excercises 3.1M in options by the end of Dec his cut will be $164.5M. Going on, El Reg reports that Jonathan has almost 1.5M options and 592K shares... so he comes away with $19.8M.
The Reg also counts up the total number of layoffs in the past 12 months at 8,000 (I assume that includes the 3,000 currently being chopped).
We got some nice news from Oracle this week by way of a FAQ: Oracle and Sun Overview and FAQ (Dated October 27, 2009). My questions regarding X86 and Solaris were included:
What are Oracle’s plans for Solaris?
Oracle plans to spend more money developing Solaris than Sun does now. The
industry leading capabilities of the Solaris operating system make it the leader
in performance, scalability, reliability, and security – all of which are core
requirements for our customers. Oracle plans to enhance our investment in
Solaris to push core technologies to the next level as quickly as possible. Today
there are more applications available on Solaris than any other operating system
in the world. In addition, the combination of Oracle and Sun engineering
teams in database and operating system open up a new set of opportunities
to create exciting innovations for customers with respect to performance,
operational efficiency, security, and cost of ownership.
What are Oracle's plans for x86?
The extremely broad and volume use of x86 makes it an important
building block for servers as well as other parts of the combined Oracle
and Sun portfolio. We plan to continue to engineer server and appliance
products based on x86. In addition, x86 is of course a key element of both
Sun and Oracle's software portfolio, with Solaris and Oracle Enterprise
Linux as well as all of the software of both companies robustly sold and
supported in the x86 marketplace.
So this fits perfectly in line with what we've heard to date, namely that Solaris rules and X86 is a critical offering as part of other offerings.
Finally, the Financial Times is reporting that Russian Anti-Trust is making life rough and FT perhaps foolishly plays up the headline by asking "is the deal about to unravel?" Read it for yourself but I'm not jumping to any conclusions.
Will this never end? Despite Oracle's pledged $9.50 per share, JAVA has dropped to $8.27 today, suggesting a lack of confidence. And I think most of us in the various communities have come to terms with the prospect of Oracle and are ready for things to get moving. There is a lot to suggest that Oracle is already calling the shots at Sun to various degrees, as we saw at Oracle OpenWorld recently. Besides that, at this point Sun is damaged beyond hope of repair... if this deal doesn't close soon we're all going to be in a world of hurt.
Lets get this deal done! Give the execs their money so they can retire and stop f***ing the company, and lets go kill IBM.
published on 2009-10-23 17:51:00 in the "cuddletech" category
Little hobby electronics company SparkFun Electronics just got a cease-and-decist from SPARC International because "SparkFun" may be confused by consumers as being associated with the SPARC trademarks.
Come on guys.... lets be level headed. I think its a clever branding and they are in no way confused with SPARC processors or any of the companies that are members of SI.
published on 2009-10-21 06:36:00 in the "cuddletech" category
Recently we talked about Solaris Auditing (BSM) in the Real World. Like BSM, Extended Accounting is a fantastic feature of Solaris that is utterly useless without tools. Solaris goes so far as giving you the capability but not so far as to hand you the rest of the solution on a silver platter. On one hand this means that the technology isn't pigeon holed due to the capabilities of a single tool, but at the same time it creates a barrier to entry that causes many people to simply ignore it all together. So, yet again, let me provide a simple tool to fill some of that void.
In a previous post, Solaris Extended Accounting, I described Extended Accounting and provided two scripts to get you started, one was a PERL script to dump Extended Accounting ("exacct") data files and the other was called "prettyproc" which output Proccess Accounting files in a more human friendly way. This post should be viewed as Part 2 of that post.
When & How to use Extended Accounting
The most basic explanation of Extended Accounting is this: a facility that records certain events upon completion for later analysis. Those certain events depend on which of the four accounting types we're using. For processes, the cumulative data maintained by Solaris microstate accounting is written into a single record as process termination. For tasks, which are groups of processes within a single project, the same applies but recorded on each task termination rather than process. For (Crossbow) net, aggregate network utilization is written out on regular intervals (15 seconds). We'll ignore IPQoS "flow" Accounting entirely for the time being.
So the first thing we should say is that Extended Accounting is not a monitoring facility. If you want to know how much CPU or Memory is being used at some given time you should rely upon Kstats or /proc statistics on a polling schedule.
What Extended Accounting is good for is reporting. Consider 'net' accounting; every 15 seconds a record is created for each data link (dladm show-link). You could easily create a report at some interval (hour, day, week, month?) for both total bytes/packets sent/recieved on each link or great a graph or perhaps most likely calculate 95th percentile on the links. Now, in this case of 'net' accounting you could also use an external system to poll the data remotely via SNMP or locally via kstats, but this might serve as a better "definiative" local record.
Proc accounting is fuzzy ground though. The best way I can explain process accounting is to imagine that every time you executed a command Solaris was secretly running "time(1M)" and then storing the output on your behalf.
benr@quadra Downloads$ time tar xfj flash_player_10_solaris_x86.tar.bz2
real 0m0.763s
user 0m0.705s
sys 0m0.070s
This is, essentially, whats happening! Solaris maintains a lot of detail on what processes are doing (known as "microstate accounting"). Normally, when a process terminates that data is simply discarded, however if Process Extended Accounting is enabled its dumped out as a record! From this record we can see interesting stats such as when the process started, when it finished (real time), how long it spent cpu time in kernel-land (sys time), how long it spent cpu time in user-land (user time), how many context switches it made, how much swapping it did, what its average RSS memory usage was, etc, etc, etc.
But as wonderful as this is, I have to make it crystal clear that this data isn't written out untill a process terminates! If MySQL runs for 4 months, it outputs a single record when it was finally shut down, and that record is the accumulation of that full 4 months of running!
Here is the exception. Proc and Task records can be "full" or "partial". When a process/task terminates and creates a record, that's a "full record". However, using "wracct" we can force a process or task to create a "partial record", which is essentially a way of saying "Just tell me what you've got so far!" The rub is that, in the proc case, that data is cumulative, so if you wanted to report on what a process has done in the last 24 hours you need to write a partial record every 24 hours and the find the difference between the partial record yesterday and partial record today. Talk about fun.
Now, besides all that, who actually bills users or reports usage based on total CPU time? Total context switches? This isn't the 1970's nor is this likely to be a Super Computer reporting computational time. In short, the data probably isn't terribly useful as a basis for billing in this day and age without some creative thought.
So then lets think... what can we determine from the data. Based on CPU usage we could determine what the top 5 CPU consuming processes were. Based on average RSS usage we could determine what the top memory consumers were. So on and so forth. Interesting perhaps... but worth it?
Go back to what I said about running "time" on every command. This data could be of used for capacity planning or, with some intelligence, behavior monitoring. Are your users complaining about commands taking too long to run, but when you ask how long they give you a bogus number or simply shrug? Extended Accounting can tell you. Are batch jobs running at night but want a record of when they started and how resource hungry they were? Here is a way that doesn't involve writing wrappers!
In short, Extended Accounting is a pretty lousy billing system on todays mulit-core systems, but it can provide useful historical statistics to questions that might be otherwise difficult to answer.
Practical Tools
The first tool I'll provide you with is a PERL replacement for the Solaris included /usr/demo/libexacct/exdump.c: exdebug.pl. This tool offers the following advantages:
exdump.c hasn't been updated for the new Crossbow provided 'net' accounting data; exdebug.pl is module agnostic and works with them all.
The output is just much cleaner and intuitive for exploring what ExAcct can do for you.
Its implemented in PERL making it easier to get in there and build something, rather than dealing with the libexacct learning curve in C. If nothing else you can quickly prototype and then re-implement in C.
In the above example you'll see the variety of objects offered by the net accounting module, including link descriptions, link statistics ('testzone0' is a VNIC and 'e1000g1' is a physical interface), and flow statistics (inbound_ssh is a flowadm defined flow).
The second tool is exacctly, a human friendly Proc Extended Accounting dumper. It is also implemented in PERL and in fact was derived from the exdebug app above.
benr@quadra exacct$ acctadm proc
Process accounting: active
Process accounting file: /var/adm/exacct/proc
Tracked process resources: extended
Untracked process resources: host
benr@quadra exacct$ pfexec ./exacctly /var/adm/exacct/proc | more
Creator: SunOS
Hostname: quadra
ZONE UID GID PID CMD | Real User Sys | Start Date | RSS AVG RSS MAX SysCalls Swaps
----------------------------------------------------+--------------------------+--------------------------+--------------------------------------------------
global 0 0 1922 acctadm | 0.07 0.00 0.01 | Tue Oct 20 03:10:01 2009 | 524 K 12904 K 450 0 | FULL
global 0 0 1920 sh | 0.07 0.00 0.00 | Tue Oct 20 03:10:01 2009 | 2036 K 12904 K 103 0 | FULL
global 25 25 1924 sendmail | 0.10 0.01 0.01 | Tue Oct 20 03:10:01 2009 | 1912 K 12904 K 543 0 | FULL
global 0 0 1927 sendmail | 0.01 0.00 0.01 | Tue Oct 20 03:10:01 2009 | 2288 K 13172 K 267 0 | FULL
global 0 0 1923 mail | 0.10 0.00 0.00 | Tue Oct 20 03:10:01 2009 | 504 K 12904 K 169 0 | FULL
global 0 0 1921 sh | 0.11 0.00 0.00 | Tue Oct 20 03:10:01 2009 | 920 K 12904 K 102 0 | FULL
The output is really wide, but everyone should have a big ol' screen these days. Notice the depth of information here. For each terminated process we see the zone it was in, user and group, PID and command name itself (ExAcct doesn't record arguments), then we see real/sys/user time in seconds (ExAcct actually has nanosecond granularity, so these are rounded numbers), the start time and other goodness. The last column reports whether the record is full or partial.
This tool is, in and of itself, useful for many administrators to start using Extended Accounting that might otherwise have ignored it. Even more so, I hope it sparks your interest and imagination as to the possibilities! Just think of all the ways to amaze your boss and fellow admins!
Data File Rotation
Like any log, don't be lazy and forget to rotate those files or you'll have a mess on your hands. Rotating your extended accounting data files will make them easier to dissect and consume less disk. Here are some examples lines you can drop into /etc/logadm.conf, Solaris's default log rotation tool:
These examples will rotate each day (-p 1d) and keep 7 logs (-C 7) before destroying. The important bit is that you can't just mv the file, you need to stop accounting, rotate, then resume it.
Remember to ensure that logadm isn't commented out in the root crontab.
Parting Thoughts & Cautions
Before I wrap up, I want to note something about Process records. Here is one as seen with exdebug:
Okey, lots of data, lots of goodness. Notice EXD_PROC_BLOCKS_IN, OUT, and CHARS_RDWR? They are useless. I can't go into why here, but don't get excited about them or bother doing anything, the values are crap. If your a veteran Kstat diver you'll recognize similar values in the Kstat cpu_stat class... same story.
Hopefully this post as helped provide you with a more practical understanding of Extended Accounting and provided you with some resources to get in there and use the data. There is a wealth of possibilities if you just avail yourself of them. :)
published on 2009-10-16 22:20:00 in the "cuddletech" category
Quite some time ago I wrote about this subject: I See You!: Solaris Auditing (BSM). As much information is out there regarding Solaris Auditing the post was well received and pretty popular but I've never been happy with where I left it. Many people feel that auditing is "difficult". Why? Because its hard to enable? No, thats simple, just run bsmconv and your done, edit 2 simple configs in /etc/security to tweak it... whats hard about that?
I'll tell you why auditing is a pain in the butt... because for all the dozens (or hundreds) of tutorials almost none of them teach you how to actually use the auditing data. So you've got these really great audit trails but now what? This blog entry is about filling that void, similar to the post I did about actually using BART: Solaris Automated File Integrity Checking: bartlog.
BSM Basics
As I said in my former post, enabling BSM is simple. There is a convenience wrapper in /etc/security which will turn on the auditd SMF service and add the following to /etc/system:
set c2audit:audit_load = 1
You reboot and auditing is going. So what about tweaking what it collects?
The following is my recommendation for /etc/security/audit_startup, these policies change the way auditing collects data:
The "+cnt" policy says that even if auditing can't record data (usually because /var/audit is out of space) keep running. In a super high secure environment you would remove this so that if auditing wasn't able to function the box would halt. Next, the "+zonename" policy adds the zonename to each audit entry, if you use Solaris Containers you want this policy. The "+argv" policy is very important, if you do not use this policy you'll see commands executing but not the arguments, and typically when your auditing for security you aren't just interested in the command but how its being executed. Additionally, you could add the "+arge" policy which would include the environment with each command, but that seems like major overkill to me.
Now, just a moment on the "+perzone" argument. By default (meaning, without +perzone) auditd in the globalzone will record everything on the box regardless of which zone it occurs in, this is why its so important to use the "+zonename" policy. So if zone "oracle1" runs a command, an audit record is made to the audit trails in the global zone. There are at least two potential problems with this: 1) The users inside the zone can't access the audit trails, and 2) The users inside the zone might not want to be audited. So by setting "+perzone" in the globalzone, each zone will audit itself and only itself. That means that the globalzone only records audit events that occur in the globalzone. It also means that each zone can choose to enable auditing within their own zone by enabling the auditd service and tweeking the configs in /etc/security.
Moving on... the other important config is /etc/security/audit_control, which determines what events are audited by default. I recommend the following:
So "flags" define which classes we're going to record by default. This can be changed per user in the audit_user file (maybe you really don't trust a particular user?). "lo" is login/logouts including su activity. "ex" is executions. So these two flags together record people coming and going and running commands. I recommend this as the default and suggest that you strongly avoid auditing more unless you know what your doing. The "naflags" are like "flags" but apply to events that are "not attributable" to a user (such as a failed login for a user that doesn't exist). If you need to know more about flags and configs and syslog, refer to my previous post.
Audit Trail Maintenance
Now that auditing is running, you'll see audit trails in (by default) /var/audit. The format is "date.date.hostname", which signifies that the audit trail is "terminated", or complete. The current audit trail will be "date.not-terminated.hostname".
There are 2 important tasks relating to maintaining these audit trails. First, we need to rotate them to keep them from growing too large. Secondly, we need to move them from the unsecure system (otherwise why would you audit it?) to a safe place.
Rotating audit trails is simple, run the "audit -n" command to terminate/close the existing audit trail and continue auditing to a new file. So the simplest way to invoke daily audit trail rotation is by adding the following line to the root crontab:
## Rotate the Audit Logs Nightly at Midnight.
0 0 * * * /usr/sbin/audit -n
So now your terminating audits every day, but you now need to get the audit trails off the local system. Some old documentation suggests mounting /var/audit as NFS... I'm not a fan of that idea. Instead, I'd recommend creating a script which runs the "audit -n" command above and then uses sftp or scp or something to move the audit trails to a centralized archive location. You might even want to compress them prior to sending, but the idea is simple enough.
One other method of storage would be to rotate the audit trail, immediately convert it to XML/HTML/text or whatever and then moving that.... but in my experience the raw audit files are much smaller than any report you produce, so compressing and storing them raw is probly the best policy.
Please note that the frequency at which you rotate and archive your audit trails depends on the sensitivity of the system. If a hacker is smart he'll notice that BSM is enabled and proceed to both disable it and destroy the audit trails. Therefore, in a highly sensitive environment you might archive as frequency as every 5 minutes! How often you archive is up to you and your environment. Every hour? Every day? Every week? It all depends, but I encourage you to spend a couple minutes thinking about it.
Okey, so now your rotating nightly and thinking about how to centrally archive the audit files, now what?
Reporting Part 1: The Boring Basics
Here's what you've always been told... use the auditreduce command to process the audit trails and then pipe the output to "praudit" to output it. Boring. Let me clarify this a bit.
praudit can read audit trails and produce ASCII text output or XML. You do not require auditreduce to use praudit. The most common method of using praudit is with the "-ls" arguments which creates an ASCII output containing one audit record per line. Its ugly and huge but it gets the job done. At that point you might use some script to parse the text file but I discourage doing this (we'll see why shortly). Output to ASCII only for debugging, nothing else.
Audit files get big, so the auditreduce command is sort of like "grep" for audit trails. It will read the raw audit trail and, based on the arguments, create a new raw audit trails containing only what you want. For instance, if I only want see login/logout records, I could do the following:
# auditreduce -c lo /var/audit/someaudittrail > new-lo-audittrail
So, in this way, we might produce several smaller raw audit trails based on the big master one. But there are lots of great options that can be handy. For instance, each audit record contains a "SID", Session ID. A session would start with login and end with logout and everything in between. So if we found a command execution that we find disturbing we would probly want to see everything done during that same session, so we could use auditreduce -s 12312312 /var/audit/someaudittrail | praudit -ls to see the entire session. Very handy indeed.
I highly recommend you take the time to look through the various search options offered by auditreduce(1M).
Okey, so all this you have probly heard before, so lets move on to some things you probly haven't seen.
Reporting Part 2: XSLT
XML makes storing data easier for programs, but its only minorly useful for humans. The way we transform an XML document into something more palatable is by creating an XSL stylesheet. By using an XSL Tranform (XSLT) engine, such as xsltproc we can transform XML into HTML or plaintext or XML-FO which is then used to convert to print formats like PDF.
Okey, why the XML review? praudit -x will output audit trails as an XML document. Look at the header of that document:
Do you notice the "xml-stylesheet" tag? Solaris ships with a proper XML DTD (Schema) but also an XML stylesheet for translation to HTML! Here is how you do it:
Using this method we could script a cronjob to produce a daily report in human readable format. Furthermore, Firefox and most other browsers can do XSLT transformations natively, so if you are using a browser on a Solaris system (so that the XSL and DTD are local) you can simply open the XML in your browser and see it in pretty HTML format!
There are 2 important take-aways on this. Firstly, creating useful HTML reports from audit data is really easy. Don't bother parsing out the praudit -s ASCII output. Secondly, and more importantly, you can spend a little time learning XSLT to create your own custom reports!
For example, I really want to see the audit report in a single table, instead of in bulleted lists. So I did just that. It took me about 30 minutes or reading and tinkering to get the basics down but it was much easier than I expected. Just copy the Solaris provided XSL and start tweaking it. Please, feel free to download and try out my modified XSL: benr_record.xsl. Please note that it is intended for "lo" reduced XML files and is far from perfect, this is for learning purposes only!
Hack it up and do some fun things. Put the data in the most useful form for you organization, add your logo to the output, etc. If you are feeling really hardcore you can download XSLT Design tools such as Altova StyleVision, but personally I found that it was easier for me to learn XSLT itself than to use the design tools.
Reporting Part 3: XML & PERL
XSLT is great, but there are limits to what it can do. If you want to create really comprehensive reports you'll need to actually parse the XML itself. The advantage of doing so is that you can loop the data multiple times to add roll-up statistics, such as a summary of sessions, number of executions, average executions per session, etc. You might be able to replicate this by using the auditreduce command but thats way more processor intensive and wasteful.
While you could use any language, being a SysAdmin, I feel most at home with PERL. Thankfully the XML::Simple module is included with Solaris, so going this route means you don't need to install anything new or potentially unsupported.
So with the power of BSM and PERL's XML::Simple at my fingertips, I decided to create a tool that could print audit trails in a really pretty and friendly way, and bsm_report is the result. Just look at how beautiful this is:
root@quadra bsm$ ./bsm_report.pl
The Incredable Human Friendly BSM Audit Dumper benr@cuddletech.com
USAGE: ./bsm_report.pl [-d] ( [-c ] -a ) | (-x /path/to/reduced.xml)
root@quadra bsm$
root@quadra bsm$ ./bsm_report.pl -a /var/audit/20091016225822.20091016225943.quadra
Reducing /var/audit/20091016225822.20091016225943.quadra ....
Processing /tmp/.audit-tmp.xml ....
C U D D L E T E C H A U D I T D U M P E R
Audit Begins: 2009-10-16 15:58:22.316 -07:00
Audit Ends: 2009-10-16 15:59:43.587 -07:00
login - ssh (failure) by benr as benr REMOTELY from lappy in zone global (3623559241)
login - ssh (success) by benr as benr REMOTELY from lappy in zone global (3415402787)
execve(2) (success) by benr as benr REMOTELY from lappy in zone global (3415402787) : /bin/cat -s /etc/motd
execve(2) (success) by benr as benr REMOTELY from lappy in zone global (3415402787) : /bin/mail -E
execve(2) (success) by benr as benr REMOTELY from lappy in zone global (3415402787) : cat /etc/shadow
execve(2) (success) by benr as benr REMOTELY from lappy in zone global (3415402787) : cat /etc/passwd
su (failure) by benr as root REMOTELY from lappy in zone global (3415402787)
su (failure) by benr as root REMOTELY from lappy in zone global (3415402787)
su (success) by benr as root REMOTELY from lappy in zone global (3415402787)
execve(2) (success) by benr as root REMOTELY from lappy in zone global (3415402787) : cat /etc/shadow
su logout (success) by benr as root REMOTELY from lappy in zone global (3415402787)
logout (success) by benr as benr REMOTELY from lappy in zone global (3415402787)
root@quadra bsm$
I have a couple more improvements to make to it and then you'll see it get its own page on cuddletech. I hope you can see the advantage of this. While I think bsm_report will be useful for a lot of people, more importantly it provides a useful example from which you can build your own tools.
Perhaps the best way to interact with audit trails is within a real database. Using this same method in PERL you could easily create a tool to pump the audit trail data into MySQL, PostgreSQL, Oracle, or, my favorite, SQLite. Imagine a centralized database for audit data and a PERL script on each node which, from cron, runs every so often to rotate the audit trails, convert to XML, and then read all that data into a centralized database. Nifty goodness.
Reporting Part 4: Existing Software
I noted earlier that BSM seems "hard" because of its DYI nature. While I'm sure hundreds or thousands of Solaris environments have great auditing infrastructures, almost all of those are custom and folks aren't sharing their tools, probly because they don't think anyone would care. I'm trying to change that. But I do not want to suggest that no other tools exist. There are 3 that I'm aware of:
BSMgui is a simple Java program which can read raw audit files and display them based on audit class. Startup the program, "open" an audit file, then click all the audit classes you want and execute a search. Nifty. Its old but by no means out of date!
the BSM Analyzer is a PHP application which gives you a web-driven way to search and report on audit trails. Its old too, but still valuable. If you (like myself) are interested in web searchable audit files this is the solution for you, or at least a great example of how to implement one!
Finally, SNARE "from InterSect Alliance, is a proprietary Log Monitoring solution that builds on the open source Snare agents to provide a central audit event collection, analysis, reporting and archival system." SNARE includes a Solaris Agent which integrates with BSM. I tried it on my Nevada box and had some minor issues but nothing serious. If you need a comprehensive end-to-end multi-platform auditing solution, have a look at it.
I'm certain there are more tools out there, namely in the form of plugins to suites like Tivoli, BMC Patrol, etc, but I won't explore those here.
Conclusion
Solaris Auditing is extremely powerful, but audit logs are pointless unless you can generate useful reports and store the data in an accessible and intelligible way. I hope you have a new appreciation for the variety of ways in which you can create meaningful and useful reports.
If you've created your own in-house tools for BSM Auditing, please consider sharing them. Maybe not all that sexxy, but there is a real need from users to have these types of tools.
Furthermore, if you have found this post helpful please let me know. If its popular enough I may convert it into a small book with much more depth.
published on 2009-10-12 07:18:00 in the "cuddletech" category
Tonight was a significant evening for Sun & Oracle. The opening keynote of Oracle OpenWorld 2009 was provided by Scott McNealy with an appearance by Larry Ellison. There is a lot to unpack here, so I'm going to break it down into sections.
The Acquisition
This is the first event I've attended as "press". As such I got all the press perks, namely access to announcement details in the press room and early access to the keynote for prime seats, in my case, right behind all the Sun Microsystems reserved seating. Talking with others there to cover the event was interesting in and of itself. Clearly everyone was looking for new information regarding the merger. None was expected but it was hoped for and we all listened attentively.
Here you can see (from left to right) Dr. James Gosling (creator of Java), Scott McNealy (Sun Chairman), and John Fowler (Sun EVP Systems). These were the speakers from the Sun side of the house, only Larry came up to represent Oracle.
Scott obviously thought the acquisition was a good thing and spent a lot of time about the history of Sun as an innovation driver (building his legacy). James came up to say that he thought it was a good thing for Java and the only hiccups along the way were with regard to Oracle learning now to interact with a community as large as that around Java. John showed off some of the new goodness from Sun and pointed to Oracle continued interest in Solaris, SPARC, and X86.
Larry wanted to hammer home the commitment they are making to Sun. He talked about the recent ads that Oracle's been putting out and how they are fighting against IBM who's trying to capitalize on the confusion. He re-iterated that Oracle will increase the money going into SPARC, Solaris, Java, and added to the list MySQL. He's very clear that nothing is getting chopped, he needs to whole company. With regard to MySQL he pointed out to Sleepycat (BerkelyDB) and InnoDB as things that Oracle owns and has invested in and been able to make some money with and intends to do the same with MySQL. He maintains that MySQL in no way competes with Oracle.
The more Larry talks the more comfortable it seems everyone is getting with this deal. Early estimates were that almost 50% of the company would be let go and there would be major changes in the companies product lineup. More and more those estimates are dropping below 30% and suggest that nothing will be cut, but rather pruned neatly into a more structured form. Best line of the nite was from larry, "We're in it to win."
The Benchmarks
Larry drove the point about synergies between Oracle and Sun home in 2 ways. The first was talking about the previously released Sun/Oracle ExaData v2 product (pictured above). The second was to show that with Sun's technology today, pre-acquisition, is the best platform available for Oracle even against IBM's monster POWER 595 system which consumes 76 standard racks. Sun's solution that beat it consumes only 9 racks, is fault tolerant, based on SPARC (Niagara), got 25% more throughput, gets 16 times better response times, and obviously uses a hell of a lot less power to boot.
I had a conversation with the PAE guys there and got a lot of great details on the configuration and how they made it work. Here are some highlights...
So the Sun system that beat out the 595 was based on T5440 (UltraSPARC T2) systems connected to the new F5100 Flash Array. In order to make all this work in a fault tolerant way COMSTAR was used and throughout the process required absolutely no modification! Apparently the biggest "problem" they ran into some some minor tweeking in the mpt and sd drivers because they weren't designed to hand the extreme number of IOPS coming from the flash arrays. More shockingly, when they got the TPC-C number that beat IBM the CPU's were 50% idle! And, if you can believe it, during the whole time Sun was working on this benchmark of all the flash modules involved, only a single one failed! Just one!
The F5100 was illuded to a couple months ago by Andy Bechtolsheim; a 1U storage array filled to the brim with Sun SO-DIMM form factor Flash Modules. It can be ordered with as little as 20 modules for 480GB raw or as large as 80 modules for almost 2TB raw capacity. Sequential Write performance on the 80 module unit is rated at 9.7 GB/sec. It physically connects via SAS.
The F20 PCIe Flash Card is just a smaller version. Up to 98GB of Flash rated at 501 MB/s Seq Write. All the goodness of high performance flash storage but you just drop it into a PCIe slot and go. A fantastic solution for databases in need of fast logging capabilities, just plug it into a PCIe slot and define it as your new log device.
Jonathan
Guess who wasn't present. Jonathan is nowhere to be found. In fact, I haven't seen him since JavaOne. Since this deal has occured Jonathan has been pushed to the back seat while Scott has insisted on driving. The question is why?
I'm very curious how history will record things with all the details filled in. Did Jonathan sell us down the river? Or, perhaps, Scott's been driving things far longer than we realize and Jonathan has been something of a pawn in the latter days of the company. Its clear that it was Jonathan's management of the company that delivered us to the point acquisition was required, but we can't forget that he did do a number of good things, even if they didn't actually benefit the company in return.
I'm not going to make a judgment call just yet... but I'm starting to almost feel like Jonathan got screwed here more than we realize. Never the less, he'll have his millions of dollars to console him while the rest of us are left holding a fist full of memories and broken dreams.
published on 2009-10-11 18:47:00 in the "cuddletech" category
Oracle OpenWorld is starting right now (Sunday) in San Francisco. Shockingly I applied for a press/blogger pass and got one! So i'm heading there tonight for the big opening keynote starring Larry Ellison and Scott McNealy. But if you can't make it, watch it! Oracle OpenWorld is streaming live!.
Stay tuned for updates tonight following the keynote.
published on 2009-10-07 17:14:00 in the "cuddletech" category
In an effort to reach out to "new media", HP had a tech-day event at the Cupertino Campus for a variety of invited bloggers. For some reason I was among them, because inviting a Sun zealot to an HP-UX love-fest is always a recipe for a good time. I wore a Sun Microsystems shirt, of course, just to keep things clear. Some of the other attendees included David Adams of OS News, Saurabh Dubey of ActiveWin, and Andy McCaskey of SDR News.
To make a long story short, the event celebrated the 10 Year Anniversary of SuperDome and the 25 Year Anniversary of HP-UX... both, they say, are still on top and going strong.
For my non-enterprise readers you may or may not have heard or know much about HP SuperDome, so here's the scoop. SuperDome is HP's "High End" for the "Integrity" line of servers, which can utilize either Intel Itanium or PA RISC, although the latter has gone the way of the dodo. While competitive offerings, even from Sun, have come and gone during the 10 Years of SuperDome I readily admit that it remains a viable platform due to the simplicity and modularity of its architecture of of that HP should be, and is, very proud. So proud in fact that the model they showed off to us was a prototype unit that is still in use to day, simply upgraded to newer processors and used for performance testing and analysis.
SuperDome's architecture is simple. Power at the bottom, cooling at the top with additional cooling elements in the middle. The core of the system is a powerful backplane which connects "cell boards" to each other and to IO expansion. Nothing fancy, just big ass cables from point A to B. Up to 2 SuperDome chassis (or "frames" depending on which audience you address) can be melded together by wiring the backplanes together which is why you see some single cabnet SuperDome's and some that are 2 side-by-side joined at the hip, as it were. Each "Cell Board" contains memory slots and CPU sockets. At present they're offering Quad-Core Itanium, and each chassis supports up to 8 Cell Boards, so you get up to 32 sockets (128 cores) per chassis, but as I said you can cable 2 chassis together increasing the total "single" system capability to 64 sockets.
One of my big nit picks was that when HP SuperDome first arrived it was direct competition to the Sun "StarFire" Enterprise 10,000 ("E10K") which had double the capabilities of the HP! Hardly competition. But time has proven HP out. While the E10K was an excellent HPC Super Computer (in its day) for very large SMP requirements, the SuperDome instead focused on being carved up using partitioning. This is a two fold strategy, first using "nPars" to "hard partition" cell boards into individual systems and then to use the HP-UX vPars to "virtualize" instances of HP-UX within those nPars. The result was a platform that was truly about flexibility and centralization as opposed to being the biggest and baddest gun in the west. Lets not forget that both E10K and E15K both had, I think, the best partitioning technology available, but the markets they appealed to were very different. When you combine SuperDome's simplistic architecture along with its marketing focus of consolidation and flexibility you get a solution with far more staying power.
Of course, the trouble in my mind is that HP-UX is a steaming pile of crap.... but they assure me this is not so. As though little had changed in the last 10 years, HP still seem to think of the real competition being IBM mainframe installations. They hardly acknowledge Sun as competition because there is so little focus by Sun in that competitive space. So the OS war they are fighting isn't HP-UX vs AIX vs Solaris, but rather HP-UX vs IBM z/OS.
HP-UX 11i v3 Update 5 (and you thought Solaris naming was stupid!) just released. The roadmap current charts out to HP-UX 11i v5... so if your wanting for HP-UX 12 you'll wait longer than for Solaris 11.
I had several conversations with important persons of interest regarding Solaris and the jist was generally the same as the market at large. They find DTrace to be annoying, because while people need it they don't use it. Furthermore they claim that DTrace like facilities have existed within the HP-UX kernel for years but they never considered it of interest to end customers and therefore appeared to come late the the party. With techonologies like SMF they agreed that the end-user excitement wasn't as great as they expected, and hands down everyone saw ZFS as the really big ticket win in Solaris.
But.... HP-UX vs Solaris is little more than a side-note. In the enterprise space in which they see themselves as relevant OS wars don't exist. CFO's, they think, make the decisions and its not about whether Solaris is better than HP-UX or AIX but whether the hardware provides value and the supplier offers excellent service... the OS is, in that case, simply a glue layer. At least, that's the vibe they put off.
And whats more, I consider Itanium to have run its course... but HP are quite confident that its still the best processor available and are even minorly annoyed at the idea of putting Intel Xeon processors in the SuperDome cell boards. In short, not happening.
The event was fun, I was glad to attend. It was a throwback to the world pre-X86, when servers were big and RAS was the standard. I congratulate HP on producing a fantastic product with so much staying power and wish them continued success in the future.
published on 2009-09-27 00:33:00 in the "cuddletech" category
For some reason I love "work clothes", in particular construction trade clothing. I suppose it comes from wearing a kilt for so long. When I first started wearing kilts on a daily basis, years ago, I did so because a) you stick out in a crowd, b) they are super comfortable, and c) they just look kool. Kilts are the ultimate "man" clothing. Thanks to the Utilikilt I also found kilts to be far more versatile and functional than traditional pants (meaning Levi jeans or slacks, and the like).
I found this particularly to be the case when I became a father. In the hospital one thing you can't buy is the attention of the nursing staff... no doctor or nurse forgot which room "that guy in the kilt" had. If Tamarah needed ice, we got ice, when she wanted to talk to someone they came pronto. But then as we were out on the town and Nova was 6 months, I needed to carry a baby, daipers, food, a bottle, in addition to keys, wallet, phone, etc. With my Utilikilts I could carry everything and still have absolute flexibility to do all the aerobics that parents do with young kids, and stay cool at the same time... no bags required.
For some reason this blossomed into a general love of work clothes. For instance, Carhartt overalls are just awesome:
The same goes for German Lederhosen, not that cheap crap you dress kids in or wear to the pub but proper leather ("leder") britches. So sad the bad wrap lederhosen have gotten; any male garment with a built in chest strap was intended for serious action, with an axe or chainsaw.
But perhaps my greatest respect if for Japanesse work clothes. The pinnacle of form and function. If you picture Japanesse men as little frail guys, you've not seen the real Japan:
Have a look at the TOBI Catalog or my favorite store in Tokyo, Mannen Ya (Manly Man). Nikka pants are the best.... I'm wearing a pair now:
Nikka Zubon pants are for construction workers in high places. There are lots of variations for different professions in Japan and lots of reasons to wear them, but they have a lot of advantages. First, they're comfortable and give you completely unrestricted movement, secondly the pockets are deep and plentiful, no spilled change with these, and the "poof" in the material adds to your sensory experience by being more aware of objects around you in tight spaces or wind direction. Shockingly, they don't snag on things like you'd think and they are made of a really strong material... these are, after all, work pants.
Is there a point to this post? No, not at all, but some times its fun to branch out from the usual and perhaps as a guy that sits in a chair for a living I have an appreciation of things that are otherwise overlooked and under appreciated. Working men of the world, I salute you.
