Mirrored from My Hidden Room.

This weekend I got a hankering to make one of my favorite dishes, Puerco Pibil.  It’s a delicious slow roasted pork dish from the Yucatan Peninsula.  It features tangy flavors, and moderate heat from the habenero pepper.  My recipe is pretty much the one that Robert Rodriguez outlines in his entertaining 10 Minute Cooking School video.  This weekend, I made a big batch, about 15 lbs of it.  It did this because the meat vendor I use, Cash’n'Carry, sells the pork butt in ~15 lb bags.  $1.48/lb was a sweet deal.

I get all my spices from World Spice, located behind the Pike Place Market.  They’re inexpensive and delicious, and can be shipped.

My recipe, as made this weekend:

• 15 lbs pork butt, cubed into ~2″ cubes (roughly, precision not important)
• Banana leaves (asian grocery! cheap!)
• 1/4 cup minced, de-veined, de-seeded habenero peppers (~16 habeneros)
• 1/2 cup coarsely chopped garlic
• 1 cup orange juice, freshly squeezed
• 2 cups lemon juice, freshly squeezed
• 1.5 cups white vinegar
• 3/8 cups delicious reposado tequila
• 8 oz annatto seeds, ground finely (measured as whole annatto)
• 1 oz cumin
• 1.5 oz black peppercorns
• 24 allspice berries
• 1.5 tsp cloves

Set aside the banana leaves and cubed pork.

Take the rest of the ingredients.  Grind all the spices into dust, and combine with the garlic, habenero, juices, and vinegar.  This is your annatto paste, and is what the pork will marinate in.  In a large bowl, combine the pork with the annatto paste, mixing thoroughly.   I have a large (6 qt) food bucket that I use for marinating the pork.  Put the pork into a large container with a lid.  Put this into the refrigerator for at least four hours.  I try to marinate it overnight.

On cooking day, pre-heat the oven to 325F.  Line a large pan with banana leaves, leaving enough overhanging banana leaf to allow you to fold it over.  Fill the cavity with pork mix, making sure to put all the delicious marinade into the pan.  Fold the banana leaves over this, and cover with more banana leaves, tightly packing it.  Then cover with aluminum foil.  Or not, but I do.

Bake at 325F for four hours.  You will know it’s nearly done when your house smells delicious.

Enjoy on rice, or soft corn tortillas, or as tamale fill.  There are tons of ways to enjoy this versatile dish.

Mirrored from My Hidden Room.

So at work, I had a blog.  Not my blog, of course.  One I support.  So this blog was thrown together quickly to facilitate business goals.  Like you do.  And that’s great.  We met the deadline.  We got the product functional.  But performance kinda sucked.  A little backstory.  Here’s how it got set up to start.

We love virtualization here.  Everyone should.  It’s a fantastic way to take adventage of hardware that you’ve got laying around that’s being idle.  Idle hardware is lame. So we run KVM.  This means we get to manage machines more effectively, and can provision things faster.  That’s awesome.

So this blog.  It’s a cluster of boxes, made up of a pair of webservers, and a pair of DB backends.  The DB backends are a master, and a backup.  This is a small project, so these got set up with just 4GB of ram.  So that’s fine.  The web servers are each a VM with 1 vcpu, 4GB of ram, and some disk space.

So we set that all up, and it did fine.  But not great.  here’s what it did.

Incidentally, if you’ve got data you’ve collected for performance with Pylot, and need to CSV the averages to make a pretty graph, here’s my way of extracting the averages from the results HTML to feed into gnuplot.

for file in load-dir/*/results.html ; do cat $file  | egrep 'agents|avg' | head -3 | perl -e 'while(<>){s/<\/?t[dr]>//g; s/(\d+\.?\d*)/\t$1/g; chomp; @p=split /\t/,$_; push @r,$p[1]; }; printf "%s\n", join ",",@r; '; done


So what we can see then from these graphs is that a pair of single core boxes, tuned with the default cherokee+php installation don’t do all that great.  They can handle 1-2 simultaneous requests, but past that, the response time gets pretty bad. That’s where the project got left for a while, until the other day I got the request to make it handle 500 requests per second.  “Wow, shit.” I thought.  So I dove in to see what I could do to improve performance on the blog, and it turned out that there was a lot I could do.

1. Single-core boxes don’t have a lot of performance available to them.
2. Cherokee uses PHP in FastCGI mode, and does a terrible job of tuning it out of the box.  It defaults to a single PHP thread.
3. WordPress is very hungry for CPU time.  It chews up a lot of resources doing its DB queries and wrangling them.

To address these points, I did the following.

The VMs themselves were restarted with more CPU cores — four cores per box.   This allowed me to dive into discovering that Cherokee wasn’t tuned well.  Under a single core, I saw 80% cpu utilization on PHP, with high system wait time.  That sucked.   But after I bumped up the core count, it still looked bad.  Still only one CPU @ 80%. WTF.  So then I turned to Cherokee, and I tuned PHP so that it would invoke 16 children, enough to handle the request volume.  This helped a lot, but there was still room to do better.  So I added APC, the Alternative PHP Cache, to the configuration.  This helped out a lot.  I then looked at wordpress specific caching engines, and settled on using W3-Total-Cache to help out.  This brought performance into the range of fulfilling the customer’s request.  I felt great about it.

I used pylot to graph performance at various points through this project so I could figure out how I was doing a better or worse job of tuning the boxes.

Here’s the performance data showing the improvements that caching at various layers added to the party:

So it turns out that these are some great ways to make sure your blog performs:

• Make sure that there’s enough hardware supporting it
• Cache in depth and breadth, early and often
• Tune your PHP/WebServer to handle the project
• Employ performance testing to measure your real improvements

With some hard work and performance tuning, you can turn an abysmal 7 requests/sec peak performing blog into one that can sustain 350 requests per second, and do it in a tiny fraction of the time.

-Gabriel

Mirrored from My Hidden Room.

This guide leans heavily on the work of http://simplygenius.com/2010/08/jconsole-via-socks-ssh-tunnel.html — what I’ve done is collect his work into something a little more manageable for my environment.  In here, I will go over the shell commands that can be used to make doing this easy.

Read the rest of this entry »

Mirrored from My Hidden Room.

Turns out this is hard to do.    I’m writing this here for my benefit, and everyone else’s, too.  I’ve got a client using Cassandra, and of course you monitor that stuff.  So I figure that using the cassandra-cacti-m6 stuff is a good plan.  That’s cool, it works with Cacti.  Cacti’s pretty snazzy.   My customer is using Centos 5.5 on the monitoring box.  Turns out there are a lot of hoops to jump through for that to work.  So here’s what I did.

1. Install jpackage-utils
2. Install the jpackage.repo into /etc/yum.repos.d, and enable the rhel5 targets
3. Install JDK6 Update 3 (because of the following step)
4. Install java-1.6.0-sun-compat from ftp://jpackage.hmdc.harvard.edu/JPackage/5.0/generic/RPMS.non-free/

Then I go to install ANT, but I discover something — trying to install ant.noarch complains about “Missing Dependency: /usr/bin/rebuild-security-providers is needed by package”.  This sucks.  So I do some googling, and figure out that someone solved this issue on Centos 5.x.  I do what they did (documented at http://plone.lucidsolutions.co.nz/linux/centos/jpackage-jpackage-utils-compatibility-for-centos-5.x ).  This works great.  The package is built, I install it out of the local build dir.

Then I install ant.noarch and ant-nodeps.noarch.  These do the trick.

The I build cassandra-cacti-m6 as documented in its source tree.  Woo.

It took me a while to dig up and get all this working, but for you, I hope it’s fast and easy.  Enjoy!

-Gabriel

Mirrored from My Hidden Room.

Tomorrow is my first day with the new gig.  I resigned from my old job about ten days, and had a nice break.  But now it’s time to get back to things.  I’m a little apprehensive about this whole thing, as this is a new way of working for me.  Changing gears in so many ways after my long time at PopCap (4.5 years – Wow) is filling me with all kinds of feelings of volatility.  What happens now?

I’m eager to get into the swing of things, and past the anxious new beginning with this.   Until tomorrow.

Mirrored from My Hidden Room.

Thanks to flickr user redjar for the image.

I hate airports.  Not flying, flying I love.  But the process of dealing with airlines, and luggage, and most especially the TSA/Security Theater aspect of the process drives me crazy.  I flew down to SJC this week for a technical conference, MySQLConf.  It was a great conference: I met neat people, I learned new things, I got excited about tech again.  I loved that part.  The flight down was fine.  Easy, no problem through the airport.  Only the usual anxiety about the process.

Read the rest of this entry »

Originally published at My Hidden Room. You can comment here or there.

Today we had to do a very large alter table on some production data.  The table in question is big. 47 million rows big.  We have to make some changes to drop a column and change some indexing to support our application.  Fine, ALTER TABLE blah blah blah.  But how do we keep an eye on the progress of the alter?

There’s nothing in the documentation for MySQL that speaks to a method, nor does any google searching pull up anything (except for the one InnoDB file per table patch.)

So what’s a clever systems administrator to do?

I looked at the output of lots of status commands from MySQL, and finally I came across something useful in the InnoDB engine status output.  I noticed that there was an entry in the TRANSACTION section that speaks to the undo log entries.  I noticed that after several hours of our ALTER TABLE running that the undo log entries value was up around 33 million.  And then it occurred to me that since InnoDB does row-level operations, that this value might correspond to the number of rows that have been operated on throughout the process of the ALTER:

---TRANSACTION 0 2769988559, ACTIVE 53210 sec, process no 4115, OS thread id 1157654880 inserting, thread declared inside InnoDB 91
mysql tables in use 2, locked 2
111667 lock struct(s), heap size 13580272, undo log entries 33707735
MySQL thread id 948913367, query id 1662722204 localhost root copy to tmp table


So I wrote some bash.  And some awk.  And some perl, and I made a hacky little one-liner script to dump out the rate of change, the average rate of change, the % complete, and an estimate (based on average rate of change) as to when the query might be complete.

That script is below, and you are welcome to it.  I have yet to wrap it in more clever perl to dump out an actual status bar, but I’m happy with it, and hopeful that this is a meaningful way of keeping an eye on the rate of change, and the progress of ALTER statements.

The script:

#!/bin/bash
(while(true); do  \
(mysql -e 'show innodb status \G' | grep undo\ log\ entries ; sleep 1 ;  \
mysql -e 'show innodb status \G' | grep undo\ log\ entries ) |    \
egrep '[0-9][0-9][0-9][0-9]‘ |awk ‘{print $10;}’ ; done ) | \
perl -e ‘$t = ROWS_IN_TABLE; while(1) { \
$n ++; $nn; $a = <>; $b = <>; $nn += ($b-$a); \
printf “Rate: %d, avg: %d, %0.3f%% complete, done in %d sec\n”, \
$b-$a, $nn/$n, ($b/$t)*100, ($t-$b)/($nn/$n); }’;


I hope you find this tool useful.

It took a LONG time for the query we performed to run, and it was very helpful to have this to gauge the time it’d take.   The moving average is not the best for long-term accuracy; restart the script periodically for better views (and wait for ~100+ trips through the loop.)

It was within three minutes of accurate.  It look a little longer than the script indicated (log undo entries was greater than the # of rows, but not by more than a small percentage (less than 0.01%.)  The total query time was 22 hours, 9 minutes, and 33 seconds.  So it was off by only 1 hundredth of a percent.

This is only valid for ALTER TABLE, but appears VERY useful.

WOOHOO!

Originally published at My Hidden Room. You can comment here or there.

So in June, I’m having my first art opening.  That’s exciting!   Over the last half-year I’ve been talking to the folks at Utilikilts about whether they’d be interested in seeing my works in their store for Art Walk.  They expressed interest that yes, indeed, they would like that.  So we talked, and figured things out, and now I’ve got an opening coming up.

This collection of works ranges over a few areas of interest for me: Burner Culture, People, and some other pieces.   Please come see my work next month, on June 4th for Art Walk at the Pioneer Square Utilikilts store.  There will be a live DJ and drinks to be had.

It’s very exciting, and I’m looking forward to it.  I hope to see everyone there.

Originally published at My Hidden Room. You can comment here or there.

This past weekend was Norwescon, a sci-fi/fantasy convention.  It was awesome.   I put on a shoot for free out of my room, and I got a good turnout, and I had a lot of fun.  It was a good experience, and good practice.  Next year, I know some things I’ll do better.

I got a lot of good feedback, and pleased participants.  Woohoo.

I think it’s important to do things like this to give back to the community — to let people come if they want to have a photo made, and share in a creative process.  The best photos were those when the subject played along, and had ideas about what to do.  I like it when it’s a collaborative project.

Anyway, enjoy.  Click the image for the flickr.

-Gabriel

Originally published at My Hidden Room. You can comment here or there.

It’s been a long time since I’ve updated this thing. Lots of changes, some photos. Good times.

Christmas has come and gone, thank goodness. The holidays aren’t for me. Winter is over, and I’m glad. But it’s been too long without updating. Without creating. Without feeling good about things. It’s time for that to stop, and for things to get back on track.

So, I’ve got some things cooking that’ll be good. I have a strong lead on a show in the summer (more to follow later), some leads on good ideas/concepts to push myself, and I’m biking to work now. 11 miles in the morning is good fun.

But where to go from here, that’s the thing. I have ideas, but to make them real…