Manage Your Child Labor with Ruby

There’s a fine Unix tradition of “fork servers” — servers that create new child processes and farm work out to them. Ruby can do this too, of course. A forked process is generally referred to as a “child process,” and you’re forking them to do work.

“Child Labor,” and other things we don’t usually say about fork servers.

Naturally, I’ll tell you how to do it! By the end of the article, perhaps you’ll have thought about when you want to.

Basic Mechanics

Ruby has a “fork” call and it’s straightforward to call it with a block:

puts "Before fork!"
fork do
  puts "This is printed by the child process!"
  exit!  # Always call exit! when finishing a child process in Ruby
end
puts "This is only printed by the parent process!"

This is nice and simple. Calling fork! spawns a child, and the child can do work (or not — this one just prints and exits).

The “exit!” call above is to make sure that any at_exit handlers are only called by the parent, not the child — you don’t care here, but you should still get in the habit. When calling fork, make sure the child finishes with at_exit! Your unit tests will thank you, eventually.

In this case, I’ll assume you have a fair number of child processes working. One of our processes has almost 30 workers for one of its tasks, for instance.

child_mill_worker
Photo credit: Library of Congress, LC-DIG-nclc-01379, Lewis Wickes Hine, 1918

Doing Work

Sometimes you just fork and already know what to do. For instance, in one OnLive server we fork a child process for each availability zone we connect to — fourteen zones, fourteen children that each connect to a zone-specific server. Easy peasy.

But sometimes you want the parent to handle incoming requests — like Unicorn, a fork server that handles HTTP requests. When that happens, you usually want to open a unix pipe between the parent and child. The child can handle its current request, then read the next one from the pipe. The parent can accept incoming requests and then tell each child, via one pipe per child, to handle them.

Full details of using Unix pipes are beyond the scope of this article, but rest assured — they work fine in Ruby, just like in C.

Handling Shutdown

Of course, it’s a little hard to shut down a process with a lot of children. You don’t want to have to kill every process. You can kill the whole process group (kill dash nine, baby!)… But it would also be nice if it behaved properly when you send a SIGTERM. Here’s one way to do that in Ruby:

  Signal.trap("TERM") do
    child_processes.each { |process| `kill -TERM #{process[:pid]}` }
    shutdown()
  end

Basically, if you get a SIGTERM (a “terminate process” request), pass it on to all your children and shut down.

child_angel
Photo by Mike Krzeszak on Flickr

Dead and Zombie Children

Sometimes, processes shut down for more-or-less random reasons. Your script may not be perfect. Your network connectivity might not be perfect. Somebody else might have hogged all the RAM on the machine.

In a real-world scenario, you should assume that children can die.

When they do, you may want to restart them.

To do that, watch for SIGCHLD:

  pid, status = Process.waitpid2 0, Process::WNOHANG  # Did any child die?

This little incantation should be called from the parent process and it checks: has any child died? You can remove the Process::WNOHANG parameter if you want to wait around until a child dies. You can switch the “0″ to a specific child’s pid if you want to wait for one worker in particular.

When a child dies, you probably want to restart it. Or maybe shut everything down and exit, like SIGTERM above. It’s up to you!

Note that if you don’t watch for SIGCHLD (or detach the process), you’ll get zombie processes. Those are processes that finished already but are just waiting to return their status to a parent.

Zombie child processes!

child_zombies
Photo by Ben Schumin, at the Silver Spring Zombie Walk; SchuminWeb on Flickr

Optimization

Fork benefits greatly from sharing memory — most of a parent’s state can be shared with all children, as long as it doesn’t change. So make sure to preload anything that most children will need.

But garbage collection can move things around, spoiling the benefits. You can improve the situation by using Ruby 2.0 or later, which has more copy-on-write-friendly garbage collection. Or you can just kill each child every twenty-or-so requests and let the parent restart it — then it will restart with a nice clean copy of the current parent state.

This only gets you better memory usage, though, so only do it if you need to optimize that.

Memory Use

If you’re worried about runaway memory use, you’ll want to limit memory explicitly. And if you’re using ActiveRecord or anything similar, you should be worried about runaway memory use.

Luckily, you can limit the memory use in the child process right after fork:

  Process.setrlimit :RSS, 10_000_000  # This sets resident set maximum to 10MB

You can separately limit the stack, the heap (also known as “sbrk” for these purposes) and the resident set — limit any or all of them in any child, the parent or both.

Why Not Use Another Process Manager?

This seems like a lot of stuff. And if it’s process management, can’t you use an existing process manager?

Some do exist: daemontools, God, BluePill and Monit, for instance.

Unfortunately, God, Bluepill and Monit require being root — most fork servers shouldn’t run as root, just as Unicorn doesn’t. Daemontools wants a directory per process set up and it’s sometimes hard to spin up new ones dynamically — which can be fine or not-fine, depending.

I think it wound be wonderful to have a library for this purpose. But the existing contenders are clearly designed with a different use case in mind.

Again, the case of Unicorn is instructive. You could imagine it running with one of these as a manager, but instead they wrote their own process management. So did Passenger. I co-wrote something similar for integration testing.

Some day we may have a great library for this, but that day isn’t here yet.

glass_factory
Photo Credit: Library of Congress, LC-DIG-nclc-01151

End Child Labor?

This article has a lot of small, tactical recommendations. If I were to give you a final strategic recommendation, it would be this:

Fork servers are great when you’re looking at cases where processes can fail. Prototyping is a case Ruby is good for, and often failure comes with the territory. Learn to write a fork server in Ruby, and especially learn all the ways it can fail and how to handle them!

So what do you think?

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: