Practicing Ruby

USP: Handling excessive CPU Usage within a Process

normalperson@yhbt.net (Eric Wong) — Tue, 27 Mar 2012 01:11:00 -0400

If you’ve ever had issues with a runaway process eating up CPU time and want to kill it automatically, it’s possible to use Process.{get,set}rlimit and a signal handler for SIGXCPU to accomplish this.

The following snippet of code implements an example of a CPU limiter. This self-contained example aborts itself once CPU utilization reaches 50%.

# Users of older Rubies may want to use Process::RLIMIT_CPU
# instead of :CPU in this code.

soft, hard = Process.getrlimit(:CPU)
last_update = Time.now
max_cpu_time = 5

# this signal handler runs once we've hit max_cpu_time:
trap(:XCPU) do

 # Calculate CPU utilization based on elapsed (wall clock) time
 # and max_cpu_time
 elapsed = Time.now - last_update
 cpu_usage = max_cpu_time / elapsed
 puts "CPU utilization: #{"%0.2f" % (cpu_usage * 100)}"
 if cpu_usage > 0.5 # 50%
   abort "CPU utilization exceeded 50%"
 end

 # If we're below our CPU usage threshold, install a new limit based on
 # current Process.times
 last_update = Time.now
 t = Process.times
 Process.setrlimit(:CPU, max_cpu_time + t.utime.round + t.stime.round, hard)

 # We now return to our regularly scheduled looping ...
end


# Install an initial limit for max_cpu_time using existing process times
t = Process.times
Process.setrlimit(:CPU, max_cpu_time + t.utime.round + t.stime.round, hard)

# The signal handler registered for :XCPU will run automatically
# after the CPU time hits max_cpu_time seconds
loop do
 # do something to use the CPU...
 100000.times { 1 + 1 }

 # you can change the amount of time slept to influence CPU utilization
 sleep 0.01
end

I learned this example a few years ago from reading the PulseAudio source code:

       $ git clone git://git.0pointer.de/pulseaudio
       $ $EDITOR src/daemon/cpulimit.c

GPLv3 does not cover this message, which is CC0: To the extent possible under law, Eric Wong has waived all copyright and related or neighboring rights to the contents of this message.

USP: Implementing signal handlers - some caveats

normalperson@yhbt.net (Eric Wong) — Tue, 27 Mar 2012 00:25:00 -0400

Signal handlers may run at any time

If your program receives a signal, Ruby will invoke the associated signal handler as soon as it is able to. This means a signal handler can hijack your existing code flow just about anywhere in your program (including inside methods of any libraries you use). Normal code execution resumes once the signal handler finishes execution.

This is a big difference from most event-driven programming frameworks where callbacks for a given object/event fire synchronously and will not step over existing code flow.

Reentrancy vs thread-safety

While reentrancy and thread-safety are related concepts, it is absolutely critical to understand is they are NOT the same, and one does not imply the other.

(There are existing articles on this, so I won’t dive into this more.)

Signal handlers must be reentrant

When writing a signal handler, you must ask yourself:

What happens when another signal arrives while this handler is still running?

Since signal handlers are invoked as soon as possible, they can even run while another (or the same) signal handler is running. Thus signal handlers must be able to tolerate reentrancy.

Back to thread-safety: some constructs required for thread-safety fail horribly when used in situations that require reentrancy. Most mutex implementations (including the Ruby Mutex class) deadlock when used inside a signal handler.

Consider the following snippet:

       lock = Mutex.new

       # XXX this is an example of what NOT to do inside a signal handler:
       trap(:USR1) do
         lock.synchronize do
           # if a second SIGUSR1 arrives here, this block of code
           # will fire again.   Attempting Mutex#synchronize twice
           # the same thread leads to a deadlock error
         end
       end

Thus, you must ensure any code you use inside a signal handler is reentrant-safe. Even using the Logger class in the Ruby standard library (which can call Mutex#synchronize) can deadlock inside a signal handler.

Signal reliability

Signals are not completely reliable in Ruby (nor many applications, for that matter). If multiple, identical signals are received in a short time frame, you’re guaranteed to fire a handler for /at least/ one of the signals, but not all of the signals received.

This is because Ruby implementations must¹ block signals from firing while manipulating internal data structures. When normal signals get blocked, they do not queue up in the OS kernel and instead only get a boolean bit set.

¹ – back to reentrancy, temporarily blocking signals to ensure reentrant-safe data manipulation is analogous to using a mutex lock to accomplish thread-safe data manipulation.

Untrappable, unblockable signals

Regardless of the Ruby runtime state, SIGSTOP still suspend a process immediately (until SIGCONT is received), and cannot be trapped by the Ruby runtime (or any userspace process).

Similarly, SIGKILL terminates a process immediately. Processes are are given no chance to stop or react to them. Thus, no blocks registered via Kernel#at_exit, END, nor object finalizers can run upon SIGKILL.

Sending “Ctrl-Z” from a terminal generates SIGTSTP, not SIGSTOP, and SIGTSTP is trappable.

Deferred signal handling

POSIX defines a very small number of C functions that are safe to use inside a signal handler². As Ruby programmers have little direct control over which C functions they end up calling, Ruby implementations (at least modern ones) implement deferred signal handling.

Thus Ruby implementations register trap signals using C functions (via sigaction(2)) and dispatch Ruby signal handlers out when the VM/interpreter is in a safe state.

Modern versions of Perl (and presumably other high-level languages) also use deferred signal handling.

Even with deferred signal handling implemented by the language runtime, it is still a good idea to also implement deferred signal handling in your Ruby applications to avoid the same reentrancy pitfalls.

² – Linux signal(7) manpage is one place that lists these functions

License: GPLv3 or later, http://www.gnu.org/licenses/gpl-3.0.txt

USP: Unix signals - primitive interprocess communication

normalperson@yhbt.net (Eric Wong) — Tue, 20 Mar 2012 21:56:00 -0400

Signals are the most basic way for processes to communicate with each other on a Unix system. Each process has signal handlers configured by default, so some actions (e.g. killing a process) are always available.

There are two distinct classes of signals defined by POSIX:

standard and real-time

Currently, Ruby implementations (that I’m aware of) only deal with standard signals. Ruby implementations also control the signal mask and will temporarily block/unblock signals to perform critical operations in the interest of (portable) thread/async-safety.

Operating systems have a pre-defined list of signals. This list of signals is available for all processes to send and receive. Each signal has a human-readable name (e.g. “KILL” or “SIGKILL”) and non-negative number (e.g. 9) associated with it. To see which ones are supported by both Ruby and your OS, the “Signal.list” method returns a Hash mapping their names to respective signal numbers.

Some signals (e.g. SEGV, VTALRM) may be reserved by the Ruby implementation for internal use. Other signals (STOP, KILL) may not be blocked, ignored, or handled at all by any userspace process.

Signals may be assigned (and reassigned) handler functions (aka “callbacks”) which execute upon delivery of a signal. Those of you familiar with event-driven programming models will realize signals are another type of event (not much unlike I/O) which you may define callbacks for.

However, writing signal handlers requires more care and attention than “normal” code in your application. The universal wisdom is to keep signal handlers simple and short, regardless of the programming language you’re writing in.

While Ruby helps users avoid many of the low-level details and caveats of implementing signal handling, there are still pitfalls to be aware of and avoid.

Methods for working with signals in Ruby

“Process.kill” maps to the kill(2) / killpg(2) system calls for sending signals to processes or process groups. “Signal.trap” (Kernel#trap) is analogous (but not directly mapped to sigaction(2) (or signal(2)).

USP: Unix processes and their attributes

normalperson@yhbt.net (Eric Wong) — Fri, 11 Nov 2011 22:37:00 -0500

Processes are kernel objects that run user space code. As we’ve established before, each Unix process is identified by an integer process identifer (PID) and has a file descriptor table.

PIDs

Each process knows two PIDs: its own and its parent’s (PPID).

Process.pid and $$ are wrappers for the getpid(2) syscall and Process.ppid is a wrapper for getppid(2). While the PID of a process never changes, its parent PID (PPID) may change.

$0 / $PROGRAM_NAME

Each process has a name, this variable may be reassigned to change the name of this process in the output in tools like ps(1).

Ruby does not expose $EXECUTABLE_NAME ($^X) like Perl does, however $PROGRAM_NAME is set to the executable name for Ruby programs at startup.

ARGV

This is an Array of command-line arguments given to a process. ARGV[0] in Ruby is the first argument given to the executable, not the executable name. Thus "ARGV[0]" in Ruby is "argv[1]" to C programmers.

Modifying this array will not affect the name of the process in tools like ps(1). Command-line option parsing libraries like ‘optparse’ and ‘getoptlong’ in the Ruby standard library may modify ARGV.

Signal mask / handlers

Ruby implementations have internal system-level signal handlers that may later dispatch Ruby code (registered via Signal.trap). Signal handlers written in Ruby are not subject to the limitations of system-level signal handlers.

ENV

This Hash-like object holds process environment variables. Environment variables are global to the process and are used to share information with system libraries and child processes in a language-independent manner.

Null-terminated C strings are mapped to Ruby Strings in both the keys and values of this Hash-like object.

While the system environment appears Hash-like, but it is not exposed as a hash to user space, so it does not have the same performance / algorithmic complexity as a Ruby Hash.

Working directory (Dir.pwd, Dir.chdir)

All processes have a working directory it runs in.

Process.times

Each process keeps track of CPU time spent in user space and the system (kernel) as well as the accumulated times of its children. Process.times is used to implement the Benchmark module in the Ruby standard library.

Usage counters (getrusage(2))

There is no Ruby API for getrusage(2), unfortunately, however some GC implementations may use it to provide profiling statistics.

Process.getrlimit / Process.setrlimit

These get and set process resource limits such as the maximum number of open file descriptors, resident set size, niceness, CPU time, etc. They are wrappers for the respective getrlimit(2) and setrlimit(2) syscalls.

User IDs / Group IDs

Each process has a real, effective and saved UID and GIDs.

Process::UID.change_privilege and Process::GID.change_privilege are the most portable and easiest ways to change the UID and GID of a process.

umask (File.umask)

The umask is the file mode creation mask of the process. This governs the filesystem permissions of newly-created files (File.open) and directories (Dir.mkdir).

(I don’t plan to write much about Unix credentials / permissions, if anybody else wants to contribute an article it would be greatly appreciated. It’s not a subject that’s ever interested me.

On the other hand, part of me dies whenever I see a directory tree where somebody ran “chmod -R 777 .” on because they didn’t understand Unix permissions…)

License: GPLv3 (or later, at the discretion of Eric Wong)

USP: IO#dup and the dup(2) system call

normalperson@yhbt.net (Eric Wong) — Wed, 19 Oct 2011 04:59:00 -0400

IO#dup vs. Object#dup

IO#dup is Object#dup in Ruby: it creates a shallow copy of an existing object. To create a shallow copy, the IO#initialize_copy callback method performs the dup(2) syscall on the underlying file descriptor the IO object wraps.

Like Object#dup in Ruby, dup(2) is a shallow clone that does not copy the underlying open file object in the kernel, but creates a new reference to an existing kernel object.

Thus, two (or more) file descriptors in the same process can refer to the same open file in the kernel.

Before calling IO#dup, we have a 1:1:1 relationship:

one Ruby IO object
one file descriptor
one open file object in the kernel

    [Ruby]    user space   |  kernel space
    ------------------------------------------------
                           |
    io_orig ----------- fd[orig] ----> file object
                           |
    ------------------------------------------------
    (file descriptors (fd) are the bridge here kernel and user space)

After we call IO#dup, we have two 2:2:1 relationship:

two Ruby IO objects
two file descriptors
one file object in the kernel

    [Ruby]    user space   |  kernel space
    ------------------------------------------------
                           |
    io_orig ----------- fd[orig] -\
                           |       >---> file object
    io_copy ----------- fd[copy] -/
                           |
    ------------------------------------------------

IO#dup can be called on the same IO object any number times, so there may be an N:N:1 relationship as long as the process (and system) resource limits are not exceeded.

Most kernel-level (but not user space) changes to one IO object are immediately visible in the IO object(s) it was copied from (or copied to).

Effect on IPC

IO#dup means IO#close / close(2) will only remove a reference to the file object in the kernel. Only when the last file descriptor for a given file object is closed is the actual file object closed and released in the kernel.

For applications relying on receiving an end-of-file condition (from a socket or pipe), IO#dup¹ can (sometimes inadverdantly) prevent the end-of-file condition from being reached in the reader.

¹ – and similar functions, like fork()

License: GPLv3 (or later, at the discretion of Eric Wong)

USP: Unix time and the Ruby Time class

normalperson@yhbt.net (Eric Wong) — Sat, 24 Sep 2011 02:28:00 -0400

The Unix epoch is defined as January 1, 1970, 00:00:00 UTC (Coordinated Universal Time¹). Unix systems return the time in seconds as a signed integer relative to the Unix epoch. Negative numbers are interpreted as dates before the Unix epoch.

Using seconds (and fractions of a second) to represent time allows application authors to rely on arithmetic rather than specialized functions for calculating time intervals.

File systems expose timestamps for all files in Unix time with varying degrees of accuracy and granularity. The Ruby File and File::Stat classes can support timestamps with sub-second granularity on some systems.

Modern kernels do not know nor care about timezones. Conversions from UTC to the local timezone (and vice versa) are done in user space². Different processes running concurrently on the same machine do not necessarily share the same local timezone.

Ruby Time class

The Ruby Time class includes methods for reading the system time and allows converting from an Integer (seconds since the epoch) to a Ruby Time object.

Ruby Time objects have a timezone attached to them, but remember this is a user space construct.

Time.now / gettimeofday(2) / clock_gettime(2)

gettimeofday(2) is the commonly available system call for reading the Unix time with (up to) microsecond resolution. On some systems, clock_gettime(2) may be used for nanosecond resolution³.

Time.now will return the current time as a Time object taking into account the timezone of the current process (in user space). Ruby 1.9 may use clock_gettime(2) internally if available, but fall back to
gettimeofday(2) to retrieve the system time.

Time#tv_sec / Time#tv_usec / Time#tv_nsec

These accessors allow access to the underlying values returned by gettimeofday(2) or clock_gettime(2) as Integers. For systems without nanosecond resolution, Time#tv_nsec may just be the value of Time#tv_usec multiplied by 1000.

Time.at – converts from Unix time to a Time object

If you have the number of seconds (and maybe microseconds) as integer seconds since the Unix epoch, you can convert that into a Ruby Time object using the Time.at class method:

       Time.at(seconds[, microseconds ])

This is is useful if you encounter a Unix timestamp anywhere but want to have a Ruby Time object. The local timezone is attached to the created object.

There is no underlying syscall needed for this, this conversion can be performed entirely in user space (although it may require memory allocation).

Process timezone

The “TZ” environment variable controls the timezone for a given process, and thus the timezone attached to a Ruby Time object. If “TZ” is unset, the timezone of the process is implementation-defined.

Time#utc

If your process is not running under UTC, you may wish to convert Time objects to UTC with this method.

Setting system time

Ruby does not provide methods to change the system time. Changing system time typically requires administrator (root) privileges and few applications (especially high-level ones) need to change the system time. For machines with network access, system time is commonly synchronized via NTP and gradually adjusted via adjtime(3) and/or non-portable syscalls.

Monotonic clocks

Some systems support a monotonic clock which is not adjustable (even by the administrator). A monotonic clock is useful for maintaining consistent timing and scheduling across system time changes.

Ruby currently (as of 1.9.3) does not provide access to the monotonic clock. However, Ruby implementations may use the monotonic clock internally for timing (via clock_gettime(2)) and recording time differences.

Avoiding system calls

You may notice the lack of gettimeofday(2) or clock_gettime(2) syscalls if you’re tracing syscalls. This is because C libraries can work with the kernel to optimize away the syscalls for gettimeofday(2) and clock_gettime(2). These implementations may trade accuracy for speed, but may be disabled (by the administrator) if accuracy is more important (than speed).

¹ One should know UTC is not exactly the same as Greenwich Mean Time (GMT)

² Timezone information is usually on (regular) files, so it may require syscalls to read those files.

³ You’re not guaranteed this level of accuracy is supported by your kernel nor your hardware. Consult your operating system and hardware documentation for more information.

License: GPLv3 (or later, at the discretion of Eric Wong)

USP: An introduction to the Unix pipe

normalperson@yhbt.net (Eric Wong) — Fri, 23 Sep 2011 00:35:00 -0400

The Unix pipe is the first example of an anonymous file we will cover.

Unlike regular files and directories, anonymous files do not exist on the filesystem and do not persist across reboots. Only the process that creates the anonymous file (and its descendants) can access anonymous files. Anonymous files only persist until the last process using it closes it.

At its heart, the Unix pipe is just like a pipe in plumbing, a one-way buffer for transporting payload from one end to another.

It is implemented as a circular ring buffer in the kernel:

                     user space  | kernel space
       -----------------------------------------------
                                 |
         data in --> fd[writer] ->- file[writer] -->\
                                 |                   \
                                 |                [buffer]
                                 |                   /
        data out <-- fd[reader] -<- file[reader] <--/
                                 |

The buffer is created by the pipe(2) system call and persists until both files are closed. In Ruby, the IO.pipe method calls pipe(2) and returns an array of two Ruby IO objects, a reader and writer:

       reader, writer = IO.pipe # kernel creates internal buffer

Writing can be accomplished by making the write(2) syscall, the writer IO object should default to IO#sync=true at creation.

       writer.write("HELLO")

Reading can be accomplished by making the read(2) syscall.

       reader.read(5)  # => "HELLO"

Finally, the kernel memory for the buffer is released after both file objects (through file descriptors and IO) are closed:

       writer.close
       reader.close
       # Kernel frees the buffer

As with regular files, pipes operate on byte streams and leaves higher-level functionality (like encoding) to user space.

Pipes also have important atomicity requirements standardized by POSIX. Given the importance of pipes in Unix programming, we will revisit and give pipes much more coverage in the future.

License: GPLv3 (or later, at the discretion of Eric Wong)

USP: Directories on the filesystem

normalperson@yhbt.net (Eric Wong) — Fri, 23 Sep 2011 00:21:00 -0400

Directories are containers for files, and directories are also files. Similarly, Ruby Hashes are containers for Objects, and Hashes are also Objects.

The contents of the “/home/johndoe/” directory could be expressed as the following Ruby pseudocode:

       root = {
         "home" => {
           "johndoe" => {
             ".bashrc" => Inode[123],
             ".vimrc" => Inode[456],
             ".mutt" => {
               ".muttrc" => Inode[789]
             }
           }
         }
       }

As one may guess, it’s impossible for two identical file names to appear in the same directory at the same time.

Directory Entries

As hashes are composed of key-value pairs, directories are composed of directory entries, or “dirent” structures which store file names and inode numbers. Like “struct stat” information, the “struct dirent” exposed to user space contains only the standardized, common information for user space, not the actual, private information in the kernel or file system.

Ruby does not expose the dirent structure in any way (the standard C library does), but one could express the dirent structure with the following Ruby Struct:

       Dirent = Struct.new(:d_name, :d_ino)
       Dirent.new(".bashrc", 123)

Reading Dirents

The Ruby Dir class handles reading directories, but only exposes names to the user (not inode numbers).

There are no portable syscalls for reading dirents from a directory, only standardized library functions such as readdir(3) and readdir_r(3). These library functions are wrappers for non-portable system calls.

Furthermore, common functionality like partial name matching (globbing) is handled by user space. As we learned before, proper Unix file systems have no notion of encodings/case-sensitivity in kernel space, thus inexact name matching rules need to be applied in user space.

Sorting is also done in user space. The standard C library functions never guarantee entries are returned in any particular order. This behavior matches the behavior of Hashes in Ruby 1.8 and earlier. In fact, some file systems store or index name-to-inode mappings in a hash structure on disk.

All dentries (except “..”) within one directory belong to the same device (and often the same physical device¹), so only the inode number is stored in the dirent.

Creating and Removing Directories

For regular files, we’ve covered the File.open, File.link, and File.unlink methods. Creating directories requires the mkdir(2) syscall provided by the Ruby Dir.mkdir method, not the open(2) syscall.

Once created, it /is/ possible to use open(2) (and thus File.open) to “open” a directory like a regular file and IO#stat it.

The rmdir(2) (not unlink(2)) syscall is used to remove directories and that is provided to Ruby by Dir.rmdir.

High-level wrappers like FileUtils.mkpath (or “mkdir(1) -p”) and FileUtils.rmtree (or “rm(1) -r”) need to invoke the appropriate syscall for every entry they wish to create or remove.

On modern Unicies, it is not possible to create hard links for directories. Allowing directory hard links would lead to infinite recursion loops when traversing directories.

Thus the File::Stat#nlink field for directories shows the number of entries the directory contains.

“.” and “..” entries

Every directory contains a entry for itself “.” and to its parent “..”. The root “/” directory is special and is its own parent, so “..” points to itself.

rename(2)

The File.rename method wraps the rename(2) syscall to change the name of a file (of any type) within the same device. rename(2) can occur within the same or different directories as long as the source and destination exist on the same device.

For regular files:

       File.rename("foo", "bar")

is roughly equivalent to:

       File.link("foo", "bar")
       File.unlink("foo")

However, it is impossible (on a POSIX-compliant filesystem) for userspace to access both “foo” and “bar” at the same time when using File.rename. Additionally, File.rename will work on directories whereas File.link and File.unlink do not work on directories.

Commands like mv(1) will use rename(2) if both the source and destination exist on the same device. This makes mv(1) very fast in cases where it works on the same device, but slow (and non-atomic) if the source and destination need to cross devices.

¹ There exist some union filesystems which break this assumption, however they are uncommon and often restricted to read-only access.

License: GPLv3 (or later, at the discretion of Eric Wong)

USP: Regular files and metadata

normalperson@yhbt.net (Eric Wong) — Tue, 20 Sep 2011 04:13:00 -0400

Regular files on a persistent filesystem are pointers / links to internal metadata known as inodes. The implementation details of inodes vary between various filesystems and is usually not the concern of user space. User space developers should only need to know a few common traits about inodes

device number (File::Stat#dev)

The device number is a unique integer identifier for the device an inode belongs to. Each file system partition is considered its own “device”, so a device number does not necessarily correspond to a physical device (e.g. hard disk drive)

This device number is unique within a running Unix system, no two “devices” may have the same device number at the same point in time.

inode number (File::Stat#ino)

The inode number is unique integer identifier for an inode within a particular device. Like PIDs, FDs, and device numbers, they are opaque pointers to complex objects in kernel space.

The combination of device and inode number is required to uniquely identify any inode (and thus file) in the system.

File Names

In many cases, there is a 1:1 relationship between file names and inodes. That is, each file name points to its own inode, much like how one variable points to one Ruby object:

“.bashrc” → inode_number=123, device_number=456

It is important to know the file name → inode relationship is one way. Inodes do not know which file name(s) it has pointing to it. Similarly, Ruby objects do not know which variable name(s) are assigned to it.

When working with File objects, one should know the File#path method return value in Ruby is determined in user space when File.open is called, not when File#path is called:

       file = File.open(".bashrc")
       file.path       # returns value of ".bashrc" determined above

Creating Inodes and Files

Making the open(2) system call with the O_CREAT flag will create a file.

File.open can accept the equivalent IO::CREAT constant or with fopen(3)-style mode strings (“w”, “a”) to create an inode if the specified file name does not exist.

The Unix touch(1) command and FileUtils.touch method both create files with open(2) in this way. Historically there is also a creat(2) system call for creating and opening files, but open(2) is more flexible and preferred. Ruby does not use nor expose creat(2) to the user.

Hard links

Since you should understand how two (or more) variables can point to the same object in Ruby, you should be able to understand how two (or more) file names can point to the same inode (and thus the same file) on one device.

Like two variables pointing to the same Ruby object, two (or more) links pointing to the same inode are indistinguishable from each other:

       "filename_1" ------+
                          |
       "filename_2" ------+----> inode_number=123, device_number=456
                          |
       "filename_3" ------+
                          |
                         ...
                          |
       "filename_X" ------+

Hard links to existing files are created with the File.link method which makes the link(2) system call. There is usually a system-dependent limit on the number of links to a particular inode, File.link will raise Errno::EMLINK if this limit is hit.

Number of Hard Links (File::Stat#nlink)

While inodes do not know which file name(s) point to it, inodes are aware of how many file names point to it. Each time a hard link is created, the number of hard links to the referenced given inode is incremented.

File names may be removed with the unlink(2) system call (via File.unlink in Ruby). Unlinking a file will decrement the number of links an inode has. It is possible for an inode to have zero links to it and remain valid (which we will surely discuss later).

File::Stat

File::Stat is the Ruby class to represent inode information to the user. All of the fields mentioned above (dev, ino, nlink) are available from a File::Stat object.

File.stat is a wrapper for the stat(2) system call and returns a File::Stat object for a given path.

If you have a Ruby IO object (and thus file) open, you can use IO#stat to get the File::Stat object for the file descriptor belonging to the IO/File object. IO#stat makes the fstat(2) system call instead of stat(2).

File::Stat is the Ruby equivalent/wrapper of the “struct stat” structure seen by C programmers.

We will discuss File.lstat when we discuss symbolic links.

License: GPLv3 (or later, at the discretion of Eric Wong)

USP: basic IO methods

normalperson@yhbt.net (Eric Wong) — Mon, 19 Sep 2011 21:28:00 -0400

(Consider C library functions (3) analogues while the system call (2)
functions are inevitably called by Ruby)

Opening (and Closing) Files

File.open – fopen(3) which wraps open(2) and provides a File object
IO#close – fclose(3), close(2) Copies Ruby buffers to the kernel, invalidates the IO object and releases the file descriptor so it may be reused by the kernel.

These methods will be covered in more detail as we dig deeper.

Some File Operations for Reading and Writing

IO#read – fread(3) – read all specified bytes
IO#write – fwrite(3) – write all specified bytes
IO#sync, IO#sync= – setvbuf(3) – controls Ruby (write) buffering
IO#flush – fflush(3) – copies existing Ruby (write) buffers to kernel
IO#syswrite – write(2) – copies application buffers to kernel
IO#sysread – read(2) – copies kernel buffers to application

(there are more methods which we’ll cover in time, of course)

Setting IO#sync= to true will make IO#write copy application buffers directly into kernel buffers, bypassing Ruby buffers. IO#sync does not influence read buffering.

If IO#sync is false, IO#write may copy small application buffers into Ruby buffers to before internally doing IO#flush.

The non-sys* methods will retry until the operation is complete (or raise on any errors). The sys* variants do not retry if the kernel returns a partial read(2) or write(2) and will also raise on errors.

Mixing sys* and non-sys* methods is not recommended and will result in errors or exceptions.

If you need more detailed descriptions, check `ri’ or other Ruby doc for Ruby methods. For C library functions and system calls, your Unix-like OS should provide manpages and you can search online, too. I have no intention of regurgitating API documentation.

License: GPLv3 (or later, at the discretion of Eric Wong)

USP: The Ruby IO Class

normalperson@yhbt.net (Eric Wong) — Sat, 17 Sep 2011 03:33:00 -0400

The IO class wraps file descriptors as Ruby objects and provides instance methods which wrap system calls. Each IO object wraps one OS file descriptor¹. IO also provides userspace buffering to avoid system calls and utility methods to make a programmer’s life easier.

C programmers may find the Ruby IO class analogous to the (opaque) “FILE” struct in stdio.

IO is often used via subclasses:

File class is mostly intended for regular files
TCPSocket for TCP sockets
UNIXSocket for UNIX domain sockets
UDPSocket for UDP sockets
…

All of those classes are based on the IO class and wrap an integer file descriptor.

Layers of Buffering

There are at least four distinct layers of buffering within a machine.

       Application buffers     - what your application sees
       Library buffers         - implemented by Ruby or libc
       ----------------- kernel-user space boundary -----------------
       Kernel software buffers
       Hardware buffers

Buffers may be implemented for both reading and writing.

The Ruby IO class can do the following:

accept application buffers from the user (IO#write)
return application buffers to the user (IO#read)
manage library buffers internally
copy (and remove) library buffers to kernel buffers
(hopefully) force kernel and hardware buffers to storage/network

We will cover how to use/influence/avoid them individually as we go further. For now, just know they exist and what IO does.

Scope and Sharing of Buffers

A wider view of the above table would be:

       process[0]  | process[1]  | process[2]  | ... process[N]
       ------------+-------------+-------------+---------------
       App buffers | App buffers | App buffers | ...
       Lib buffers | Lib buffers | Lib buffers | ...
       ------------- kernel-user space boundary ---------------
                       Kernel software buffers
                       Hardware buffers

User space buffers are not shared between different processes in Ruby.

Kernel buffers are shared, allowing read(2)-after-write(2) consistency between different processes. This is one (of many) ways for cooperating processes to share data.

A few applications² may manage process-shared buffers in user space, but (stock) Ruby does not have this.

Library buffers (like most memory) in the Ruby IO class /are/ shared between different Threads and Fibers in Ruby.

Sharing of application buffers between Threads/Fibers is possible but usually not a good idea. It is of course, application-dependent.

¹ Multiple IO objects may wrap a single file descriptor, but that is often a bad idea (to be explained later :)).

² database servers like PostgreSQL come to mind

License: GPLv3 (or later, at the discretion of Eric Wong)

USP: The Unix object model (as seen in procfs)

normalperson@yhbt.net (Eric Wong) — Sat, 17 Sep 2011 03:32:00 -0400

Eric Wong wrote:

If you’re a Rubyist, you may already be familiar with object-oriented designs. A running Unix/Unix-like kernel exposes two primary types of objects to userspace:

Processes
Open files

Now that we know what filesystems and virtual filesystems are, several systems have a procfs virtual filesystem mounted under /proc. I don’t believe it’s standardized anywhere between different systems, but your system probably supports it.

Among other things, procfs allows you to see the relationship between processes and their file descriptors on the filesystem.

You can inspect which file descriptors a particular process is using by looking in the “fd” subdirectory belonging to the PID belonging to the process. For example, my current mutt process has PID=19245 and file descriptors 0, 1, 2, and 4 open:

       /proc/19245/fd
       /proc/19245/fd/0
       /proc/19245/fd/1
       /proc/19245/fd/2
       /proc/19245/fd/4

Each PID has a directory in /proc/, and for each directory belonging to a particular PID, there is an “fd”

Utilities like ps(1) (are likely to) use information found in procfs to generate its output. Other information in procfs is less-consistent across different implementations, so consult your OS documentation.

License: GPLv3 (or later, at the discretion of Eric Wong)

USP: Avoiding system calls

normalperson@yhbt.net (Eric Wong) — Thu, 15 Sep 2011 00:34:00 -0400

Eric Wong wrote:

As mentioned before, syscalls are the interface user space interacts with kernel space. When a user space application makes a syscall, it is telling the kernel to execute code on its behalf.

The mode switch from user space to kernel space has more overhead and is slower than a normal library function call¹.

Thus user space can (and will often attempt to) aggregate several user space calls into fewer system calls to avoid this switching overhead. This is a common concept found in user space code and Ruby itself is no exception. This aggregation often does not happen transparently, so it should be understood and explained to avoid confusion.

I/O Buffering

As a Ruby programmer, you’ll notice the IO class (and subclasses like File) will buffer data you write and you need to call IO#flush or set “IO#sync = true” to ensure other processes can read it.

If you’re a C programmer, you’ll know the stdio library can do the same type of buffering in user space. In fact, MRI 1.8 used the stdio library internally for its user space buffering needs.

Kernel space may also implement its own buffering to avoid overhead when interacting with the storage and network layers. This buffering can be influenced in some cases from Ruby. We’ll cover this later.

Memory Allocation

While Ruby programmers do not often worry about memory allocation, sometimes the following question comes up:

Why did my Ruby process stay so big even after I’ve cleared all references to big objects? I’m /sure/ GC has run several times and freed my big objects and I’m not leaking memory.

A C programmer might ask the same question:

I free()-ed a lot of memory, why is my process still so big?

Memory allocation to user space from the kernel is cheaper in large chunks, thus user space avoids interaction with the kernel by doing more work itself.

User space libraries/runtimes implement a memory allocator (e.g.: malloc(3) in libc) which takes large chunks of kernel memory² and divides them up into smaller pieces for user space applications to use.

Thus, several user space memory allocations may occur before user space needs to ask the kernel for more memory. Thus if you got a large chunk of memory from the kernel and are only using a small part of that, that large chunk of memory remains allocated.

Releasing memory back to the kernel also has a cost. User space memory allocators may hold onto that memory (privately) in the hope it can be reused within the same process and not give it back to the kernel for use in other processes.

¹ Why this is expensive outside the scope of this list so explaining this is left as an exercise for the reader :)

² Via brk(2), sbrk(2), mmap(2) or various non-portable methods.

License: GPLv3 (or later, at the discretion of Eric Wong)

USP: System calls in Unix

normalperson@yhbt.net (Eric Wong) — Thu, 15 Sep 2011 00:34:00 -0400

As mentioned before, syscalls are the interface user space interacts with kernel space. When a user space application makes a syscall, it is telling the kernel to execute code on its behalf.

Ruby itself provides global “syscall” method on many platforms. It is useful for learning and experimentation, but not recommended for general use as it is fragile and non-portable. There is usually no need to use this method as many useful syscalls are already provided + wrapped by Ruby methods

For a user space application to make a system call: architecture and OS-dependent code must be invoked. At the lowest (userspace) levels this is implemented in non-portable assembly code.

Fortunately, most system calls are already provided as wrappers by the system C library (libc) so they appear to user space as portable C functions. Ruby itself wraps these C functions as Ruby methods. Even non-C Ruby implementations are likely to call these C functions in libc (rather than implement the non-portable assembly themselves).

Thus:

IO.pipe in Ruby is a wrapper for the pipe(3) C function,
which in turn wraps the pipe(2) system call.

You may not find the pipe(3) manpage because pipe(3) is a very thin wrapper for the pipe(2) syscall and the pipe(2) manpage is equivalent.

License: GPLv3 (or later, at the discretion of Eric Wong)

USP: Traditional Unix filesystem behavior

normalperson@yhbt.net (Eric Wong) — Wed, 14 Sep 2011 23:55:00 -0400

Eric Wong wrote:

Traditional filesystems are backed by stable storage (hard drives, SSD) and persists across reboots and shutdowns. Some of these are designed for Unix and considered POSIX-compliant, while others were designed for other operating systems and incompatible with POSIX to various degrees.

POSIX-compliant filesystems are the standard, other (including pseudo) filesystems attempt to expose a user-interface similar to those in POSIX-compliant filesystem. Unfortunately this means there are sometimes leaky, incompatible, and even dangerous abstractions may be present.

Unix filesystems are byte-oriented. They have no notion of encodings, and thus cannot be case-insensitive. They do not normalize path names or data in any way. The lack of conversion and normalization makes path lookup operations fast.

In a path name, only the \0" byte is disallowed. The “/” byte is used to delimit directories so it may not exist in a path component itself. Otherwise, path names may contain any other byte value (even gibberish).

Regular files will store any binary data stream without altering it in the kernel. Filesystem operations have no notion of “lines” and won’t convert CRLF line endings to LF or vice-versa.

There are many filesystem operations defined to be atomic in POSIX. These atomic filesystem operations are ideal for helping multiple processes interact with each other safely. Unfortunately, they also make POSIX-compliance difficult for implementations of network/distributed filesystems.

We will cover atomic operations in detail as we progress, including caveats for working with non-POSIX filesystems.

License: GPLv3 (or later, at the discretion of Eric Wong)

USP: An overview of the Unix filesystem

normalperson@yhbt.net (Eric Wong) — Wed, 14 Sep 2011 23:30:00 -0400

Unix features a hierarchical filesystem visible to all running processes. It is the primary shared namespace within a Unix system.

As you may know, the top of this namespace starts at “/” (or the root) and subdirectories may be created under it, each path component separated by “/”.

Filesystems may contain other filesystems. The root filesystem on a modern Unix system almost always contains other filesystems.

Filesystems may have several types of files: directories, symlinks, UNIX domain sockets, FIFOs, character devices, block devices and of course “regular” files.

Various pseudo-filesystems exist, but they are mostly non-standardized. Among pseudo-filesystems, there are memory-only filesystems, network filesystems, proc (process information) filesystems, device filesystems, and filesystems for tuning / inspecting kernel and hardware.

Pseudo-filesystems allow Unix-based operating systems to avoid implementing system calls only useful within a limited scope..

The “everything-is-a-file” design and use of filesystems (even for non-storage tasks) allows a few system calls to be useful for multiple purposes. This consistency improves the user experience by reducing the amount a systems programmer needs to know and also improving discoverability of the interfaces.

As the amount of system calls is relatively small, learning Unix systems programming should be easier than in other operating systems.

Plan 9 (the intended successor to Unix) takes “everything-is-a-file” more literally to further minimize the need for specialized system calls. This Plan 9 idea (among others) is being co-opted into existing userspaces and operating systems.

License: GPLv3 (or later, at the discretion of Eric Wong)

USP: The Unix object model (as seen from user space)

normalperson@yhbt.net (Eric Wong) — Tue, 13 Sep 2011 04:24:00 -0400

If you’re a Rubyist, you may already be familiar with object-oriented designs. A running Unix/Unix-like kernel exposes two primary types of objects to userspace:

Processes
Open files

Processes

Processes are instances of running applications. Ruby programs typically run as their own process¹ in userspace. Kernels expose a Process IDentifier (PID) as positive integers to userspace programs to uniquely identify processes.

Each PID represents one (private) process object in the OS kernel. The PID is only a reference or pointer to that object. Two running processes cannot have the same PID, and two PIDs cannot refer to the same process, but PIDs can be recycled and reused over time.

PIDs are global in scope to each running Unix system.

Processes are created from its parent using the fork(2) system call (which is wrapped by the Kernel#fork method in Ruby).

We will cover processes more in later chapters / posts.

¹ although experimental Multi-VM implementations exist that allow multiple Ruby VMs within a single process

Open Files

Running processes may create, open, and release objects known as “files” in kernel space using APIs provided in user space. Much of Unix systems programming revolves around manipulating “file” objects of various types.

The kernel exposes non-negative integers known as “file descriptors” to userspace. These are similar to PIDs as they are integers which point to objects within the kernel.

You may have heard the term: “everything is a file” in Unix literature.

Indeed, each integer file descriptor may refer to one of several types of objects within a running kernel. All of these objects can be called “files” even if they’re not stored on hard disks or visible on the filesystem.

File descriptors may be reused and recycled within the lifetime a process. They are more frequently recycled than PIDs.

Each process has its own file descriptor table (inherited from its parent).

Unlike PIDs, several file descriptors may refer to the same file object inside the kernel.

File descriptors have properties of their own that are not tied to the underlying file object in the kernel.

As open files are the primary interface for processes to interact with the kernel, I expect the majority of my future posts to cover various aspects of file manipulation in-depth.

Summary

Processes are containers for open files (among other things). Open files are objects in the Unix kernel that can be accessed from within processes.

License: GPLv3 (or later, at the discretion of Eric Wong)

USP: Prerequisite Knowledge

normalperson@yhbt.net (Eric Wong) — Tue, 13 Sep 2011 01:29:00 -0400

Introduction (by RKlemme)

This series started out with this posting of Eric to ruby-talk:

I would like to do a series of mailing list posts on the subject of Unix systems programming in Ruby. I only intend to cover standardized, POSIX-compliant features of the standard Ruby distribution. No extensions or Linux-only parts.

I volunteered to edit his postings and mirror them on the “Practicing Ruby” blog (formerly “Ruby Best Practices”). I will try to mirror articles as quickly as possible but the blog will certainly be lagging behind a bit.

You may discuss articles here on the blog or on the mailing list where there are originally published: usp.ruby@librelist.org (subscribe)

Now let’s start with the original text.

What you should know

You should know basic Ruby syntax.
Bourne shell knowledge will be useful, but not required.
You do not need to know C, though I’ll refer to and map Ruby methods to corresponding or analogous C functions.
You must know that that two (or more) variables (in Ruby) can refer to the same underlying object. This concept is present in Unix, so it is very important one understand this:

       a = "this is a string"
       b = a      # 'b' refers to the same String object as 'a'
       a << "!"   # modifies the String object 'a' points to

       "this is a string!" == a

       # You should understand why the following statements are true:
       "this is a string!" == b
       "this is a string" != b

Notations/conventions

Ruby method documentation references (should be the same as Ruby documentation)
- IO.pipe — class/singleton method
- IO#stat — instance method
C function documentation references
- pipe(2) — “#{function_name}(#{section})”
  The C function manpage can be accessed as:
  man #{section} #{function_name}
  so you can open documentation for pipe(2) using the command: man 2 pipe

That’s all I can think of for now…

License: GPLv3 (or later, at the discretion of Eric Wong)