Channels ▼
RSS

Google's Summer of Code: Part IV



NetBSD-SoC: Efficient Memory Filesystem
Pyasynchio: The Python Asynchronous I/O Library

NetBSD-SoC: Efficient Memory Filesystem

Name: Julio M. Merino Vidal
Contact:jmmv84@gmail.com
School: Barcelona School of Informatics, Spain
Major: Computer Science
Project: NetBSD/tmpfs
Project Page: http://netbsd-soc.sourceforge.net/projects/tmpfs/
Mentors: William Studenmund
Mentoring Organization: The NetBSD Project (http://www.netbsd.org/ )

NetBSD inherited the Memory File System (MFS), a filesystem that uses virtual memory pages to store its contents, from 4.4BSD. MFS implements the on-disk Fast File System-s lower layer to store blocks into memory. This is inefficient because the same algorithms used to lay out data on disk are used for memory, introducing redundancy and complex data treatment. Despite that, the worst problem with MFS is that, once it has mapped a page, it never releases it.

tmpfs (short for "temporary filesystem") solves these problems. It is a completely new filesystem that works with NetBSD-s virtual memory subsystem, UVM. It aims to be fast and use memory efficiently, which includes releasing it as soon as possible (for instance, as a result of a file deletion).

At this writing, tmpfs is already integrated into NetBSD-current (3.99.8), which makes testing and further development a lot easier. All the filesystem's functionality is written and works, albeit with some known bugs. The most significant problem is that it currently uses wired (nonswappable) kernel memory to store file metadata.

Let's now dive a bit into tmpfs' internals (full documentation is included in the project's tmpfs(9) manual page). As with traditional filesystems, tmpfs organizes all the files stored within it in a tree graph. Each file has a node associated with it, which describes its attributes (owner, group, mode, and the like) and all the type-specific information. For example, a character device has a major and minor number, a symbolic link has the name of its target, and so on.

Directories are implemented using linked lists, making their management extremely easy. Directory traversal is quick, and node creation and deletion need not be concerned with entry proximity nor gaps between entries.

Regular files, on the other hand, are interesting because they are managed almost completely by the UVM. Each file has an anonymous memory object attached to it, encapsulating the memory allocated to the file. This memory is managed solely by the virtual memory subsystem. This design meshes well with the Unified Buffer Cache used by NetBSD, where files are UVM pagers, and file reading and writing is performed by mapping and accessing pages of the file-s UVM object. Thus, the only difference between files on other filesystems and those on tmpfs is the nature of the UVM object holding the pages.

Back to Top


Pyasynchio: The Python Asynchronous I/O Library

Name: Vladimir Sukhoy
Contact: vladimir.sukhoy@gmail.com
School: Iowa State University
Major: Ph.D. candidate, Applied Mathematics
Project: Pyasynchio
Project Page: http://pyasynchio.berlios.de/pyasynchio-about.htm
Mentor: Mark Hammond
Mentoring Organization: The Python Software Foundation (http://www.python.org/)

Pyasynchio is a Python library created to support cross-platform asynchronous I/O (AIO) operations. Compared to the famous asyncore library, for example, pyasynchio is simple and does not require you to use object-oriented approaches. It just provides a way to initiate AIO and poll completion status. It is up to library users to implement patterns for processing operation results. At the moment, Pyasynchio supports file and socket AIO (only on Windows, but POSIX AIO and Solaris AIO support is coming). Pyasynchio uses built-in Python file and socket handles as AIO targets, so users can plug Pyasynchio into the middle of their code.

AIO is a way to alleviate I/O bottlenecks without multithreading. Usually, AIO is efficient in terms of resource usage (sometimes more efficient than multithreading, select/poll, or mixed solutions). AIO allows additional speed optimization at the OS and driver level and does not require an operation to complete during the call to the function that started it. A well-known object-oriented implementation of AIO on top of different low-level OS facilities can be seen in the ACE library (http://www.cs.wustl.edu/~schmidt/ACE.html).

It is important to note that AIO is different from synchronous nonblocking I/O (often, nonblocking I/O is erroneously called "asynchronous" I/O, but it really isn-t because each nonblocking I/O operation is still synchronous). Pyasynchio makes use of the Proactor pattern and Asynchronous Completion Token pattern, but the dynamic type system of Python makes some of Proactor-s object-oriented burden unnecessary. Applications of Pyasynchio may be even greater when Python introduces coroutines support (PEP 342). With coroutines, some I/O processing tasks can be coded in a synchronous manner (without distributing connectionprocessing logic over numerous handler classes/callbacks) but will still be able to take advantage of nonblocking I/O or AIO.

It is worth mentioning that a coroutine-based approach is possible with ordinary nonblocking synchronous I/O (as outlined in PEP 342), but AIO coroutines may be more effective in terms of resource usage and/or speed and/or implementation clarity. Pyasynchio code and binaries are available under an MIT license.

DDJ

Back to Top


Related Reading


More Insights






Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

 
Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.
 

Video