I'm currently hearing a lecture on programming paradigms which has left me rather fascinated. Up to now, it has primarily been about the Haskell Language, which seems to be based on simple and elegant concepts.

As a side note, this is my first time getting in practical touch with functional programming and working with such a language, so I might get things wrong.

Basically, in functional languages there's no state and all functions calls lead to some kind of evaluation (or rather transformation) of expressions that return a value. This is totally different from what is contemporarily known as OOP.

Like every so often I have some gut feeling but can't put a finger on it. So I feel like dumping a WIP.

As you might know, I design an object system. Or maybe I'm creating some Frankensteinian monster out of several systems and concepts that influenced my thoughts, most notably Smalltalk, Self, Newspeak and maybe Lisp (despite not ever having written a line of code in any of them, except for the first one).

Anyways, let me explain how the concept looked the last time I worked on it (which already was several months ago).

struct Object
{
    const struct Object* const slots[SLOTS_SIZE];
};

(Of course, the level of detail has been reduced heavily.)

A message send in this system, for example a foo: b (to use Smalltalk syntax), is just syntactic sugar for the evaluation of a code object stored in the specified slot (for example in slot foo:), using appropriate parameters (foo value: b).

A code object, or rather code generally, is a closure (comparable to a Smalltalk block), which basically is a lambda expression whose free variables have been bound.

When an execution context is set up, a certain variable (this/self) is defined to reference the object receiving the message. This way, we can use the well-known programming style of methods.

Since we're using the capability security model, there are no globally accessible objects and no objects can be accessed that have not been passed as a parameter. Note that since everything is an object, it follows that there is no global state. The accessibility relation, however, is transitive, so one can chain multiple message sends and probably have access to a vast part of the object graph.

Also note that both pointers and pointees are const. This means that all objects and all indirectly reachable ones are immutable[1]. In other words, our message sends are referentially transparent or maybe even pure, probably with the downside of complex "hidden" state.

But this seems backwards. What use is a immutable operating system? The real world does have state.

This is a requirement of versioning. At each point in time, the system is in a fixed state. A historic state can not be changed by definition. However, with the blink of an eye you can go backwards im time, fork a new branch of evolution and switch between several of them -- on an object/subtree level, of course.

Currently, I plan to use few special types of transparent native references, primarily for improving capability security. In this text, the focus lies on an update-following reference used to give its owner the impression that the current state of the system is mutable by allowing certain objects to always automatically reference the latest version of another object.

Unfortunately, a update-following reference reintroduces side effects, even if they are restricted to certain situations.

Consider the following snippet of code for such a system:

f := [ ... ].
((f value: x) = (f value: x)) ifFalse: [
    self error: 'Referential transparency?'.
].

Now that snippet will always pass if it is run in a context in which references are not automatically updated when a new version of an object is created[2].

Such modification of the context is required anyways. After all, we want to be able to "check out" historical state and continue to work from there. This requires even automatically updated references to point to the latest version of an object as it was at that point in time. Similarly, the context can be set up to only include object versions whose date is smaller or equal to the date the snippet was started to be evaluated.

This raises several questions. Can this simple and elegant[3] scheme reduce side-effects significantly? Can it improve the quality of the system? Is object-orientation without/with less/with controlled side effects possible?

Furthermore, code evaluation in this system is still following the execution-pattern and is not designed to be thought of as transformation of expressions. So, can such an object-oriented scheme provide a practically usable way to evaluate expressions lazily?

---

1. Let's ignore const-casts, the code above is just an illustration
2. This might not be completely true. How is nondeterminism handled, e.g. random numbers and external events/input?
3. Well, at least I hope it is simple and elegant.

Once again I realized that there's still so much we don't know, so much we still have to learn. All of us, but especially me.

I'm refering to the following words of Alan Kay on the FONC mailing list:

All I can say, is that it is not about code, and I'm pretty sure that playing with code won't help STEPS. In many ways it's about escaping from code! Escaping from thinking in terms of code.

It's about concepts in dynamic relation, that beg to be represented clearly, and then be put into a runnable form, most often via a new language made for the purpose.

"Well, of course,", I should've thought, "he's been saying this for several years now!":

We don't know how to design systems yet. So let's not make what we don't know into a religion for god's sake. What we need to do is to constantly think and think and think about what's important. And we have to have our systems let us get to the next levels of abstractions as we come to them.

-- Alan Kay in his talk at OOPSLA 97

Somehow, I managed to not get it until now.

agg is a news aggregator that stays focused on its goal, namely reading a feed and creating a representation that can be worked with in the whole system. The process is straight-forward: store all items[1] that have a publication date newer than that of the latest item received previously.

That works remarkably well in the common case. A less common case is feeds without publication dates for their items. In fact, the pubDate tag is not required according to the RSS 2.0 specification. What to do in this case?

The solution is simple: if the author ignored publication dates, the aggregator shall do so, too. This has the inconvenient effect that once you've read all items and probably deleted them afterwards, running agg again on that feed will lead to all these items being there again!

So we need some form of caching[2]. However, such a rare case that hasn't much to do with agg's job anyways should not end up in the same piece of software. Here are my tools for the output format of versions 0.2:

agg_filter to remove an item iff it has already been cached:

ITEM_NAME="$1"
FEED_PATH="`dirname "$ITEM_NAME"`"
FEED_NAME="`basename "$FEED_PATH"`"
grep -q "`agg_hash "$ITEM_NAME"`" ".$FEED_NAME.cache"
[ $? -eq 0 ] && nomtime "rm -rf" "$ITEM_NAME"

agg_cache to cache an item:

ITEM_NAME="$1"
FEED_PATH="`dirname "$ITEM_NAME"`"
FEED_NAME="`basename "$FEED_PATH"`"
agg_hash "$ITEM_NAME" >> ".$FEED_NAME.cache"

agg_hash to create a hash of an item:

cat "$1/title" "$1/desc" "$1/link" 2>/dev/null | sha1sum | awk '{ print $1; }'

With the agg_each script posted in using agg 0.2, the scripts are usually used as follows:

> $fetch_all_feeds_and_pipe_to_agg
> agg_each agg_filter "feed without pubDates"
> agg_each agg_cache  "feed without pubDates"
> agg_each agg_read *

But why this hassle? Why not just integrate it into agg, or use a real newsreader for that matter?

To be fair, on operating systems with more powerful concepts, agg is unnecessary since it basically does nothing more than performing a deserialization of a news feed.

UNIX, however, is a rudimentary operating system and lacks powerful concepts. In this case the problem boils down to the completely rudimentary native objects and methods of communication.

When a set of people communicates openly with the requirement that everyone should be able to take part in the communication, every person in the set must speak in a way everyone can understand. Thus, the communication can only be as smart as the dumbest person in that set. Else, you'd have to introduce even more people into the set because translators between the smart and dumb ones are required. Not only is this process cumbersome, but also you'll be occasionally lost in translation.

Speaking in terms of this metaphor, UNIX is dumb. Processes are only expected to communicate via streams of lines of text, files and directories[3] this is especially true for the whole base system.

So, in order for users to be able to actually use the system (as opposed to using yet another application), agg either has to be file-based or needs even more (and even more complex) auxiliary tools, which indirectly leads to exactly the problems mentioned here.

Yes, it's a hassle. But it's the only practical way to at least partially achieve something Alan Kay has explained in his talk The Computer Revolution Hasn't Happened Yet at OOPSLA 1997:

Well I had programmed Caesar Franck's heroic piece—and if you know this piece, it is made for the largest organs that have ever been made. The loudest organs that have ever been made, in the largest cathedrals that had ever been made, because it's a nineteenth century symphonic type organ work, and Biggs was asking my friend to play this on this dinky, little organ.—He said, But how can I play this, on this? Biggs, he said, Just play it grand. Just play it grand. To stay with the future as it moves, is to always play your systems more grand than they seem to be right now.

---

1. Currently there's a bug, but this is the concept anyways.
2. A cache of your brain, that is, so that the software doesn't try make you read data that's already in there again.
3. Yes, and *argv[], fooenv(), shmfoo() etc. They don't matter in our case since we have "complex" data structures and no sane person would ever imagine using a computer by writing C and following the ancient edit-compile-debug cycle. Also, this issue has already been covered in TraditionalApplicationsConfigurationInterfaces and DisconnectedMonoliths.

Version 0.2 of agg, has just been released. It is mostly consistent with the 0.1 versions in terms of bugs, but the output format is completely different.

The previous version created (absolutely poorly formatted) HTML files to represent the news items. This was good enough for my use case. But when I briefly told a friend about this project, he proposed a different use case that was not possible in the current concept.

As always, my subconscious started working on this issue, and some time later something popped into my mind. I've been claiming that agg does the simplest thing that could possibly work, namely only dumping news feed items. But this was not entirely true. In fact, agg also knew a bit about HTML and formatted the output accordingly. This not only violated one of agg's goals (having a single responsibility) but also virtually discarded all meta information. Such meta information, however, is required for use cases like the one proposed by said friend of mine.

Starting with the 0.2 versions, agg represents news items as directories with all supported (and available) properties as single files (currently title, desc(ription) and link). For starters, here's a new (compacted) version of the CLI I posted for the previous versions.

agg_each:

CMD="$1"
while [ $# -gt 1 ]; do
    shift
    find "$1" -mindepth 1 -maxdepth 1 -type d -exec "$CMD" '{}' \;
done

agg_read:

ITEM=$1
function delete() { nomtime "rm -rf" "$1"; }
function tui() { agg_htmlize "$1" | elinks; }
function gui() {
    TMPFILE="/tmp/`basename "$1"`.html"
    agg_htmlize "$1" > "$TMPFILE"
    opera "$TMPFILE" # asynchronous if already running
    sleep 3
    rm "$TMPFILE"
}
function TUI() { elinks "`cat "$1/link"`"; }
function GUI() { opera "`cat "$1/link"`"; }

function prompt()
{
    echo "$1"
    CMD=
    while [ "$CMD" != t -a "$CMD" != T -a "$CMD" != g -a "$CMD" != G -a "$CMD" != d -a "$CMD" != n ]; do
        echo -n "[t]ui, [g]ui, [T]UI, [G]UI, [d]elete, [n]ext: "
        read CMD
    done
}

while [ "$CMD" != n -a "$CMD" != d ]; do
    prompt "$ITEM"
    case "$CMD" in
        t)  tui "$ITEM";;
        g)  gui "$ITEM";;
        T)  TUI "$ITEM";;
        G)  GUI "$ITEM";;
        d)  delete "$ITEM";;
        n)  ;;
        *)  exit 1
    esac
done

Htmlizing can work as follows:

TITLE="`cat "$1"/title`"
BODY="`cat "$1"/desc`"
LINK="`cat "$1"/link`"

cat << EOF
<html>
<h1>$TITLE</h1>
<p>$BODY</p>
<p><a href="$LINK">Link: $LINK</a></p>
</html>
EOF

The workflow then looks as follows:

> cd ~/feeds
> $script_to_fetch_all_feeds
> agg_each agg_read *
./Foo Feed/Just in: Bar happened
[t]ui, [g]ui, [T]UI, [G]UI, [d]elete, [n]ext:
...

Each item, one after another, its link and all the links it contains can be browsed using the browser (text/gui) selected. By using the capitalized key, the respective browser directly opens the link specified by the news item (if any), which is useful for sites that crop their feed items excessively.

Additionally, browsers should not have problems with the file contents anymore, provided you perform proper htmlization that the browser recognizes as such.

Now the interface is truly flexible.

Hibernation, suspend-to-disk, whatever you want to call it. I'm talking about ACPI S4: Saving the contents of the main memory to secondary storage and powering the machine completely down. Upon the next restart the previous memory contents are restored.

This is a great feature since one can continue working as if nothing happened. At least in theory.

There are several issues. What if there is not enough space on the hard disk? What about devices and their drivers?

Hardware problems concerning ACPI S4 (and others) are very common on all traditional operating systems[1]. I believe this is partly due to the fact that this feature has been mostly hacked on top of the rest of the system.

The system itself still uses explicit memory management which can lead to a problem that I've encountered often. When the system is commanded to hibernate midst in (memory) intense work there might not be enough space on secondary storage[2]. However, if you instruct the application(s) manually to store their state, close them and then start them again to load the previously state, all their internal non-essential state[3] has been dropped. Oftentimes hibernation is now possible.

Now, image if you have to do this for a dozen applications, each a different interface and underlying mechanism. Pretty lame, isn't it? The advantage of hibernation, both speed and convenience, have neatly been destroyed.

So, what would happen if we put this kind of functionality into the operating system? Memory management actually is already in there. The problem is that applications are free to claim memory they don't need -- with good intentions. I'm not going to research a "proper" conservative solution, since it is inconsistent (see next paragraph) and most likely hard to implement -- and probably impossible to enforce.

Again, a proper solution is simple but contrary to traditional systems. If the whole secondary storage is only a swap space for all the objects in memory, applications can claim as much memory as they want -- until the whole storage is used up[4]. Also, if the changed contents of the main memory are periodically written back, you get hibernation for free. Even better, you get "hibernation" always, even in case of sudden power failures.

This is known as transparent persistence. And I want it.

---

1. Except maybe for a certain vendor that sells both its own hardware and OS.
2. At least on GNU/Linux. Honestly, I don't get it. Having a swap partition of four times the size of the main memory and using not even close to half of it should be sufficient. Maybe my data is inaccurate, but still it looks buggy to me.
3. e.g. application-level caches
4. Quotas are orthogonal to this issue and needed in traditional systems, too.

And if so: Is it the fault of the internet?

Jakob Nielsen published a new alertbox recently. He claims that

[t]oday, many users are so reliant on search that it's undermining their problem-solving abilities. -- (source)

I think I developed this pattern, too. Now that I noticed it, its time to fix it.

But maybe it has always been this way for me. It might just be my university's demands that show me how I can improve even further.

Anyways, aforementioned article contains an interesting thought:

It's sad to think of the vast number of patients who get misleading medical guidance from the Internet because the main search engines currently prioritize popular sites instead of useful ones. -- (source)

At the first glance this looks worth being fixed. Noted.

What follows is a post that has been transcribed from an (complete, but orthographically awful) electronical manuscript, probably dating to mid 2010.

---

At first, I'd like to say that I grew up with several terms and don't want to complicate this post by introducing new ones. It should be clear that just because "we always did it this way" is not an argument for not improving the concept of programming.

To make things clear, in this article, code is a graph of statements that can be executed by the OS[1]. Writing code means creating code via one or multiple ways of interaction with a system (text files, graphical programming languages, etc). Said operating system is assumed to be a end-user system, desktop operating system, consumer system, PC OS, or whatever you want to call it.

Any serious operating system provides a (primary) mechanism to be programmed, to which I in following refer to as "the language".

---

Traditional operating systems require that you have mastered programming before you can use them fully. And it does not much to help you on this way.

With traditional concepts, the bare operating system is a really bad teacher. Both in how it works and can be used, and how it can be programmed. Using and programming are not this different in concept, actually, if you're abstracting enough. It is essentially the concept of traditional systems that keeps the distinction between programmers and mere users intact.

When you can not learn the language from the OS you have to learn it from elsewhere. For starters tutorials on the web provide the first teacher substitute. They have you writing code to feed the OS and give some comments on what you have been doing. However, the OS could be your personal tutor that could show you programmatical equivalents of your day to day interactions, explain stuff and teach you what really is relevant to you.

Traditional operating systems have the additional problem that their language is low-level and/or not powerful[2]. The problem of the latter is obvious. The former makes it significantly harder for beginners to learn it.

So, let's assume our hypothetical user has managed to learn both programming and the language of the system, and encountered a problem that she wants to solve. Finally, all these hours will pay off! But where to start? What is she going to do? Read tons of documentation provided by the OS in hope to find a pointer that gets her further? Most likely she'll ask Google. Afterwards she read some specific documents of the OS. But maybe she's already found a solution because other people weren't able to come up with one to a similar problem themselves.

With third party programs like script language interpreters that come with batteries included[3], it is significantly easier to solve problems. However, they suffer from the same flaw: They usually do not completely teach you programming and do not allow you to programmatically explore the system. They even may not provide mechanisms for controlling every part of the underlying hardware that could be controlled. If they do, they're rather evolving to an OS themselves[4].

I understand every user that is afraid of trying to learn programming a traditional OS. And I understand every user that does not want to spend many hours with static tutorials[5] to solve a simple problem each few weeks.

Simple things should be simple, complex things should be possible. -- Dr. Alan Kay

I understand every user that wants using the OS to be fun.

And I also understand that it is the job of the OS designers to make it easy for users to solve their problems. Making the whole system itsself dynamically programmable, creating a reflective, unified user interface that includes and teaches programming facilities, and employing progressive disclosure throughout the system could be a good way to give users the power and comfort they deserve.

---

1. fuzzyness intended
2. namely C and sh
3. hint hint
4. or an OS API, but the user doesn't really care about hidden implementation details
5. that might actually be outdated and/or completely wrong

Intuitively, it was already clear to me that we have to overcome the traditional way of programming if we want to achieve progress.

In CodingStandardsAreMisleading I explained that writing text that then is translated into business logic is an aritrary decision. This process is well-understood and we always did it this way. In the days of slow CPUs and scarce memory it actually made sense. But these days are over, at least on PCs. So should be the age of textual programming.

Introduce Subtext, yet another programmable system I discovered some days ago. But this one is different. Not different in the sense of various new Smalltalk forks compared to C, no, this one is different.

I always felt that some kind of GUI that increases efficiency drastically would be the interface for programming in the future. But your gut should not replace your head, and the latter is what Jonathan Edwards obviously used.

We're doing boolean algebra in our heads when there's a computer sitting right in front of us. Meanwhile, the computer is emulating a teletype.

Basically, Edwards moved programming from the first dimension (programs as more or less sequential list of instructions) into the second one (and added appropriate IDE support).

How he did do this? See for yourself.

Today I released the first version of agg, a news aggregator following the UNIX philosophy.

As clearly stated, it has many bugs. However, they are predictable and agg is working fine for all of the other feeds I've subscribed to.

Since a news aggregator that "just dumps" the feed's contents into a directory hierarchy provides only rudimentary efficiency[1] (but much higher flexibility and freedom than those monolithic, unprogrammable, totalitarian systems usually used), I've written a small interface for it (or rather for the file system structure).

These scripts are nothing special. They provide only the necessary features and are simple enough to achieve high efficiency[2].

How agg can be used to subscribe to multiple feeds is already shown in the man page. For reading the news items, I'm using two scripts: agg_read to read a specific item of a specific feed, and agg_read_all that calls the former for every item.

agg_read_all is trivial:

find -type f -exec agg_read '{}' \;

agg_read is still simple:

### CONFIG

TUI_READER=elinks
GUI_READER=opera

### END CONFIG

set -e

ITEM=$1

function delete()
{
    owd="`pwd`"
    cd "`dirname \"$1\"`"
    nomtime "rm '`basename \"$1\"`'"
    cd $owd
}

function prompt()
{
    echo "$1"
    CMD=
    while [ "$CMD" != t -a "$CMD" != g -a "$CMD" != d -a "$CMD" != n ]; do
        echo -n "[t]ui, [g]ui, [d]elete, [n]ext: "
        read CMD
    done
}

while [ "$CMD" != n -a "$CMD" != d ]; do
    prompt "$ITEM"
    case "$CMD" in
        t)  $TUI_READER "$ITEM";;
        g)  $GUI_READER "$ITEM";;
        d)  delete "$ITEM";;
        n)  ;;
        *)  exit 1
    esac
done

The workflow then looks as follows:

> cd ~/feeds
> agg_fetch_all
> agg_read_all
./Foo Feed/Just in: Bar happened
[t]ui, [g]ui, [d]elete, [n]ext: 
...

And each item, one after another, its link and all the links it contains can be browsed using the browser (text/gui) selected.

A simple solution, and only an example of the seemingly endless amount of interfaces that could be written for the output of agg. None of them needs to give a damn about XML, RSS, Atom[3] or download logic.

Heck, even agg itsself knows nothing of networking!

---

1. human-time
2. as usual, efficiency measured in human-time
3. Atom not implemented as of yet

Disconnected Monoliths

This is a follow up to TraditionalApplicationsConfigurationInterfaces, which claimed

[on] traditional operating systems, programs [...] run as monolithic objects that are detached from the environment.

At first, let me correct the above statement. It is just wrong that "programs run [...] detached from the environment". Parameters and configuration files have been mentioned, environmental variables have been forgotten. However, what that article was trying to say remains valid, but has not been explained properly.

The original article showed how it practically impossible to dynamically change the runtime behaviour of an individual application.

This article will explain why it is practically impossible for application developers on traditional systems to allow for dynamical reuse of their application

---

As explained, on traditional systems all input to programs is considered to be text[1]. But so is output.

On static class-oriented systems, one could pass any object of any class that inherits from the expected base class or implements the neccessary interface. On dynamical object-oriented systems, one could pass any object that understands the required message protocol[2]. These are valid demands[3].

On current operating systems, even if application developers factor their code thoughtfully, there is still no possibility to pass in arbitrary[4] objects. This is not only because the input is just text, but also because the output is just text, too.

There exists no space between different processes. There are no objects. There is only void; and null-terminated byte strings.

Of course, since these systems are turing complete one could make passing objects possible. However, is it the job of application developers to write code that enables their applications to accept and pass arbitrary, live objects? Or is it the job of operating system designers to provide an easily usable environment?

I will create a universe.

---

1. Actually, its not even text but strings of bytes.
2. Or, you could pass any object but get an exceptions if it does not understand a message it has been sent.
3. Again, see Open-Closed-Principle.
4. As explained in TraditionalApplicationsConfigurationInterfaces