OSDev.org

I am 99% sure you didn't read the papers, or if you did read them, you didn't understand them. How exactly is Singularity "getting inbetween the processor and the program"? The OS is compiled to native code, the apps are compiled to native code, there is no JIT'ing...?

There are some overheads like bounds checking and GC, but some (like bounds checking) are largely eliminated by the compiler. This is especially easy since there is no dynamic linking -- Bartok can do whole-program optimization of every app you install. Also, eliminating the need for address space switching saves a lot of TLB misses and makes the kernel faster (less mucking about with page tables).

While there are tradeoffs, I think the idea has a lot of merit, and I think you've blown the run-time overhead way out of proportion and completely ignored the performance gains of this type of architecture.

I don't whether to be happy your wrong, or embarased i did read alot of the papers. As far as i can tell it does JIT on the apps in order to do pre execution verification on the MSIL. I aplogise if i misunderstood it.

Why don't you have a look at the Mono project, and embed a CLI (along with common libraries) into your OS. Your drivers will still need to be asm/C but you could write all your userland programs (shell, GUI, everything else) in any .NET language.

Actually.. this has given me an idea.

I would like to have a static compiler, not a JIT. That's how they have done it in the Singularity project, and it has many advantages, for example in performance.

AHAHAHAHAHAHAHAHA!!! Me and my friend a long while back made that quote and started spreading it everywhere. Funny as heck to see it now.

Well, as said about python on page two, being more sophisticated than C. I very much have to agree with that statement. Python my look simple but you can do unimaginable things with it. All those linux installers you see in SuSe and or Red Hat or Fedora, they're written in Python. Anaconda, the Fedora/Red Hat graphical installer, is Linux Kernel + X + Python Interpreter + GTK/PyGTK + Installer and maybe a few dynamic C/C++ modules to assist in kernel communication between python and the system. Very impressive if you ask me. Also, the desktop environment/panel system for the Winodw Manager, OpenBox, is completely written in Python.

This is why I take the time to write a backend system that runs python scripts from C for most programs I write . Of course, my OS doesn't use python, but it could IF I could be bothered to implement it .

The activity of off-topic posts on this forum pitches to edges just like sine curve.
Are we talking about C# OS programming or some other things?

inflater

C# I think.. I'm not sure anymore?

Someone could start an open source C#->binary compiler. But that would mean reimplementing the System namespace as a binary too. Microsoft used Sing# in Singularity, an extension of C#, for low-level coding. If someone released any Sing# documentation, I'd start working on a naive compiler.

EDIT: **native compiler.

I believe that to be the most convinient spelling mistake ever...

I find singularity concept really interesting in a lot of senses.

I personally find that the requirement for static compilation of whole programs at once is largely an artifact of the system semantics, and can be relaxed without sacrificing too much of either security or performance, while gaining advantages in a much more dynamic system. The idea is interesting though.

It is also important to realize (and I only realized how easy it would be a few days ago) that it's really quite easy to extend such system to also deal with untrusted native programs, though one then loses much of the benefits of language based security for those programs.

Anyway.. one important thing is that for the totally static approach of singularity to work, naive compiler is almost totally useless. The key to making the whole idea of using a huge monolithic blob-of-a-language as a system development language as in singularity, (and as far as I understand correctly) is specifically the idea that heavy global optimizations allow the compiler to generate code that in-a-sense only inline those parts of the runtime system that are actually necessary, which allows the resulting binary to avoid needing a runtime, yet keeps the binary sizes feasible even if the full runtime was huge.

As an example of one such optimization, say you have a class hierarchy like this:


Now, if the compiler see that in some place of your code you need to figure if "foo_parent" is really a "foo_child_A" or "foo_child_B", then it needs to store runtime type information in foo_parent.

But if we can do full program analysis and prove that this case of foo_parent is the only place where one needs to do this "dynamic cast", then "object" and the rest of the class hierarchy doesn't need any runtime type information, and we can just use a regular if- or switch- which checks a newly-added field in foo_parent for whether it's 0 (parent), 1 (child_a) or 2 (child_b) and get rid of all the rest of the mess that deals with runtime types of objects.

Or a class called "extensible" defined a overridable (in C++ virtual) function called "extendme", and compiler can prove (by some more global analysis) that this function isn't really ever extended, it can be called just like a static member function, without any vtables or anything.

In a typical library there's tons of stuff that deals with the fact that we don't know anything about the code that's calling the library, yet if we did global analysis of the program, we could often prove that we don't need any of the special cases, and hence can drop all the code that deals with 'em, as well as all the code that said dropped code would have dragged with it.

...

As for "everything you can do in C# you can do in C++" it's important to realize that everything you can do in C# you can also do by writing your code with a hex-editor (and yes, that includes even garbage collection). You might not want to do that ofcourse. Partly it's because some languages are more expressive than others, and party it's because some runtime environments contain more useful stuff than others.

Does someone know a way to compile c# code to native x86 (using mono or any other compiler)? Compiling an empty file with the "ahead of time" flag still leaves a lot of references to libraries.

I know of no current compiler... you would probably have to write your own... Also, i don't know what structures are kept to "manage" the code or how you would also compile the code that watches pointers etc, but u would need to do that also.

It sounds to me like you really have two separate problems: How to compile C# to native code, and how to build it to a freestanding binary with no external dependencies. You can always use ngen to compile any .NET assembly to native code. The real problem is the dependencies -- I don't know of any tools that will let you do this (MS keeps Bartok under wraps, unfortunately).

C# is a peice of ****, only microsoft could make C suck. C# is C++, but minus everything good about it (this may be java's influence). Just do it in C or C++, writing an OS in C# will give you a slow os, whose only potential advantage would be in security, but even that would require a lot of work, and could easily be replicated w/o C#.

Theres simply no good way to compile C# into native code, if as i suspect, MS did not release their assembler and compiler. If you are deadset on using C# and are up to the challenge, write a microkernel that can host a stripped down version of mono (open source .net platform), and write servers and applications that run above it. It would be tough, but in the end kinda neat.


If you want to write an OS in a non-native language, at least use LISP, then you OS can rewrite itself!


Peace.[/b]

Arguably, the only native language for a processor is it's own machine code. One could even claim that when you break each byte into two 4-bit parts in order to display it in a hexadecimal format, you are no longer working in the truly native format, though in this case the required conversion in trivial and bijective.

Once you introduce a macro-assembler which automatically selects best possible (usually shortest) encoding for a given logical instruction (such as using shortest possible offset encoding for a relative jump), you lose this bijection; now several "native" encodings would map to same logical encodings in the assembler language upon disassembly.

You can ofcourse regain bijection by only considering the semantics of the native encoding, instead of it's physical form or alternatively you could establish a bijection between native encoding generated by said assembler, and it's disassembly, provided your assembler is deterministic. At this point stuff is still pretty theoretical and uninteresting....

Now, what this has to do with rewriting a system on the fly? Well, when you introduce offline components like a linker, stuff gets more interesting: consider a linker, which takes care of replacing labels with real addresses, and links the result in a flat binary. Now, starting from the original source with labels, assembling and linking, we have no way to regain that information when disassembling... unless ofcourse we store debugging symbols or some such...

However, for the purposes of dynamically replacing a system, we not only want to be able to disassemble the system, but also modify it, then assembling it again. Ofcourse if we keep the original source, we can cheat somewhat by modifying said source, and replacing previous compiled version with the newly modified and compiled version. However, if the system contains information which is not accessible at runtime, then modifying such information at runtime will likewise be impossible.

The nature of the language used is not really important. Rather, in order to actually avoid rewriting native code on the fly, we want to use late-binding for everything that is a candidate for runtime modification. If stuff like procedure addresses are hardcoded into other procedures, then we need to modify those procedures in order to replacing another procedure... or implement some sort of kludges in order to reroute the calls. If on the other hand, there is an indirection with said calls, then simply modifying this indirection will be enough.

If Lisp is good for this, it is simply because (and only when) said Lisp is very late-binding. Smalltalk could easily do the same. There is even research on extending C++ like vtables to support dynamic late-binding object models, which allow methods to be replaced on the fly.

Doing a complete replacement of a system is ofcourse not quite trivial, since one will normally need at least small amount of "black box" runtime code to get any higher level language (including C) running, but if said runtime is only needed to get the actual system running, and any parts of it necessary during execution use some form of late-binding, then it becomes possible to replace those parts after startup and we can consider the system able to rewrite itself as a whole.

--------------

I am actually doing some research on this stuff right now. The current approach I'm looking into basicly considers a "byte code" form to be the "native" or canonical representation of code. With a fairly simple indirection mechanism one should be able to arrange functions to be JIT compiled to native form the first time they are called. At any point of time, one can throw the native representation away, and instead resort to evaluating the bytecode directly, or JIT compile it again.

So why bother? Well, the point is to be able to garbage collect code. Now, if the native code is nothing but a cached version of the bytecode, and the bytecode is nothing but data, then one avoids having to disassemble native code to see what data is refers to. It also means that if something is statically bound at JIT compilation time, if those assumptions (like locations) change later, all we have to do is throw the native code away, and it'll get recompiled automatically as the system attemps to execute it.

Finally, dumping the whole system state into a file becomes a lot easier if you don't need to try to dump native code. Instead just have the heap with the bytecode dumped, and have the loader contain an interpreter. JIT compiler can be bytecode in the heap just like everything else.

....

Well, then the idea is to throw a REPL on it, get it to run in priviledged mode on bare hardware, without relying on static runtime after startup, give it direct access to memory, and have fun.

And after that add a bytecode validator which can figure out if running some code would be dangerous, and you got language based security.

And yeah, my current plan is to use something like Lisp. Actually Scheme without first-class continuations would be closer.