Discussion:
New kid on the block / introduction
M. Edward (Ed) Borasky
2006-10-30 22:40:45 UTC
Permalink
I just discovered the TUNES project yesterday, following trails on the
Internet (Linux assembly and various Forth interpreters). I was a Forth
hacker in the golden FIG days, but I haven't done much with it recently.
Right now, I'm mostly interested in high-performance implementations of
the Ruby language ... virtual machines, assembly language kernels,
compiler hacks, etc.

In particular, I'm mostly interested in efficient use of multi-core
64-bit Intel and AMD processors on Linux platforms. Windows and MacOS
compatibility are less desirable -- I really want (to build) the
ultimate 64-bit multi-core multi-node distributed Ruby platform.
Tom Novelli
2006-11-02 12:39:07 UTC
Permalink
Welcome aboard, Ed. Sounds like you got into this the same way I did,
through Fare's Assembly-Howto. You probably found RetroForth too...
the newer versions might be useful as an example for interfacing with
Linux.

It's good to have a Ruby hacker here, as none of us seem to be, and
it's an unusual language. I've only played with it a little, but it
looks like LISP without parenthesis... Eventually we might like to
incorporate Ruby into our great multi-language compiler system,
when/if we write it, but don't hold your breath -- it could be
decades. I would encourage you to go ahead with your project... If
you succeed, it'll make Ruby a more viable language. Elegant,
high-level, high-performance implementations (for any language) are
hard to find.

- Tom
Post by M. Edward (Ed) Borasky
I just discovered the TUNES project yesterday, following trails on the
Internet (Linux assembly and various Forth interpreters). I was a Forth
hacker in the golden FIG days, but I haven't done much with it recently.
Right now, I'm mostly interested in high-performance implementations of
the Ruby language ... virtual machines, assembly language kernels,
compiler hacks, etc.
In particular, I'm mostly interested in efficient use of multi-core
64-bit Intel and AMD processors on Linux platforms. Windows and MacOS
compatibility are less desirable -- I really want (to build) the
ultimate 64-bit multi-core multi-node distributed Ruby platform.
M. Edward (Ed) Borasky
2006-11-02 15:56:58 UTC
Permalink
Post by Tom Novelli
Welcome aboard, Ed. Sounds like you got into this the same way I did,
through Fare's Assembly-Howto. You probably found RetroForth too...
the newer versions might be useful as an example for interfacing with
Linux.
Yep -- RetroForth, RevaForth and FreeForth/FASM. Right now I'm leaning
towards the FreeForth/FASM, since ANS Forth compatibility isn't critical.
Post by Tom Novelli
It's good to have a Ruby hacker here, as none of us seem to be, and
it's an unusual language. I've only played with it a little, but it
looks like LISP without parenthesis... Eventually we might like to
incorporate Ruby into our great multi-language compiler system,
when/if we write it, but don't hold your breath -- it could be
decades. I would encourage you to go ahead with your project... If
you succeed, it'll make Ruby a more viable language. Elegant,
high-level, high-performance implementations (for any language) are
hard to find.
Ruby is mostly a hybrid of Smalltalk and Perl. It has lambdas, blocks,
closures and continuations, but not tail recursion, and there's quite a
bit of debate in the core Ruby developer community about the future of
continuations. Its connections with Lisp and Scheme are mostly through
Smalltalk and not directly from the Lisp family. See

http://borasky-research.blogspot.com/2006/10/rubyconf-2006-impending-ruby-fracture.html

There was a project to build a Ruby VM using gForth/vmgen but its
developer has abandoned the effort. At the moment I don't have any
64-bit hardware to test on; that will probably be March. So I'm going to
build a 32-bit system optimized for Linux and Windows 32-bit Vista on
the machines I have -- a Pentium 3 and an Athlon XP. I don't see any
reason in the world *not* to use an assembly language core, but I
haven't decided which assembler to use.
Tom Novelli
2006-11-03 00:25:30 UTC
Permalink
Post by M. Edward (Ed) Borasky
Yep -- RetroForth, RevaForth and FreeForth/FASM. Right now I'm leaning
towards the FreeForth/FASM, since ANS Forth compatibility isn't critical.
I'm not offended :-)
Post by M. Edward (Ed) Borasky
Ruby is mostly a hybrid of Smalltalk and Perl. It has lambdas, blocks,
closures and continuations, but not tail recursion, and there's quite a
bit of debate in the core Ruby developer community about the future of
continuations. Its connections with Lisp and Scheme are mostly through
Smalltalk and not directly from the Lisp family. See
http://borasky-research.blogspot.com/2006/10/rubyconf-2006-impending-ruby-fracture.html
I saw your wish list there... macros, tail recursion, continuations,
numeric arrays, assembler... I've been playing with the CMUCL/SBCL
Lisp compilers the past few weeks, and they do ALL that, seemingly
pretty well. Couldn't you write a Ruby-to-Lisp parser? If that
works, I think you could create a stripped-down version of SBCL, load
your parser, and save out a modest-size executable... and if it works
for Ruby, it ought to work for Python, Javascript, etc.. and they'd
all share the same garbage collector, native code compiler, etc.
M. Edward (Ed) Borasky
2006-11-03 03:27:33 UTC
Permalink
Post by Tom Novelli
I saw your wish list there... macros, tail recursion, continuations,
numeric arrays, assembler... I've been playing with the CMUCL/SBCL
Lisp compilers the past few weeks, and they do ALL that, seemingly
pretty well. Couldn't you write a Ruby-to-Lisp parser? If that
works, I think you could create a stripped-down version of SBCL, load
your parser, and save out a modest-size executable... and if it works
for Ruby, it ought to work for Python, Javascript, etc.. and they'd
all share the same garbage collector, native code compiler, etc.
Yeah ... there actually are some Ruby/Lisp bridges floating around. I
think the current Ruby interpreter stores the abstract syntax tree as an
S expression, so it might be even easier.

Is the SBCL compiler better than the CMUCL compiler, or just a more
lively project and community?
Tom Novelli
2006-11-04 16:02:10 UTC
Permalink
Post by M. Edward (Ed) Borasky
Is the SBCL compiler better than the CMUCL compiler, or just a more
lively project and community?
SBCL is a cleaner version of CMUCL. I think you can strip SBCL down
to the core, without the full 1500+ functions of Common Lisp, where as
CMUCL too more intertwined. SBCL is still "beta" but seems quite
popular already. See http://cliki.net/SBCL

Be aware that you're hearing this from a complete Lisp newbie!

- Tom
Armin Rigo
2006-11-04 11:41:04 UTC
Permalink
Hi Tom,
Post by Tom Novelli
Couldn't you write a Ruby-to-Lisp parser? If that
works, I think you could create a stripped-down version of SBCL, load
your parser, and save out a modest-size executable... and if it works
for Ruby, it ought to work for Python, Javascript, etc.. and they'd
all share the same garbage collector, native code compiler, etc.
Newsgroups like comp.lang.python are full of statements like "just
implement Python in Lisp and everything will be great". By now, you'd
think that someone would really have tried, but there is no such
implementation showing great results - even as a prototype.

It doesn't work. There are several reasons for that. One reason (try
google for more detailled answers) is that Python, Ruby, etc. are 3%
syntax, and 97% semantics defined by a large library. It's trivial to
translate the AST of these languages to S-exps, e.g. with a Ruby-to-Lisp
parser. Then you only solve 3% of the problem. The difficult and
tedious bit is to implement the semantics and library and object model,
and Lisp compilers are not going to magically make these 97%
super-efficient. (I should add that I consider the 3% the boring bit;
parser generation stuff was solved decades ago, but sadly many compiler
construction lectures still focus mostly on that.)


A bientot,

Armin
Tom Novelli
2006-11-04 15:46:30 UTC
Permalink
On 11/4/06, Armin Rigo <***@tunes.org> wrote:

Hey, Armin... just the guy we need to talk to!
Post by Armin Rigo
Newsgroups like comp.lang.python are full of statements like "just
implement Python in Lisp and everything will be great". By now, you'd
think that someone would really have tried, but there is no such
implementation showing great results - even as a prototype.
It doesn't work. There are several reasons for that. One reason (try
google for more detailled answers) is that Python, Ruby, etc. are 3%
syntax, and 97% semantics defined by a large library. It's trivial to
translate the AST of these languages to S-exps, e.g. with a Ruby-to-Lisp
parser. Then you only solve 3% of the problem. The difficult and
tedious bit is to implement the semantics and library and object model,
and Lisp compilers are not going to magically make these 97%
super-efficient. (I should add that I consider the 3% the boring bit;
parser generation stuff was solved decades ago, but sadly many compiler
construction lectures still focus mostly on that.)
I figured as much. I just noticed that Python's lexical scope rules
are incompatible with Lisp, and broken, IMHO. In LISP, I could have
defined a few nested functions, but in Python there's no way to modify
the outer function's local variables from the inner functions.

Is anyone looking at the big picture? LISP seems to have a mature,
well-designed semantic model, while these script languages just have
ad-hoc semantics cobbled together over the years, still changing every
year, and borrowing more and more from LISP. Has anyone tried to
define a variant of Python (or Ruby, ECMAscript, etc.) that's
compatible with the Lisp/Scheme model? I ask because I'm interested
in doing that, and I wouldn't want to waste my time if it's already
been done, especially if it's not promising!

I hope I'm not asking too many dumb questions... I didn't paid much
attention to these issues until a year or two ago, and I have a lot of
catching up to do. Hopefully the new TUNES CMS will help us summarize
this knowledge... and it'll look good on my resume. I'm not sure I
*want* to give up surveying for a programming career, but it would
allow me to devote more effort to this stuff.

Let's talk about Python sometime, after I've read up on PyPy etc.

- Tom
M. Edward (Ed) Borasky
2006-11-04 16:48:45 UTC
Permalink
Post by Tom Novelli
Hey, Armin... just the guy we need to talk to!
Post by Armin Rigo
Newsgroups like comp.lang.python are full of statements like "just
implement Python in Lisp and everything will be great". By now, you'd
think that someone would really have tried, but there is no such
implementation showing great results - even as a prototype.
It doesn't work. There are several reasons for that. One reason (try
google for more detailled answers) is that Python, Ruby, etc. are 3%
syntax, and 97% semantics defined by a large library. It's trivial to
translate the AST of these languages to S-exps, e.g. with a Ruby-to-Lisp
parser. Then you only solve 3% of the problem. The difficult and
tedious bit is to implement the semantics and library and object model,
and Lisp compilers are not going to magically make these 97%
super-efficient. (I should add that I consider the 3% the boring bit;
parser generation stuff was solved decades ago, but sadly many compiler
construction lectures still focus mostly on that.)
I figured as much. I just noticed that Python's lexical scope rules
are incompatible with Lisp, and broken, IMHO. In LISP, I could have
defined a few nested functions, but in Python there's no way to modify
the outer function's local variables from the inner functions.
Is anyone looking at the big picture? LISP seems to have a mature,
well-designed semantic model, while these script languages just have
ad-hoc semantics cobbled together over the years, still changing every
year, and borrowing more and more from LISP. Has anyone tried to
define a variant of Python (or Ruby, ECMAscript, etc.) that's
compatible with the Lisp/Scheme model? I ask because I'm interested
in doing that, and I wouldn't want to waste my time if it's already
been done, especially if it's not promising!
Programming languages, their creators and their users co-evolve. It's
that simple. There is no "big picture" -- it's a complex adaptive
system, as defined by John Holland.

The languages that have been around a long time -- Fortran, Lisp and
Algol, in its current dialects of C, C++, C# and Java, are actually
recognizable to people who learned them long ago. I can still (probably)
program in Fortran IV on a modern compiler, though I think Fortran II's
"READ INPUT TAPE" and "WRITE OUTPUT TAPE" are gone from most current
dialects. And I can still (probably) write LISP 1.5 code in any of the
Common Lisp systems out there.

Now, given co-evolution and competition, you end up having niches. The
so-called scripting languages evolved as a replacement for csh/ksh
driving sed and awk. So they pretty much all do regular expressions,
pipes, and, as Armin notes, wrap humongous collections of libraries.

And they all have their own unique well-defined C language interfaces.
The SWIG project is an attempt to collect all of this interface
knowledge into a package, and a noble effort, and eminently useful. That
would be where I'd start -- look at the semantics SWIG has to deal with
in wrapping nearly all of C and C++ into something like a dozen
scripting languages, including Clisp and a couple of Scheme
implementations, Perl, Python and Ruby, PHP4, and Lua, Pike and OCaml.

When you come right down to it, there are really only half a dozen or so
truly unique programming languages. My list is *macro* assembler,
Fortran, Lisp, Forth, APL and Smalltalk. I've gotten into arguments
about whether Algol should be in there or not, but so far, nobody has
convinced me that it represents anything more than a merging of concepts
from Fortran and Lisp. And perhaps macro assembler and Forth don't need
to both be there.
cstb
2006-11-05 09:27:20 UTC
Permalink
At 08:48 AM 11/4/2006, M. Edward (Ed) Borasky wrote:
...
Post by M. Edward (Ed) Borasky
When you come right down to it, there are really only half a dozen or so
truly unique programming languages. My list is *macro* assembler,
Fortran, Lisp, Forth, APL and Smalltalk. I've gotten into arguments
about whether Algol should be in there or not, but so far, nobody has
convinced me that it represents anything more than a merging of concepts
from Fortran and Lisp. And perhaps macro assembler and Forth don't need
to both be there.
Not a bad list, old timer.

Seems to me that a "merging of concepts from Fortran and Lisp"
was pretty significant at the time, and this carries through to
the present. Block structure was unique as well, not to mention
'thunking' (a bad idea, but unique). So I'd include Algol.

Forth steals the 'macro-like' bits, sure, but as a threaded code
interpreter, I'd say it was/is representing something unique.

I'd add a few more that you seem to have hopped over:

Cobol
Basic
Simula
C
Snobol
Prolog

I'd pile all the shell/scripty things together in a lump
and include it as Perl, reluctantly.

Seems the last decade or two present little to qualify with.
If you consider Linda a language, rather than just an architecture,
then it probably belongs on the list. Erlang, maybe. Eiffel, maybe.

Can't think of anything else with defendable credentials.
Someone ought to put up a fight for ADA, I suppose. But not me ;-)

Cheers,

-cstb
Tom Novelli
2006-11-07 00:55:41 UTC
Permalink
Post by M. Edward (Ed) Borasky
Programming languages, their creators and their users co-evolve. It's
that simple. There is no "big picture" -- it's a complex adaptive
system, as defined by John Holland.
[.....]
Post by M. Edward (Ed) Borasky
When you come right down to it, there are really only half a dozen or so
truly unique programming languages.
[.....]

I'd like to address this one point, because it's a central theme for
the TUNES project... It hasn't been dealt with on the website but it
will be.. think of this as a rough draft:

Computers *could* be a lot more useful, without all the headaches
about incompatibility and such, if only someone had the foresight to
figure out how to please everybody. Problem is, everyone who really
studies computers gets distracted by minor details and goes off on
some tangent, only adding to the confusion. Maybe it's "The Geek
Syndrome" or maybe it's just the nature of computers. Either way,
it'll take a concerted effort to get over this hurdle.

I realize there's a learning process... every programmer goes through
it, and we're still in it as a whole. We want to speed up that
collective learning process. So for example when someone says "I have
a great new idea" we can say "It's not new and it's not so great
either" -- and this is where we have to do better, by saying it in a
way they can understand, without belittling them. Experimentation is
great, and we don't want to discourage it, but we're sick of all the
hype surrounding every nifty little toy OS/Library/Language that comes
along.

BTW, I'm not passing judgement on Ruby.. I actually looked into it
some more over the weekend. It seems to pull together the important
parts of 4 or 5 languages on your list... which raises the question--
are they really unique?

I'm on my way to a jam session... there'll be a dozen people playing a
half-dozen different instruments, variations on two basic mechanisms
(strings and winds), all tuned the same, all reading the same notes...
and we can play tunes written over 400 years ago. If only my life as
a programmer were as easy as my life as a musician!

- Tom

M. Edward (Ed) Borasky
2006-11-05 10:57:26 UTC
Permalink
Seems to me that a "merging of concepts from Fortran and Lisp"
was pretty significant at the time, and this carries through to
the present. Block structure was unique as well, not to mention
'thunking' (a bad idea, but unique). So I'd include Algol.
Well ... Algol was certainly on the list back in 1958 when it first came
on the scene, but the computer scientists rather quickly realized it was
theoretically equivalent to the lambda calculus, which sort of made it a
dialect of Lisp. I consider Algol the merging of Fortran and Lisp.
Cobol
I'd actually think SNOBOL or COMIT would be more unique than COBOL.
Basic
BASIC was essentially a dialect of Fortran, and its only other claim to
fame -- being an interactive language -- got beaten out by the much more
elegant APL.
Simula
Well ... really a dialect of Algol too.
C
Another dialect of Algol.
Snobol
Yeah ... this probably belongs in the list.
Prolog
Well ... really a dialect of Lisp ... see "miniKanren" for an
implementation in Scheme.
I'd pile all the shell/scripty things together in a lump
and include it as Perl, reluctantly.
Algol with regular expressions :). Another reason to have either SNOBOL
or COMIT in the list.
Seems the last decade or two present little to qualify with.
If you consider Linda a language, rather than just an architecture,
then it probably belongs on the list.
Now that's an interesting point. Even today, we do concurrency mostly at
the OS level and wrap it in libraries called by non-concurrent
languages. All the wonderful theory of concurrency (Petri nets, CSP and
CCS, process algebras, etc.) only showed up in one or two languages --
Occam and Concurrent Pascal are the only two I can think of right now --
but hasn't really been formally part of any mainstream language.

The current crop of dual and quad core processors will no doubt spawn
(pun intended) a few new "languages", but I'm sure the technology to get
us back on serial machines will once again appear and we won't have to
do the hard work of reasoning about concurrency.
Erlang, maybe.
Lisp ... so is Haskell and ML. :)
Eiffel, maybe.
I don't know anything about Eiffel.
Can't think of anything else with defendable credentials.
Someone ought to put up a fight for ADA, I suppose. But not me ;-)
PL/I, which is a dialect of Algol, but with syntax from another dialect
of Algol, Pascal.
Armin Rigo
2006-11-02 16:35:19 UTC
Permalink
Hi all,
Post by Tom Novelli
I would encourage you to go ahead with your project... If
you succeed, it'll make Ruby a more viable language. Elegant,
high-level, high-performance implementations (for any language) are
hard to find.
I suppose that now is a good time to drop a link to our project, PyPy.
That's a shameless plug - I don't pretend that this is tightly related
to Tunes, but still, it might be a concrete first step in a direction
that some people around here might find interesting.

http://codespeak.net/pypy

PyPy gives a framework to write in Python high-level language
interpreters, and compile them in a flexible way to efficient, JIT-ing
virtual machines that run on top of C/Posix, .NET, Java, Squeak, etc.
So far we have a fully compliant Python front-end, and people are
working on Prolog and JavaScript ones. All our front-ends are written
as "naive" interpreters. We have C/Posix and .NET backends, with Java
and Squeak being at the draft stage. The automatic JIT complier
generation is work in progress.

Some people considered working on a Ruby front-end, but it seems that
there are already 3 projects out there trying to write Ruby-in-Ruby.
One of these 3 projects can probably be interfaced with PyPy relatively
easily and benefit from the hard work we've already put into it (e.g.
with a straighforward translator: input the subset of Ruby that the
Ruby-in-Ruby project is written in, and output Python code that the PyPy
framework can work with).


A bientot,

Armin
M. Edward (Ed) Borasky
2006-11-03 03:43:48 UTC
Permalink
Post by Armin Rigo
I suppose that now is a good time to drop a link to our project, PyPy.
That's a shameless plug - I don't pretend that this is tightly related
to Tunes, but still, it might be a concrete first step in a direction
that some people around here might find interesting.
http://codespeak.net/pypy
PyPy gives a framework to write in Python high-level language
interpreters, and compile them in a flexible way to efficient, JIT-ing
virtual machines that run on top of C/Posix, .NET, Java, Squeak, etc.
So far we have a fully compliant Python front-end, and people are
working on Prolog and JavaScript ones. All our front-ends are written
as "naive" interpreters. We have C/Posix and .NET backends, with Java
and Squeak being at the draft stage. The automatic JIT complier
generation is work in progress.
Sadly, I know zero Python. I'm actually a Perl programmer who is
migrating to Ruby.
Post by Armin Rigo
Some people considered working on a Ruby front-end, but it seems that
there are already 3 projects out there trying to write Ruby-in-Ruby.
I don't think there are actually three *viable* projects -- it's more
like one project with a couple of different facets. For the current Ruby
language, there's one project (at Sun) to run Ruby on the Java VM, at
Microsoft there are two projects integrating Ruby into .NET/CLR, and the
other one -- "Ruby in Ruby" is Rubinius.

The future Ruby -- beyond 1.8 -- is only one project. The virtual
machine is called YARV.

Cardinal (Ruby on the Parrot Virtual Machine) and Carbone (Ruby on a
vmgen virtual machine) both appear to be dead.
Post by Armin Rigo
One of these 3 projects can probably be interfaced with PyPy relatively
easily and benefit from the hard work we've already put into it (e.g.
with a straighforward translator: input the subset of Ruby that the
Ruby-in-Ruby project is written in, and output Python code that the PyPy
framework can work with).
Which project is this? Rubinius?
Armin Rigo
2006-11-06 14:48:53 UTC
Permalink
Hi,
Post by M. Edward (Ed) Borasky
I don't think there are actually three *viable* projects -- it's more
like one project with a couple of different facets.
There is MetaRuby, Rubinius, and another single-person project which may
or may not have a name.
Post by M. Edward (Ed) Borasky
Post by Armin Rigo
One of these 3 projects can probably be interfaced with PyPy relatively
easily and benefit from the hard work we've already put into it (e.g.
with a straighforward translator: input the subset of Ruby that the
Ruby-in-Ruby project is written in, and output Python code that the PyPy
framework can work with).
Which project is this? Rubinius?
We met the MetaRuby guys at OOPSLA; that's the one I had in mind. I
suspect Rubinius could do that too, but I don't know it at all.


A bientot,

Armin
M. Edward (Ed) Borasky
2006-11-06 15:18:17 UTC
Permalink
Post by Armin Rigo
Hi,
Post by M. Edward (Ed) Borasky
I don't think there are actually three *viable* projects -- it's more
like one project with a couple of different facets.
There is MetaRuby, Rubinius, and another single-person project which may
or may not have a name.
Post by M. Edward (Ed) Borasky
Post by Armin Rigo
One of these 3 projects can probably be interfaced with PyPy relatively
easily and benefit from the hard work we've already put into it (e.g.
with a straighforward translator: input the subset of Ruby that the
Ruby-in-Ruby project is written in, and output Python code that the PyPy
framework can work with).
Which project is this? Rubinius?
We met the MetaRuby guys at OOPSLA; that's the one I had in mind. I
suspect Rubinius could do that too, but I don't know it at all.
A bientot,
Armin
Ah ... even though it was in my home town, I missed OOPSLA. I did get to
the functional language conference for the Scheme and Erlang sessions in
September, but I decided to go to RubyConf rather than OOPSLA.

Let me hunt down MetaRuby :)
M. Edward (Ed) Borasky
2006-11-06 15:24:26 UTC
Permalink
Post by M. Edward (Ed) Borasky
Post by Armin Rigo
Hi,
Post by M. Edward (Ed) Borasky
I don't think there are actually three *viable* projects -- it's more
like one project with a couple of different facets.
There is MetaRuby, Rubinius, and another single-person project which may
or may not have a name.
Post by M. Edward (Ed) Borasky
Post by Armin Rigo
One of these 3 projects can probably be interfaced with PyPy relatively
easily and benefit from the hard work we've already put into it (e.g.
with a straighforward translator: input the subset of Ruby that the
Ruby-in-Ruby project is written in, and output Python code that the PyPy
framework can work with).
Which project is this? Rubinius?
We met the MetaRuby guys at OOPSLA; that's the one I had in mind. I
suspect Rubinius could do that too, but I don't know it at all.
A bientot,
Armin
Ah ... even though it was in my home town, I missed OOPSLA. I did get to
the functional language conference for the Scheme and Erlang sessions in
September, but I decided to go to RubyConf rather than OOPSLA.
Let me hunt down MetaRuby :)
Actually it's two facets of the same project, now that I look at it.
Rubinius is by Evan Phoenix, who's part of the same group as Ryan Davis
and some of the other Rubyists in the Seattle area.

So ... to quote Dijkstra, "The purpose of thinking is to reduce the
amount of detailed reasoning required to a doable amount." How does PyPy
fit into that? What "detailed reasoning" can be dispensed with?
Loading...