|
SISC
|
Kawa
|
JScheme
|
Bigloo6
|
|
SISC
|
Kawa
|
JScheme
|
Bigloo6
|
R5RS
|
X
|
X2
|
|
X5
|
Number Tower
|
---
|
---
|
---
|
---
|
R4RS
|
X
|
X
|
X3
|
X
|
Fixed Integers
|
X
|
X
|
X
|
X
|
Proper Tail Recursion
|
X
|
X1
|
X
|
X5
|
Arbitrary Precision
Integers
|
X
|
X
|
|
|
Escaping Continuations
|
X
|
X
|
X
|
X
|
Float Precision
|
Unlimited4
|
?
|
64bit
|
64bit
|
One-shot Continuations
|
X
|
|
|
|
Ratios
|
X
|
X
|
|
|
Multi-shot Continuations
|
X
|
|
|
|
Complex Numbers
|
X
|
X
|
|
|
Compiling Support
|
|
X
|
X
|
X
|
|
|
|
|
|
- Not enabled by default
- With
restrictions
- With escaping continuations only, and only
immutable strings.
- SISC provides three numeric libraries,
one of which uses unlimited precision floating
point math, one of which supports 64 and another
32 bit floating point.
- Bigloo claims to be almost R5RS compliant, with the exception
of proper tail recursion, not present with the JVM backend.
- Targetting the JVM backend.
Gabriel Benchmark Results
The following benchmarks were performed
on an unloaded 700mhz Pentium III. The operating
system was RedHat Linux 6.2, and the virtual
machine was Sun's JDK v1.4. Debugging was off and
optimization was on for the compiles. The times are recorded
from the start of the benchmark (calling the benchmark's
entry point) to the point a value is returned. In particular,
time to load and compile the benchmark is not
included.
SISC was compiled using the 64 bit floating
point numeric library. Also, while SISC
supports dynamic-wind, the support code is not loaded
unless a program uses dynamic-wind. This has a marginal effect
on the performance of call/cc.
The systems tested were:
- JScheme 4.3.2
- Kawa 1.6.98
- Bigloo 2.5a (targetting the JVM)
- SISC 1.4.0
(I) indicates the Scheme
System is operating as an Interpreter. In
Kawa's case, most code is actually compiled on-the-fly,
so Interpreter may not be a completely accurate word.
(C) indicates the Scheme System is
operating as a Compiler.
(TC) indicates that support for
proper tail recursion (Tail Calls) was enabled.
(std) Bigloo was run with hygienic macro support, bytecode
purification, and call/cc support.
(opt) Bigloo was run with none of the std options, and
with -Obench
Benchmark times
| Benchmark |
SISC |
JScheme (I) |
JScheme (C) |
Kawa (I) |
Kawa (C) |
Kawa (I/TC) |
Kawa (C/TC) |
Bigloo (std) |
Bigloo (opt) |
| cpstak |
318 |
1017 |
767 |
Failed |
Failed |
435 |
160 |
Failed |
Failed |
| ctak |
653 |
21484 |
12508 |
10254 |
9380 |
18221 |
17631 |
10765 |
10095 |
| dderiv |
1325 |
1818 |
986 |
346 |
211 |
872 |
584 |
197 |
36 |
| deriv |
1248 |
1163 |
547 |
307 |
160 |
764 |
Failed |
181 |
32 |
| destruct |
1409 |
3397 |
4304 |
1086 |
263 |
1109 |
277 |
551 |
127 |
| div-iter |
532 |
1411 |
1448 |
84 |
37 |
89 |
40 |
48 |
14 |
| div-rec |
533 |
1304 |
636 |
86 |
41 |
406 |
205 |
34 |
14 |
| fft |
319 |
Failed |
Failed |
308 |
75 |
Failed |
Failed |
140 |
13 |
| fread |
68 |
16 |
154 |
33 |
35 |
34 |
37 |
78 |
20 |
| fprint |
8 |
5 |
5 |
21 |
16 |
Failed |
Failed |
51 |
49 |
| puzzle |
79 |
164 |
224 |
69 |
10 |
Failed |
Failed |
46 |
4 |
| tak |
306 |
694 |
234 |
132 |
15 |
295 |
65 |
59 |
2 |
| takl |
2976 |
6306 |
1745 |
631 |
33 |
2192 |
339 |
38 |
9 |
| takr |
351 |
701 |
764 |
369 |
46 |
728 |
218 |
101 |
14 |
Relative performance
This table indicates the relative performance between systems, with positive
values showing SISC is n percent faster on that benchmark.
| Benchmark |
JScheme (I) |
JScheme (C) |
Kawa (I) |
Kawa (C) |
Kawa (I/TC) |
Kawa (C/TC) |
Bigloo (std) |
Bigloo (opt) |
| cpstak |
219.8 |
141.2 |
Failed |
Failed |
36.8 |
-98.8 |
Failed |
Failed |
| ctak |
3190.0 |
1815.5 |
1470.3 |
1336.4 |
2690.4 |
2600.0 |
1548.5 |
1445.9 |
| dderiv |
37.2 |
-34.4 |
-282.9 |
-528.0 |
-51.9 |
-126.9 |
-572.6 |
-3580.6 |
| deriv |
-7.3 |
-128.2 |
-306.5 |
-680.0 |
-63.4 |
|
-589.5 |
-3800.0 |
| destruct |
141.1 |
205.5 |
-29.7 |
-435.7 |
-27.1 |
-408.7 |
-155.7 |
-1009.4 |
| div-iter |
165.2 |
172.2 |
-533.3 |
-1337.8 |
-497.8 |
-1230.0 |
-1008.3 |
-3700.0 |
| div-rec |
144.7 |
19.3 |
-519.8 |
-1200.0 |
-31.3 |
-160.0 |
-1467.6 |
-3707.1 |
| fft |
Failed |
Failed |
-3.6 |
-325.3 |
Failed |
Failed |
-127.9 |
-2353.8 |
| fread |
-325.0 |
126.5 |
-106.1 |
-94.3 |
-100.0 |
-83.8 |
14.7 |
-240.0 |
| fprint |
-60.0 |
-60.0 |
162.5 |
100.0 |
Failed |
Failed |
537.5 |
512.5 |
| puzzle |
107.6 |
183.5 |
-14.5 |
-690.0 |
Failed |
Failed |
-71.7 |
-1875.0 |
| tak |
126.8 |
-30.8 |
-131.8 |
-1940.0 |
-3.7 |
-370.8 |
-418.6 |
-15200.0 |
| takl |
111.9 |
-70.5 |
-371.6 |
-8918.2 |
-35.8 |
-777.9 |
-7731.6 |
-32966.7 |
| takr |
99.7 |
117.7 |
5.1 |
-663.0 |
107.4 |
-61.0 |
-247.5 |
-2407.1 |
Graphs
Summary
Optimized Bigloo comes up on top in most tests.
SISC outperforms interpreted JScheme in almost all tests,
often by a substantial margin. SISC is more than thirteen
times faster than any other tested system in running CTAK,
which tests performance of continuation capture and invocation.
The exceptional performance in CTAK allows SISC to nudge out even
Bigloo in the overall times, despite not having the 'advantage' of one or
more failed marks contributing nothing to the total time.
Also worth noting is that the Gabriel benchmarks are "numerically biased",
stressing different aspects of an implementation but largely driving a
computational process.
Many real world apps based on non-computational loads (text-processing,
networking, data-mining, etc) are probably underrepresented.
Remember, though, that these are only benchmarks. Scheme systems
should be evaluated primarily for suitability to your given objective and
performance for the code you wish to execute.
|
