Octave-2-Matlab and back (2)

Like I mentioned earlier here. I finally managed to put together a AST-visitor pattern that
loads the user-script parses it, and gives it to the conversion tree-walker code. This implements
the changes mentioned in the oct2mat package (find it here) by Paul Kienzle .

The email announcing this code and patch was posted

Hello there,

I have a oct2mat like the one Paul Kienzle mentions in his
README file based on the AST walker. Utilizing Octave's
parse tree to do the conversion is very helpful and accurate
to a larger degree. Its still a work in progress (my disclaimer).

Large parts of the code are directly derived from Octave
source written by JWE.

This script is essentially to help deploy code to your colleagues
who want to use it on Matlab systems for whatever reason. Primary
development is still expected to be based on an Octave based
environment.

You can invoke the script as,

octave:1> oct2mat("full/path/my_cool_octave_function.m",1)
octave:2> oct2mat("full/path/my_cool_octave_script.m")

and the code is output to stdout, where from you can use some shell
scripts to redirect the output to another matlab directory.

I have attached the files. Im waiting for JWE to accept some
global export patches to access some parser information. Once that is
done, I plan to add it to the extras branch of octave-forge.

This code works only on the cvs version of Octave, as a external add-on.

Limitations include

1. Can treat only function files or script files not both.
2. Pollutes your command history, as it pipes out to octave_stdout.
3. Cannot handle a script file with function definitions
4. Doesnot write to a separate directory or file, just all output on
stdout. So you need to create a driver program.
5. Not sure if it works correctly for all cases.

Best,
-Muthu


The Code / Tree Walker

I also made patches against existing code from Octave tree (maybe obsolete right now), so
as to help integrate my tree walker into the existing visitor.

Index: pt-pr-code.cc
===================================================================
RCS file: /cvs/octave/src/pt-pr-code.cc,v
retrieving revision 1.54
diff -u -p -r1.54 pt-pr-code.cc
--- pt-pr-code.cc 12 Oct 2007 21:27:33 -0000 1.54
+++ pt-pr-code.cc 10 Jan 2008 05:56:20 -0000
@@ -35,6 +35,240 @@ along with Octave; see the file COPYING.
#include "pr-output.h"
#include "pt-all.h"

+
+static std::string
+matlab_unary_op_as_string (octave_value::unary_op op)
+{
+ std::string retval;
+
+ switch (op)
+ {
+ case octave_value::op_not:
+ retval = "~";
+ break;
+
+ case octave_value::op_uplus:
+ retval = "+";
+ break;
+
+ case octave_value::op_uminus:
+ retval = "-";
+ break;
+
+ case octave_value::op_transpose:
+ retval = ".'";
+ break;
+
+ case octave_value::op_hermitian:
+ retval = "'";
+ break;
+
+ case octave_value::op_incr:
+ retval = "++";
+ break;
+
+ case octave_value::op_decr:
+ retval = "--";
+ break;
+
+ default:
+ retval = "";
+ }
+
+ return retval;
+}
+
+static std::string
+matlab_binary_op_as_string (octave_value::binary_op op)
+{
+ std::string retval;
+
+ switch (op)
+ {
+ case octave_value::op_add:
+ retval = "+";
+ break;
+
+ case octave_value::op_sub:
+ retval = "-";
+ break;
+
+ case octave_value::op_mul:
+ retval = "*";
+ break;
+
+ case octave_value::op_div:
+ retval = "/";
+ break;
+
+ case octave_value::op_pow:
+ retval = "^";
+ break;
+
+ case octave_value::op_ldiv:
+ retval = "\\";
+ break;
+
+ case octave_value::op_lshift:
+ retval = "<<"; + break; + + case octave_value::op_rshift: + retval = ">>";
+ break;
+
+ case octave_value::op_lt:
+ retval = "<"; + break; + + case octave_value::op_le: + retval = "<="; + break; + + case octave_value::op_eq: + retval = "=="; + break; + + case octave_value::op_ge: + retval = ">=";
+ break;
+
+ case octave_value::op_gt:
+ retval = ">";
+ break;
+
+ case octave_value::op_ne:
+ retval = "~=";
+ break;
+
+ case octave_value::op_el_mul:
+ retval = ".*";
+ break;
+
+ case octave_value::op_el_div:
+ retval = "./";
+ break;
+
+ case octave_value::op_el_pow:
+ retval = ".^";
+ break;
+
+ case octave_value::op_el_ldiv:
+ retval = ".\\";
+ break;
+
+ case octave_value::op_el_and:
+ retval = "&";
+ break;
+
+ case octave_value::op_el_or:
+ retval = "|";
+ break;
+
+ case octave_value::op_struct_ref:
+ retval = ".";
+ break;
+
+ default:
+ retval = "";
+ }
+
+ return retval;
+}
+
+static void
+do_oct2mat_oper(std::ostream &os, tree_binary_expression &expr)
+{
+ std::string oprval="";
+
+ // this converts code into whatever form
+ // oct2mat requires.
+ if ( expr.is_boolean_expression() )
+ {
+ const tree_boolean_expression & tbr
+ = dynamic_cast (expr);
+
+ oprval="";
+
+ switch ( tbr.op_type() )
+ {
+ case tree_boolean_expression::bool_and:
+ oprval = "&";
+ break;
+
+ case tree_boolean_expression::bool_or:
+ oprval = "|";
+ break;
+
+ defualt:
+ oprval = "";
+ }
+ }
+ else
+ {
+ oprval=matlab_binary_op_as_string(expr.op_type());
+ }
+
+ os << " " << oprval="">";
+
+ oprval=matlab_unary_op_as_string(expr.op_type());
+ os << " " <<>accept(omat);
+ buf_str = buf.str();
+ os << idx="0;">= 0);
+
+ s << ""; + if (printing_newlines) + { + if (beginning_of_line) + { + s << i =" 0;" beginning_of_line =" false;">accept (*this);

- os << " " << op2 =" expr.rhs">accept (*this);

@@ -237,7 +474,10 @@ tree_print_code::visit_simple_for_comman

indent ();

- os << "endfor"; + if ( convert_to_matlab ) + os << "end"; + else + os << "endfor"; } void @@ -280,7 +520,10 @@ tree_print_code::visit_complex_for_comma indent (); - os << "endfor"; + if ( convert_to_matlab ) + os << "end"; + else + os << "endfor"; } void @@ -401,7 +644,10 @@ tree_print_code::visit_octave_user_funct indent (); - os << "endfunction"; + if ( convert_to_matlab ) + os << "%% endfunction"; + else + os << "endfunction"; newline (); } @@ -414,7 +660,17 @@ tree_print_code::visit_identifier (tree_ print_parens (id, "("); std::string nm = id.name (); - os << (nm.empty () ? std::string ("(empty)") : nm); + + if ( convert_to_matlab ) + { + std::map::iterator iter= identifier_lut.find(nm);
+ if ( iter != identifier_lut.end() )
+ os <<>second ;
+ else
+ os << (nm.empty () ? std::string ("(empty)") : nm); + } + else + os << (nm.empty () ? std::string ("(empty)") : nm); print_parens (id, ")"); } @@ -459,7 +715,10 @@ tree_print_code::visit_if_command (tree_ indent (); - os << "endif"; + if ( convert_to_matlab ) + os << "end"; + else + os << "endif"; } void @@ -693,11 +952,75 @@ tree_print_code::visit_no_op_command (tr void tree_print_code::visit_constant (tree_constant& val) { + std::string buf_str; + std::ostringstream buf; + int idx = 0; + octave_value sval=val.rvalue(); indent (); print_parens (val, "("); - val.print_raw (os, true, print_original_text); + + if ( convert_to_matlab ) + { + if ( sval.is_string() ) + { + buf_str=sval.string_value(); + // see if you need to escape the string stxt. + os << "'"; + for( idx=0; idx < buf_str =" buf.str();" idx="0;">accept (*this);

- os << e =" expr.operand" s =" indent_to_string">accept (*this);
+ // print a commented code.
+ if ( convert_to_matlab )
+ {
+ std::string ind_prfx = indent_to_string( );
+ do_print_commented_code(unwind_protect_code,os,ind_prfx);
+ }
+ else
+ unwind_protect_code->accept (*this);

decrement_indent_level ();
}
@@ -1022,7 +1374,10 @@ tree_print_code::visit_unwind_protect_co

indent ();

- os << "unwind_protect_cleanup"; + if ( convert_to_matlab ) + os << "%%unwind_protect_cleanup"; + else + os << "unwind_protect_cleanup"; newline (); @@ -1032,7 +1387,13 @@ tree_print_code::visit_unwind_protect_co { increment_indent_level (); - cleanup_code->accept (*this);
+ if ( convert_to_matlab )
+ {
+ std::string ind_prfx = indent_to_string( );
+ do_print_commented_code(cleanup_code,os,ind_prfx);
+ }
+ else
+ cleanup_code->accept (*this);

decrement_indent_level ();
}
@@ -1041,7 +1402,10 @@ tree_print_code::visit_unwind_protect_co

indent ();

- os << "end_unwind_protect"; + if ( convert_to_matlab ) + os << "%%end_unwind_protect"; + else + os << "end_unwind_protect"; } void @@ -1075,7 +1439,10 @@ tree_print_code::visit_while_command (tr indent (); - os << "endwhile"; + if ( convert_to_matlab ) + os << "end"; + else + os << "endwhile"; } void @@ -1192,7 +1559,10 @@ tree_print_code::print_comment_elt (cons if (c == '\n') { if (prev_char_was_newline) - os << "##"; + if ( convert_to_matlab ) + os << "%" ; + else + os << "##"; newline (); @@ -1206,7 +1576,11 @@ tree_print_code::print_comment_elt (cons indent (); - os << "##"; + + if ( convert_to_matlab ) + os << "%%"; + else + os << "##"; if (! (isspace (c) || c == '!')) os << " "; Index: pt-pr-code.h =================================================================== RCS file: /cvs/octave/src/pt-pr-code.h,v retrieving revision 1.36 diff -u -p -r1.36 pt-pr-code.h --- pt-pr-code.h 12 Oct 2007 21:27:33 -0000 1.36 +++ pt-pr-code.h 10 Jan 2008 05:56:20 -0000 @@ -26,6 +26,7 @@ along with Octave; see the file COPYING. #include
#include
+#include


I still have bugs in the driver routine, but hey it works!.
-Muthu

Octave Projects for the fearless

Ok, here goes. Theres a list of projects I just racked my brains to think whats next to keep ourselves excited about Octave. Specifically the interpreter. Of late, I've been mindlessly compiling Octave, and trying to mix and match this with several packages. As my mind puts the mix-n-match, pick-n-choose mental workings on the back seat, I prefer to let the harder problems emerge; this is simply improving the interpreter. Specifically,

1. provide hooks for profiler, and debugger as AST evaluators.
2. creating a JIT compiler for Octave, to quicken up the loopy-loop constructs.
3. all types of safe optimizations in the compiler for JIT.

Clearly all of these are beyond the reach of a novice untrained in the rigours of compiler design. Even then, moving this knowledge to Octave code-base is still a challenge.

With this prelude, I post the set of 5 projects I feel are challenging to anyone willing to work on Octave interpreter.

1. Octave Startup: study the various hooks and such for Octave startup.
2. Octave function: modify the Octave startup to not load any of the standard m-files, and just track the read-eval-print loop of a simple m-file function. Trace the parse tree, its loading into interpreter data structure (e.g, symbol table) and its evaluation.
3. Do #2 for a single for-loop.
4. Study python-byte compiling from AST to the byte code, and byte code interpreter. Look at opimizations.
   
5. Add the '?' C-language ternary operator, by modifying the Octave parser and interpreter.

I think this will set the juices flowing and create a potential test suite to check your skills, while you try to improve the Octave interpreter.

-Muthu
PS: Use doxygen with the Octave CVS in the source browser mode. It helps in learning the Octave sources, with a hyperlinked document from doxygen.

Octave & Set theory

Octave can handle first-order sets, by which I mean, all sets excluding the 'x' (cross, cartesian) operators. This means, we cannot treat power sets properly.

The various set operators and functions that implement them are:

Operation	Function
set definition	create_set
Union	union
Subset	ismember
X’s complement w.r.t Y (Y-X )	complement(X,Y)
X-Y also written (X\Y)	setdiff(X,Y)
(A-B) U (B-A)	setxor(A,B)

Some interesting features to add, would be the capacity to deal with ordered pairs, i.e the same cross product we discussed earlier. We must also be able to define functions on sets, that do operations on such sets; Also we could have subset function return proper & improper subset differences.

I would also suggest generators, for finite sets, and ability to deal with groups. But then we are moving outside the scope of the set theory; this completely limits moves us to having a group theory implementation in Octave, instead of 'solving' this problem. Maybe I dont understand this very well, anyway. Cardinality operator is just the length() function. I also dont write
here the aim of the function 'intersect' that is more like a generic CS problem, as its not a
defined set theoretic operator.

Anyway, trying to force all of mathematics down into a single linear algebra package, seems like masochism.

Octave-Lint (2)

Much as I'd like to be informal, being private remains a greater love. Thinking about the previous post on Octave-Lint over lunch, a few more useful features suggested themselves. In the spirit of a note-to-self I document them here. I know looking into places like Pychecker or Pylint can save
us time.

Things that make syntactic sense but no semantic sense, and pass the parser, but which Lint must trap.

1. x=x+ stdfunc(arg); %here if the Lint knows the standard Octave functions it has a clue to the return value and can do a little bit of type-checking analysis to figure out warnings and such. Eg: x = x + strlen('hello world') ; makes sense, but on the other hand, y = y + getenv("HOME") would probably not!, and z = strcat(system('ls'),'files') definitely doesn't make any sense. Lint knows system() always returns a single integer as its first value, and that is incompatible with strcat(). Such analysis is not performed statically, and one can discover such problems only at runtime; expensive, in my opinion.
2. Indexing over-flow / under-flow options. In modern programming languages (Java, Python?) indexing over-flow under-flow can be flagged at runtime exceptions, or even better statically discovered from code. Now for Octave, matrices & vectors are indexed starting at 1, which lets use compute and flag array/vector indices that seem to underflow. In this type of analysis the direct algorithm would be to pick & flag usage such as array(index) | index <= 0 or index is non-integral. For directly -ve indices this is uncommon; we may catch a few 0-index usages, and sometimes usage of fractional indices. The key problem remains to identify computed indices that may underflow or overflow. Overflow is also another important problem for most programs, where arrays tend to be pre-allocated, and not x=[x; new_row]; style.
3. Matrix dimension conformity in cases where we do products. This is a minefield I step into every now and then. Multiplying 2 matrices A_mxn * B_nxl = C_mxl. This conformity rule of Matrix multiplication must be enforced to flag errors at compile time when we have an idea of the program dimensions. This will be tricky to implement.
4. Misspelt variables can be suggested through looking up the symbol table accumulated during the parse-tree walking phase. String edit distance algorithms will be useful here.
   
5. Redefined / modified pre-defined constants can be flagged as errors/warnings. Candidates like i, e, eps, pi, are immutable for me. Typically many programmers write for i=1:10; end and tend to use loop index after the loop, which can be 'caught' by Lint.

Essentially, learned behaviour of a experienced programmer can be taught to the machine, as rules specified by lint, and we could improve the range of tools available for users to fallback on.

I have a post soon on Fortran & Octave, working with LAPACK and known design issues in such an interface.

Cheers,
Muthu

Missing Link: Profiling-AST for Octave

The Octave profiler was developed to provide

1. Flat-profiler with average function performance.
2. Profile anonymous and complete functions.
   
3. Call-Graph profiler to show caller-callee statistics.

I have implemented a rudimentary "flat-profiler" that can just give
statistics of the cpu-time of the functions (only), self & total average
runtimes, and frequency of call to each function.

This code is a C++ rewrite of the Ruby's flat-profiler from
$(Ruby-Install-Path)/lib/profiler.rb authored by 'Matz' (Yukihiro
Matsumoto).

Sample output of profile command testcase 'testprofile.m' (attached)
looks like:

 %   cumulative   self              self     total
time   seconds   seconds    calls  ms/call  ms/call  name
 2.41     0.00      0.00        4     0.06     0.06  anonymous1
 2.57     0.00      0.00        4     0.06     0.06  anonymous2
 0.00     0.00      0.00        1     0.00     9.71  #toplevel
ans = info


I have also added some elementary support for 'event-based' profiler ,
from the Octave side, that notifies the routines for 'calls', and
'returns' from functions. Exception handling is not implemented.
The API support touches the 4 files mentioned in the previous post.

>From this point, a simple call-graph profiler can also be implemented
with the present infrastructure. The python profiler ('import profiler')
does something like this, and serves as a good template.

[Profiler API Support]
Many design decisions remain to be made, which I must leave it to better
experienced. I would personally prefer a 'event-based' profiler API like
the JAVA's JVMPI (JVM Profiler Interface) which lets people build/use a
sophisticated or simple profiler based on the application.
Please see JVM-PI design at
http://java.sun.com/j2se/1.4.2/docs/guide/jvmpi/jvmpi.html.


There are following good points about the code,

1. It recognizes anonymous functions
2. Does CPU times with least overhead.
3. Can write a basic call-graph profiler
4. Works in rudimentary Matlab syntax

Caveats of the present code,

0. Cannot lookinto execution times of specific lines-of-code
1. Not Matlab compatible
2. Cannot handle exceptions in profiler gracefully.
3. Regular caveats of a flat-profiler in not knowing context
  of the function calls; (call-graph based profilers help here,
  sort-of)
4. No event filtering
5. Sub-Optimal API design

For compatibility with Matlab, it suffices to provide an API to reach
the level of a event-based, call-graph profiler with access to
individual lines.

I am posting the diffs against the files, and the 2 new files for
octave/src/profile.cc & octave/src/profile.h

Please comment.

Thanks,

-Muthiah

You can look at the whole thread here.

What remains to be done is to write the hooks of the profiler
function into a separate AST walker in the Octave interpreter,
and make the evaluation-AST changeable at runtime using
the profile() function, when it is set. This AST type evaluation
is required, as there will be a performance hit of about 2x compared to the non-profiling interpreter.

Making a AST type evaluator with the hooks for the interpreter
is a non-trivial task; but that involves working on making the event hook mechanism delivery on function-call and return events. This however, does not affect the design of the profiler itself. So this is the missing-link in the profiler game.

Once the profiling infrastructure is in place using the separate AST, then we can profile fine-grained events including

1. Source code line , enter-leave events,
2. Variable watch/change events,
3. Regular function-call return events, and
   
4. Exceptions or Non-local exits.

This profiling-event-generator AST we need to put in place for the people to write profilers of their choice. I have provided a Flat and Call-Graph AST's. You can also contribute your mite!

Cheers,
Muthu

Octave Developer Tools

I've always felt that Octave can have more developer tools, like

Developer Tools Wish List

1. octave-lint (static code checker)
2. octave-indent (code formatting)
3. octave-pprint (a library of functions for pretty printing)
• - I had posted earlier this month? on Octave-maintainers list,about making a pretty-printer. A good place to start looking will be the Python/Lib/pprint.py
4. octave-profiler (call-graph & static per-line, per-function profiler)
• presently in the works, wonder how it will reach into the code-base
5. octave-debug (debugging user scripts/functions from Octave)
• This will be mostly on lines of John Eaton's post asking for a AST/Tree-Walking evaluator for a profiler & a debugger.
• Presently Octave has the tree-walkers for printing code, in its
• debug mode, and a
• semantic-checker after parsing,I guess.

So if you feel like throwing some time on these tools, welcome.

PS: Originally intended to be posted on, Thu, 22 Feb 2007 00:37:08 -0600

GNU Octave: The beginnings

Hello there!

Welcome to my page on GNU Octave.

I am a part-time developer on the GNU Octave project, and I want to share my 2-cents on ideas, directions and future of the GNU Octave project.




GNU Octave is a scientific numerical computing environment, including the
Matlab-like programming language, a rich set of libraries built on powerful open
solvers, is a necessary tool in any engineer's bag.

This blog will primarily address developing GNU Octave itself, and to a lesser
extent, developing with GNU Octave.


In particular, I would like to speak about

1. Whacky ideas on extending GNU Octave
   
2. Architecture of GNU Octave
1. On programming languages
2. Octave interpreter
   
3. A future GNU Octave compiler as a GCC front end
   
3. Features of GNU Octave
1. present,
2. missing,
   
3. wanted,
   
4. can-reimplement
4. Developing fast code on GNU Octave
5. Octave C++ extensions / building external libraries

Caveat Emptor
Any and all errors are my own, and dont blame me for whatever you use the information on this blog. I want to maintain that whatever I say on this blog is my opinion. It does not reflect the opinion of my employer, school, or of GNU Octave project.