External macros with setedit

     Since version 0.4.57 of Setedit it's possible to create macros which call external programs. This allows to use the editor for things which wasn't designed beforehand. Thanks to the script provided below you add to the editor the text reformatting feature, very handy when you answer emails, since it respects the different identation levels of previous answers. It's not perfect, but it's very easy to use. Here's the text I send SET to include in Setedit's documentation:

How to use Setedit for something it wasn't meant to
===================================================
By Grzegorz Adam Hankiewicz.

SET has always said that setedit is not a word processing
program. He's right, it's best at doing things like programming. But
once you get used to it the road to building another emacs is clear:
you start wanting to use it even to cook your breakfast. This
document, however, centers on how to control text format easily
under setedit, which is not that ambitious :)

As you will notice, in some version SET added the "word wrap"
option. IMHO it should be removed, if you try to use it, it
will behave quite differently from what you expect (ie: if you
are writting tabbed lines, the editor will ignore them and put
your cursor right away in the first column ignoring any possible
autoidentation options and tab uses you might have set for the
document). Also, it's setedit's choice of characters which will be
used to split the line. Bleah... it really sucks.

But setedit is customizable. You can write macros in a lisp
like language. The language still doesn't allow you to control
tightly things like text formatting, but there's one cool option:
you can pipe a block selection to another external program and
the editor will substitute it with the output of this external
program. Certainly not the most optimized way of doing things, but
flexible enough to write a custom extension which treats your text.

And here we come... the purpose will be to create a macro, which
feeds a text block to an external program which we will write. Simple
as that, all the power you wanted under your fingers.



Step 1. Building your macro.
============================

Macros are written in slisp, and stored somewhere in a macros.slp
file. This file can be a global read only file or it can be located
at ~/.setedit/macros.slp.  It's pure text, and somewhere (ie:
at the end of the file) you will insert the following chunk of code:

;*******************************************************************
;
; MACRO:   Block reformat
;
; DESCRIPTION: This script gets the current selection and feeds it to
; an external program. If no selection is present a mesage box will
; be printed.
;
;*******************************************************************/

(defmacro 'Block reformat'
 (eval
  (setv "input" (GetSelection))
  (if (length input)
   ; Call the filter
   (eval
    (setv "output"
     (RunProgramRedir
      ; filter line here
      "~/project/email-fmt.py/email-fmt.py "
      input
     )
    )
    (SendCommands cmcCut)
    (InsertText output 1)
   )
   ; Ask the user to select something firt
   (MessageBox "Please select a text first")
  )
 )
)

Ok, don't worry about the syntax of the macro or how it's written,
if you are curious you can take a look at setedit's documentation
to find out what all that mess means. The important line is the
one which follows the commen "; filter line here". That line
is actually the path to the external program we will be using,
which can be stored anywhere on your computer, it only needs to
have execution permission. I am the only user of my machine, so I
don't need to put things under /local/bin to feel comfortable :)
Please change that to point at your filter script.



Step 2. The filter program.
===========================

The filter program has to be simple: recieves text lines as input
and prints to stdout the result. You can say: "Hey, that calls some
weird Python script which I don't have... why don't
I call GNU's fmt command instead?".  Good point. Do it. If you are
meaning to write simple texts that's ok. In fact, after the command
you can specify parameters to fmt like the width of the lines, and
there you go, you have a text formatting macro in Setedit. But if
you are meaning to answer emails (ie: setedit is your environmet
editor and mail programs like mutt call it to write messages) or
do something more sophisticate, you better use something like this
(big chunk of code follows):

#!/usr/bin/env python

import sys, popen2, tempfile, os

class LINE:
   pass

identation = " \t>:;#"

def detect_identation_characters(line):
   """Returns number of identation chars minus trailing whitespace."""
   count = 0
   for f in line:
      if f in identation: count = count + 1
      else: break
      
   return len(line[:count].rstrip()), count

def initial_scan(line):
   """Add attributes to the lines."""
   l = LINE()
   l.line = line
   l.total_length = len(line)
   l.ident, l.soft_ident = detect_identation_characters(line)
   l.length = l.total_length - l.soft_ident
   return l

def same_identation_lines(lines):
   """Returns number of lines with same identation level."""
   f = 1
   while f < len(lines):
      if lines[f].ident != lines[0].ident:
         break
      else:
         f = f + 1
         
   return f

def reformat_lines(lines, temp_filename):
   """Dumps lines in list to file, the reads fmt's output."""
   assert lines
   ident = lines[0].ident
   length = max (70 - ident, 20)

   # create temporary file with content to ident
   file = open(temp_filename, "wt")
   num = 0
   while num < len(lines):
      string = lines[num].line[ident:]
      stripped = string.lstrip()
      if stripped:   file.write(stripped)
      else:          file.write(string)
      num += 1
   file.close()

   # call external tool and read it's stdout
   stdout, stdin = popen2.popen2 (["fmt", "-w", "%d" % length,
      "-u", temp_filename])
   stdin.close()
   if ident:   padding = "%s " % lines[0].line[:ident]
   else:       padding = ""
   new_lines = stdout.readlines()
   stdout.close()
   os.unlink(temp_filename)

   # output lines, taking care of last line's trailing '\n'
   for f in range(len(new_lines)-1):
      sys.stdout.write("%s%s" % (padding, new_lines[f]))
   if lines[num-1].line.find("\n") >= 0:
      sys.stdout.write("%s%s" % (padding, new_lines[-1:][0]))
   else:
      sys.stdout.write("%s%s" % (padding, new_lines[-1:][0][:-1]))

def main():
   lines = map(initial_scan, sys.stdin.readlines())
   temp_filename = tempfile.mktemp(".email-fmt.tmp")

   f = 0
   while f < len(lines):
      look = same_identation_lines(lines[f:])
      reformat_lines(lines[f:f+look], temp_filename)
      f = f + look
 
if __name__ == "__main__":
   main()
# end of python code

Woah, that's more than the previous script. True, but this 86
line Python script is a cool wrapper: fmt doesn't
know exactly how you want to format your text. If you are replying
somebody's email, some text will have quotes. Wouldn't be nice if
fmt could detect them? Yes, but it doesn't. Remember that the unix
style is doing one simple thing, but doing it extremely well.

So we solve this with the Python script. It isn't really
long (try removing blank lines and comments), and it's pretty clear
(to any experienced functional programmer) what it does. It scans
the sent block of text for existant quote characters like those hold
by the identation variable, which are used frequently by most email
programs. These lines will be kept together, and the script will
call fmt with them, but removing the quote characters and reducing
the line width. To the result, the script will add the apropiate
quote identation level.

This means that you can select say 20 lines of text which contain
three different quote identation levels.  If these use the defined
identation characters the script will split them, format them
separately and join them before returning them to setedit. The result
is obvious: a cool formatting feature which is cumbersome to build
into setedit, and which can be customized any way you want. You
can write your script in Python, ruby, perl, C, etc,
etc... you put the limit, not the program.



Step 3. Binding your script to a keystroke.
===========================================

Nice, we have the macro, we have the script, but how do we run
it? In a horrible world you would have to reproduce these steps to
format a text block:

   1. select text block
   2. open macro menu
   3. select "Select..."
   4. choose the macro

If you did it once, you can repeat the macro with Shift+F3, but
this is not nice, it prevents you from running other macros and
you aren't saved from the punishment of doing that all once each
time you run setedit, and since we want this for email replies,
the editor will be run independently for each answer. Ugh!

Luckily you can bind a macro to a key (section 3 in the
documentation). The option is located under the menu option
called Tool&Ops, submenu Options, submenu Keyboard, submenu Key
assignment. Once you reach that, you can see all the key bindings for
the editor. Notice that none seem to use Alt for keystrokes, which
is nice becase you can presume they are free for customizing. So you
just have to type a nice combo. I use Alt+F for "Formatting". Once
you have the combo you assign the macro "Block reformat" to it.

Say ok to all menus and try it yourself: write a few lines with only
a word in each of them. Select those lines and press "Alt+F". If
you did it all right you will see all those words formatted in a
single paragraph. Now try that with email identation. Think of all
the possibilities of using this trick along with the "rectangle
selection" feature of the editor.



Examples.
=========

I prepared some examples so you could see the good things of all
this work. All the examples are the result of selecting all lines
and using the macro. You can easily reproduce them yourself.

Written in setedit:

 So I am just     a lazy boy typing this
 which I know will be
    formatted correctly with that cool
     python script                     written by a
       guy with funny name.

After the Alt+F combo:

So I am just a lazy boy typing this which I know will be formatted
correctly with that cool python script written by a guy with
funny name.

Written in setedit:

On  9 Jan 01 at 14:25, Grzegorz Adam Hankiewicz wrote:
> On Sun, 7 Jan 2001, Petr Vandrovec wrote:
> > (2) you can specify only xres 'x' yres '-' bpp '@' fv, you cannot specify for example
> >     horizontal/vertical sync polarity, sync-on-green mode, or even shift picture
> >     left-right (modify left/right/upper/lower/hslen/vslen fields of var_screeninfo)
> > 
> > First problem is probably unresolvable without linking modedb to each fbdev separately,
> > as if you removing __init, others will come to you with big staff.
> > 
> > Second problem is probably only 'invent how to write it' problem... You can look at
> > matroxfb picture size/refresh related parameters - all of them should be parsed by
> > modedb for benefit of all drivers...
> 
> Aha.. do you mean I have to add new options and switches to modedb, taking
> out the parsing code from the matroxfb module? For example, adding new
> modedb functions which have more parameters, to remain backwards
> compatible with the existant code?

In my opinion it should still take one char* and...

After the Alt+F combo:

On 9 Jan 01 at 14:25, Grzegorz Adam Hankiewicz wrote:
> On Sun, 7 Jan 2001, Petr Vandrovec wrote:
> > (2) you can specify only xres 'x' yres '-' bpp '@' fv, you
> > cannot specify for example horizontal/vertical sync polarity,
> > sync-on-green mode, or even shift picture left-right (modify
> > left/right/upper/lower/hslen/vslen fields of var_screeninfo)
> > 
> > First problem is probably unresolvable without linking modedb
> > to each fbdev separately, as if you removing __init, others will
> > come to you with big staff.
> > 
> > Second problem is probably only 'invent how to write it'
> > problem... You can look at matroxfb picture size/refresh related
> > parameters - all of them should be parsed by modedb for benefit
> > of all drivers...
> 
> Aha.. do you mean I have to add new options and switches to modedb,
> taking out the parsing code from the matroxfb module? For example,
> adding new modedb functions which have more parameters, to remain
> backwards compatible with the existant code?

In my opinion it should still take one char* and...