wily 0.13.42

[ozan s yigit]

it lives. :)

new wily version. updated ftp://ftp.cs.yorku.ca/pub/wily/src/
and will update sourceforge later tonight.

9libs had to have a micro-version increment. it is now at 1.0.1.

These are all the interesting changes from wily 0.13.41 to
0.13.42 done by tommy Pettersson <***@lysator.liu.se>. Please
let me know if you run into trouble with any of these changes.
if you have a philosophical disagreement with any of the fixes
also let me know, and i will try to resolve it.

thank you tommy. and thanks to derek peschel for bugging me
about configs. thank you to all those who have sent fixes in
the past, some of which is in this new version. i appreciate
the help.

---- changes ---

Configs for 9libs and wily-9libs have been brought up-to-date
to allow 9libs compilation under Mac OS-X.

The most complicated change is with reading utf8. Multibyte utf
sequences could get split and generate two invalid runes instead
of one valid. This could happen both on file reads, if the file
was larger then the temporary read buffer, and on output events
from external processes. Wily sometimes missed to warn about
nulls when reading large files. Nulls are now replaced with
invalid runes on file reads. Wily will also warn about both
nulls and (real) invalid runes when reading a file. It was
possible to enter a null rune into the text buffer with the
Alt-X combination, which now yields an invalid rune.

Fonts with 65536 chars made wily crash.

Unnecessary redrawing (flicker) has been reduced in several
ways. One patch is from Bill Trost and fixes flicker in
directory windows.

There were some ways to crash wily by selecting a reversed range
(from right to left) with mouse or keyboard. All these now just
revert the range and proceed normally.

Many bugs made wily crash while trying to draw the screen, and
others just trashed it. I've fixed a heap of them, but there are
surely some more left. I've used one patch from Sam Holden that
avoids division by zero when making a dir listing. I've used one
patch from Thomas Nordin that stops some ugly scrolling. I've
used one patch from Peter Canning that renders scrollbars
correctly even with very many lines in the text buffer. Fixes
include: no disappearing borders, no extra spaces around
anchors, don't trash anchors and process names in tag lines, no
faulty autoindenting on non-ascii letters, no extra newline with
autoindent, don't garbage tag line when switching font.

Another source of crashes was casting of pointers to bools and
bool comparisons where bools really are integers. Some
conditions would randomly give the wrong answer and cause
unexpected behaviors.

CapsLock did not work with XFree86 version 4, but a solid
workaround that catches this is now in place.

A renamed file buffer gets its new name as backup name when Put.

Text buffers for new files allocated twice as much memory as
needed (in addition to the extra gap space for inserting more
text).

The text search was broken in a couple of ways, but only one of
them could actually happen, and it was very very unlikely to be
used (-/,/).

Minor fixes: use dir tools in dir windows, use expanded dir path
for window destination column hash (Bill Trost), reset errno
before calling diag() (suggested by Elliott Hughes), don't
optimize away some wanted mouse button events.

[Ian Broster]

Post by ozan s yigit
it lives. :)

Congratulations! Well done. I confirm that it compiles and runs ok for me
on debian.

I have been very lax about applying patches to my copy as they have
appeared, I have been waiting for someone else to do all the hard work and
reconcile them all together. So, I'm very pleased to have this new version.

Obviously, it has not been possible to fix all bugs this time, there are a
couple that I notice straight away. I'll include them below in the hope
that someone will manage to fix them. :-)

I notice that Debian has wily as a package. I don't know who maintains
this? Can we see a debian upgrade to this new version?

Ian

P.S. two noticeable bugs:

1. Binary files. Load a binary fily, /bin/echo for example, and Put it
again as a different filename. The two files are different.
2. Window resize: shrink a window horizonally, then expand it again.
Different algorithms are used for the proportions for the existing columsn
when skrining and expanding.

[MJ Ray]

Post by Ian Broster
I notice that Debian has wily as a package. I don't know who maintains
this? Can we see a debian upgrade to this new version?

dpkg -s wily | grep Maintainer
Maintainer: MJ Ray (Debian) <***@debian.org>

That'll be me, then.

Upgrading will involve moving to 9libs, as I understand it. I'd
not done that because I didn't succeed in building a version which
ran on all my machines, so I was cautious. I'll do it Real Soon Now,
or sooner if you'll share your build notes...

Thanks,

[Ian Broster]

Post by MJ Ray
Upgrading will involve moving to 9libs, as I understand it. I'd
not done that because I didn't succeed in building a version which
ran on all my machines, so I was cautious. I'll do it Real Soon Now,
or sooner if you'll share your build notes...

./configure;make

:-)

Which machines didn't this work on?

Actually, both the 9libs version and the non-9libs version built "out the
box" on debian for me.

i.

[MJ Ray]

Post by Ian Broster
./configure;make
:-)
Which machines didn't this work on?

I don't remember. I'll see what happens this time, if it's got
that easy.

Thanks,

[ozan s yigit]

Post by Ian Broster
1. Binary files. Load a binary fily, /bin/echo for example, and Put it
again as a different filename. The two files are different.

hmm, you are right this is a yet-unfixed one. :-P i have just tried
the older version on osX and freebsd and indeed files are clobbered....

oz

[Gary Capell]

Post by ozan s yigit

Post by Ian Broster
1. Binary files. Load a binary fily, /bin/echo for example, and Put it
again as a different filename. The two files are different.

hmm, you are right this is a yet-unfixed one. :-P i have just tried
the older version on osX and freebsd and indeed files are clobbered....

I'm not sure this is a bug we want to fix? I'm guessing that this
is because (not surprisingly) some bits of /bin/echo aren't valid
UTF8. To fix it we'd have to maintain two copies of the file:
the version with valid UTF, and the original version with invalid
UTF. Actually, that just lets us replace 'Put' with a no-op,
of writing back the original data.

What _might_ make sense is to mark a file as Read Only (or
something) when the utf parsing has made changes that will
be destructive when we write it back?

... or I could be completely misunderstanding the problem.

[ozan s yigit]

Hi Gary,

i am not sure either. i think this is where years of emacs binary
patching has spoiled some of us... :)

Post by Gary Capell
I'm not sure this is a bug we want to fix? I'm guessing that this
is because (not surprisingly) some bits of /bin/echo aren't valid
the version with valid UTF, and the original version with invalid
UTF. Actually, that just lets us replace 'Put' with a no-op,
of writing back the original data.
What _might_ make sense is to mark a file as Read Only (or
something) when the utf parsing has made changes that will
be destructive when we write it back?

this is an interesting idea actually. better than silent
conversion...

oz

[Tommy Pettersson]

Post by ozan s yigit

Post by Gary Capell
What _might_ make sense is to mark a file as Read Only (or
something) when the utf parsing has made changes that will
be destructive when we write it back?

It is already made non-backupped, so the Put command doesn't
appear unless the label is changed.

Post by ozan s yigit
this is an interesting idea actually. better than silent
conversion...

As of wily 0.9.42 any input that can't be represented in the
buffer is replaced with the "invalid" rune, so an easy way to do
this is to (verbosely) refuse to Put buffers that contains
invalid runes. Since the invalid rune can be manually inserted
with Alt-X 0000, this could be (ab)used to write protect
buffers, although it complicates the feature of saving guide
files with an invalid rune in them so they don't fill the change
box or leave backups.

[ozan s yigit]

Post by Tommy Pettersson

Post by ozan s yigit
this is an interesting idea actually. better than silent
conversion...

As of wily 0.9.42 any input that can't be represented in the
buffer is replaced with the "invalid" rune, ...

i will have to re-think this invalid-rune change.

oz

[Ian Broster]

Post by Gary Capell
I'm not sure this is a bug we want to fix? I'm guessing that this
is because (not surprisingly) some bits of /bin/echo aren't valid
UTF8.

I see.

However it does seem to break a rather fundamental assumption that a load
then save will not corrupt the file.

Most text editors seem ok with binry files (presumably because the don't
do UTF at all); despite the technical difficulties of doing so in wily, I
think it's worth another think.

Post by Gary Capell
the version with valid UTF, and the original version with invalid
UTF.

Is there no way to have a raw representation in memory and a best-effort
UTF8 render/manipulation?

i.

[Gary Capell]

Post by Ian Broster

Post by Gary Capell
I'm not sure this is a bug we want to fix? I'm guessing that this
is because (not surprisingly) some bits of /bin/echo aren't valid
UTF8.

I see.
However it does seem to break a rather fundamental assumption that a load
then save will not corrupt the file.

Agreed. Hence the "ReadOnly" suggestion.

Post by Ian Broster
Most text editors seem ok with binry files (presumably because the don't
do UTF at all); despite the technical difficulties of doing so in wily, I
think it's worth another think.

Post by Gary Capell
the version with valid UTF, and the original version with invalid
UTF.

Is there no way to have a raw representation in memory and a best-effort
UTF8 render/manipulation?

Of course there's a way (probably _lots_ of ways) to do it.
The questions are:
* do any of the ways have negligible impact on code complexity
and efficiency (i.e. not requiring duplicate work on
two copies of buffers)?

* do the costs (time to design/write, added time for future
work, efficiency cost, ... ) outweight the benefit
(can edit by hand binary files)?

[Tommy Pettersson]

Post by Gary Capell

Post by Ian Broster
Is there no way to have a raw representation in memory and a best-effort
UTF8 render/manipulation?

Of course there's a way (probably _lots_ of ways) to do it.
* do any of the ways have negligible impact on code complexity
and efficiency (i.e. not requiring duplicate work on
two copies of buffers)?
* do the costs (time to design/write, added time for future
work, efficiency cost, ... ) outweight the benefit
(can edit by hand binary files)?

It is "possible" to edit binary files with wily like

<some_quoting_filter --encode <file

Post by Gary Capell
some_quoting_filter --decode >file

It is probably both easier and cleaner to write such a filter
than to represent and special case invalid byte sequences inside
wily. To convenience heavy usage it could be written as an
rpc-client with BinGet and BinPut.

I think valid perl code may contain null bytes, but that's a
slightly different problem since that is valid utf.

[q***@speakeasy.net]

thanks for clearing that up!

did you know that the original acme is now available on unix?
actually, it's better than that. plan9port (http://www.swtch.com/plan9port/)
is a pretty good plan 9-emulation environment, complete with
acme.

that version of acme has imported a feature from wily. the -$
option will replace substrings in paths with the appropriate
variable name.

neither acme nor sam edit binary files. they elide
zeros. what's the advantage of editing binaries? the disadvantage
is that all the text handling code needs to be rewritten to
allow for:

(a) non-utf-8 characters
(b) non-null-terminated strings.

- erik

-----Original Message-----
Sent: Thursday, April 6, 2006 07:24 PM
Subject: wily 0.13.42
it lives. :)
new wily version. updated ftp://ftp.cs.yorku.ca/pub/wily/src/
and will update sourceforge later tonight.
9libs had to have a micro-version increment. it is now at 1.0.1.
These are all the interesting changes from wily 0.13.41 to
let me know if you run into trouble with any of these changes.
if you have a philosophical disagreement with any of the fixes
also let me know, and i will try to resolve it.
thank you tommy. and thanks to derek peschel for bugging me
about configs. thank you to all those who have sent fixes in
the past, some of which is in this new version. i appreciate
the help.
---- changes ---
Configs for 9libs and wily-9libs have been brought up-to-date
to allow 9libs compilation under Mac OS-X.
The most complicated change is with reading utf8. Multibyte utf
sequences could get split and generate two invalid runes instead
of one valid. This could happen both on file reads, if the file
was larger then the temporary read buffer, and on output events
from external processes. Wily sometimes missed to warn about
nulls when reading large files. Nulls are now replaced with
invalid runes on file reads. Wily will also warn about both
nulls and (real) invalid runes when reading a file. It was
possible to enter a null rune into the text buffer with the
Alt-X combination, which now yields an invalid rune.
Fonts with 65536 chars made wily crash.
Unnecessary redrawing (flicker) has been reduced in several
ways. One patch is from Bill Trost and fixes flicker in
directory windows.
There were some ways to crash wily by selecting a reversed range
(from right to left) with mouse or keyboard. All these now just
revert the range and proceed normally.
Many bugs made wily crash while trying to draw the screen, and
others just trashed it. I've fixed a heap of them, but there are
surely some more left. I've used one patch from Sam Holden that
avoids division by zero when making a dir listing. I've used one
patch from Thomas Nordin that stops some ugly scrolling. I've
used one patch from Peter Canning that renders scrollbars
correctly even with very many lines in the text buffer. Fixes
include: no disappearing borders, no extra spaces around
anchors, don't trash anchors and process names in tag lines, no
faulty autoindenting on non-ascii letters, no extra newline with
autoindent, don't garbage tag line when switching font.
Another source of crashes was casting of pointers to bools and
bool comparisons where bools really are integers. Some
conditions would randomly give the wrong answer and cause
unexpected behaviors.
CapsLock did not work with XFree86 version 4, but a solid
workaround that catches this is now in place.
A renamed file buffer gets its new name as backup name when Put.
Text buffers for new files allocated twice as much memory as
needed (in addition to the extra gap space for inserting more
text).
The text search was broken in a couple of ways, but only one of
them could actually happen, and it was very very unlikely to be
used (-/,/).
Minor fixes: use dir tools in dir windows, use expanded dir path
for window destination column hash (Bill Trost), reset errno
before calling diag() (suggested by Elliott Hughes), don't
optimize away some wanted mouse button events.