Blog

Text formatting in WebIssues

Recently I wrote about version 1.1 of WebIssues and my plans to introduce issue descriptions and formatting of comments and descriptions. I also listed various markup languages which I considered for using in WebIssues. But first, let's look at how text is handled in the current version of WebIssues.

WebIssues currently uses plain text for comments. All whitespace characters, including indentation and line breaks, are preserved, making it easy to paste fragments of code directly into comments without breaking their formatting. At the same time, WebIssues wraps long lines, making it possible to write long paragraphs of text which are displayed correctly regardless of the width of the window. This basically corresponds to the "white-space: pre-wrap" CSS style. In addition, external URLs issue identifiers are automatically converted to links.

The key idea behind adding extended formatting options is to preserve compatibility with this "plain text" mode. It should be possible to edit an existing comment, enable formatting and add some markup to the existing text without breaking existing formatting. Now the problem is that most existing markup languages either ignore whitespace (for example HTML, unless wrapped in a <pre> block) or handle it in a specific way (for example, double line breaks are converted to paragraphs, indentation indicates a block of code, etc.). I'm not saying that they are wrong; this often makes sense when copying text from text files or plain text emails. However, I don't want to break habits of existing users of WebIssues. I would like to treat spaces and line breaks identically, whether formatting is enabled or not. Thanks to this, pieces of code will not break, even if they are not marked using special tags. These tags will only be used for decorative purposes, for example, by using different background color and enabling syntax highlighting.

There are generally two kinds of markup used in existing languages: punctuation (brackets, quotes, asterisks, etc.) and tags. Punctuation is used for inline formatting in various flavors of Wiki and languages such as Markdown or Textile, for example to indicate bold or italic text, though there is no single standard. It is also used for block formatting, for example trailing '>' may indicate a quote. HTML tags are commonly used by various languages, in addition to other format specifiers, though they are useful mostly for advanced formatting. Finally, various flavors of BBCode use custom tags, which are similar, but simpler than HTML. I decided to use a combination of punctuation for inline formatting and custom tags for block formatting. It's questionable whether yet another language should be invented when there's so many already, but I think it's going to be intuitive for everyone, and thanks to the embedded markItUp editor, there will be no need to remember it.

The following inline formatting tags will be supported: **bold**, __italic__, `monospace text` and [URL custom links]. The * and _ characters appear commonly in a technical text, so they need to be doubled to avoid false positives. Link syntax is quite similar to Wiki external links, however internal links can be created the same way, for example: [#123 some issue]. In the future it will be possible to introduce real Wiki functionality (where names can will be used instead of numeric identifiers).

Three different block level formatting tags will be suported. A [code][/code] pair will indicate a block of code, with optional syntax highlighing based on Google's prettify. A [quote][/quote] pair will indicate a quote with an optional header. Finally, a [list][/list] pair indicates a bullet list, where each item starts with one or more * (multiple asterisks indicate nested levels). Unlike automatic lists used by many markup languages, explicit tags will make it easy to clearly indicate where the list starts and ends. Also if will be possible to freely mix and nest all three kinds of tags.

I'm now playing with the prototype of the converter, and I may still do some minor changes, but so far I'm rather satisfied with the result. So when version 1.1 is going to be released? By the end of this year - that's all I can promise for now. I will probably release some beta version in a few months. But I still have my book and various other things to do, so don't expect miracles.

Filed under: Blog
Tags: WebIssues

WebIssues 1.1

Before I get to the main topic, just a short update on my novel :). I wrote a few more chapters and I have some new ideas, but I feel that I need some break. When I first started writing back in 2011, I was so involved that I could write all night, but now I have to get up earlier and generally I have too many things to do to be able to fully concentrate on this. So it's becoming a somewhat tedious process, quite like programming that I was trying to escape from. However, the result is still quite good and I'm certainly not going to leave it off.

Anyway, I can't ignore the fact that the idea of WebIssues 1.1 is growing in me. I've already had some items on the roadmap, but there's so many of them that I will have to split them into two releases. I'm probably going to postpone all improvements related to users, groups and permissions until version 1.2. The main improvement in version 1.1 will be issue descriptions. It's something that's clearly missing compared to other bug tracking systems. Of course, the first comment can act as a description, so the change is a bit cosmetic, but the ability to provide the description directly when creating an issue will certainly be an improvement. During the upgrade from version 1.0, the first comment will be automatically converted to a description.

Also projects will now have a description. There will be a project summary page, which in the future may contain other useful information, such as statistics, recent issues, etc.

The last (but not least) improvement in this area will be the ability to use simple formatting in both comments and descriptions. And that's an interesting problem, because there are lots of different markup languages that can be used to add formatting to a piece of text. Each existing standard has it's advantages and disadvantages:

HTML
Powerful and good for CMS, blogs, etc., but it's difficult to use by non-geeks. And it's even more difficult to display it correctly. A naive implementation opens the possibility for XSS attacks. Simple tools like kses still won't ensure that the markup is valid (e.g. check for unbalanced tags). More advanced tools like htmlpurifier are simply monstrous.
Textile / Markdown
They are quite different, but based on a similar idea: make the source text look as natural as possible. I prefer the latter, although they both seem to make more sense for writing longer articles (especially technical) than simple descriptions and comments.
Wiki markup
The main problem is that there is simply no such thing as a standard Wiki syntax. Although there are many similarities, each implementation has its own flavor. Also note that adding true Wiki support to WebIssues (i.e. being able to create cross-links based on titles, not just issue IDs) would be an entirely different story.
BBCode
Simple and widely used (also with many different flavors). On the other hand, square brackets don't seem more intuitive than angle brackets used in HTML.

This is a broader topic and I will write more about it in a separate post. So far I'm leaning towards a subset of Wiki syntax with some modifications, but I have to think more about it. And don't even get me started on the so called "WYSIWYG" editors. They are bloated and/or buggy and not 100% portable. I think I'm going to create something based on markItUp which is small, simple and easy to customize.

Yet another area of functionality that sooner or later must be (and will be) added to WebIssues is support for inbound emails. Some thoughts have been circling around and a few different persons have offered to help me implement this. If something gets done then I will include it in one of the next releases, but for now I can't promise anything.

Filed under: Blog
Tags: WebIssues

Writing

I want to write. Well, of course, I do; but I don't mean programs and technical documentation, but novels. My New Years resolution is to finish the book that I started some time ago and get it published. Why this sudden change of mind? Just a few months ago I wanted to start a business based on WebIssues. I even managed to briefly bring the attention of the management of the company I work for to it. But their idea of investing very little in order to hopefully get some profit wouldn't make too much sense. My own vision wasn't downright rejected, but considering all the political aspects that rule a corporation like this, and my complete lack of influence on these things, I can't realistically expect that this is ever going to happen.

Obviously, making a living from writing is an even more insane idea. It's a very demanding market, and in Poland also quite a narrow one. It also requires a lot of pure luck, probably even more than running a successful business. Not to mention that writing a novel requires huge amounts of time. But the real problem is that I'm really starting to hate programming. Commercial or open source, it's tedious, repetitive and rarely creative. And writing is not a new idea. I wrote some stories as a child. In high school I started writing a book with two friends; it didn't last long, but it was a lot of fun. But this time it's different, because I already have most of the plot in my mind, so the ideas are there waiting to be put on paper.

I already mentioned the novel I'm writing once or twice, but perhaps this time I will shed some more light on it. The idea came to my mind in summer 2011, while I was reading Neal Stephenson's Snow Crash, but it was also influenced by Lev Grossman's The Magicians which I read shortly before. It's basically a cyberpunk story, taking place largely in two different virtual worlds, but it also has some elements of contemporary fantasy and techno-thriller. The main characters are a few students of a school for young hackers, which is called the Academy of Magic, because in a virtual world, the boundaries between hacking and magic are blurred for the uninitiated. As my younger brother described it, when I told him about the novel today, it's like a "rolled pancake" :). I admit that mixing genres is risky, but if I do it well, maybe something interesting will come out of this.

And by the way, yesterday was my son's first birthday :). I must publish some new photos soon because I haven't done that in a while.

Filed under: Blog
Tags: personal

Habits and standards

Yesterday was the seventh anniversary of mimec.org, but I will not elaborate on that. It suffices to say that the last year was very different from the previous ones. My son Adam changed from a blurry ultrasonographic image to a little boy who runs around the house. There is no time for anything. I can hardly keep up with my paid job, not to mention the open source projects, but I still managed to make four minor releases of WebIssues, an one release of Saladin (with another one pending), Fraqtive and Descend.

A few days ago I finally got a new laptop. It has a 15" Full HD display, which for some reason is very rare these days, powerful CPU and GPU and plenty of RAM. Minecraft runs at about 50 FPS at full screen with far viewing distance :). The bad news, though, is that my company run out of Windows 7 licenses, and I was forced to install Windows 8. I'm not going to rant about it, becuse enough has been said about it already. After installing the English language pack and removing the metro-garbage from the start menu, I'm getting used to it without having to change my habits too much. It's just hilarious that the now so called "desktop" applications suddenly became legacy and are only temporarily supported for backward compatibility. It reminds me of how all existing applications suddenly became "unmanaged" when .NET was created, as if they were crippled in some way. Microsoft suggested that in a few years all applications would become "managed", and finally support for those "unmanaged" ones would be dropped. Of course I don't mind .NET; it's just the kind of marketing speech that makes me laugh.

But when I saw Office 2013 with the black and white UI and icons designed for displays that support only 8 colors, it actually made me a bit upset. For a long time Office was setting the user interface design standards for a lot of Windows applications, especially regarding toolbars and menus, because the default ones always had a very plain look. Obiously I also always tried to keep up with the trends. Over ten years ago, in Grape3D, I used third party menu and toolbar classes for MFC which mimicked the flat, semi-transparent highlighting style know from Office XP. Later I wrote my own set of classes which broke out of the Office trends for a while and looked more like IE 6. But soon after that Microsoft released Office 2003 with the spectacular bright blue and orange UI which automatically changed its colors to match the Windows XP theme. Whether it looked good or not, it became a long time standard. Just take a look at version 0.9 of the WebIssues Client, or the so called "modern" Qt style which I wrote in 2008, and you will know what I mean.

The so called "ribbon" introduced in Office 2007 was something that people complained and ranted about nearly as much as the Metro UI in Windows 8, but it eventually turned out to be a very good idea. It was not just a cosmetic change, but something entirely new. Currently all my programs use a similar concept, which is available as part of the XmlUi component. At the same time the bright colors were toned down and the whole thing looked equally good with classic Windows style as with Luna and Aero. But now that I'm getting more and more used to Windows 8 and Office 2013, even the soft gradients and slightly rounded corners of XmlUi are beginning to look a bit odd. So what is the next logical step? Should we, developers, all turn to creating rectangular, black and white UI? How soon will Microsoft change its mind and what will be the next "standard"? Or perhaps it's time to stop bothering?

Filed under: Blog

Fuzzy optimizations in QTransform

Before I get to the point, just a quick note: I recently released a new version of both WebIssues and Fraqtive and I'm planning to release a new version of Saladin by the end of this year, so I'm quite busy, as usual at this time of the year. Also check out the fractal animations created using the latest version of Fraqtive. There are much more impressive deep zoom animations available, but they were created using specialized tools which use high precision numbers. Fraqtive uses SSE2 to maximize the real-time experience, so it's limited to double precision, which allows to zoom the fractal about 10^13 times. But this is enough to produce some cool effects.

Now back to the main topic. For a long time there was a strange bug in Fraqtive that I thought was impossible to fix. As those who use it know, in Fraqtive it is possible to move and zoom around the fractal using the mouse, and when you release the mouse button, the new region of the fractal is recalculated. This works by converting the start and end position to a QTransform (which is basically a 3x3 matrix defining the offset, zoom and rotation) and calculating the relative transformation. It worked fine until a certain zoom level was reached; then it started producing weird results. I always blamed the limited precision of double numbers, though the effect could be observed a few orders of magnitude before reaching the maximum valid zoom.

Recently I debugged the entire code, including the calculations performed inside QTransform class and discovered that it performs some optimizations which cause the wrong results. The documentation mentions that QTransform performs some optimizations based on the type of the matrix. For example, if the transform includes translation only, the scale and rotation components are ignored. What the documentation doesn't mention, though, is that Qt uses fuzzy compare functions (such as qFuzzyCompare) to determine the type of the matrix.

The problem with qFuzzyCompare and it's undocumented qFuzzyIsNull counterpart is that they assume that only about 12 digits are significant in a double precision floating point number. This often makes sense, because limited precision can result in some subtle differences which cause the strict comparison operator to fail. We all know that 10 / 3 is 3.33333... and that value multiplied by 3 gives 9.99999.... In mathematics, this value is equal to 10, but for a computer, these numbers are not necessarily equal.

The effect of using fuzzy comparisons is that at a zoom level higher than 12, the scale factors are considered equal to 0, so the transformation is considered non-invertible, when in fact it is. Also rotation tends to be ignored at this zoom level. The solution is not to use QTransform when high precision is required or to write custom functions which do not have these side effects. See also [#QTBUG-8557] where a similar problem is discussed.

Note that fuzzy comparisons are used not only in QTransform, but also in other classes like QMatrix4x4 and QVector3D. Some time ago I came across the a similar problem with QVector3D::normalized which caused Descend to incorrectly calculate normal vectors for the surfaces. The problem is that Descend calculates three samples that are very close to one another in order to precisely determine the normal vector. In some areas of the surface it would hit the 12 significant digit limit, so I ended up writing my own version of this function which didn't use fuzzy comparisons.

Filed under: Blog
Syndicate content