Getting the basic IntelliSense operational proved to be relatively easy - I used the Iron Python project as a guide and with a good bit of cutting and pasting and nipping and tucking got a dummy IntelliSense system working pretty quickly. I’m not normally a great fan of Iron Python as a ‘how-to’ - it’s way too complicated. But as a reference project incorporating everything in the Visual Studio SDK pantheon it’s getting pretty good. However, I do think that a simpler MyC end-to-end example would be handy.

But then the next problem was how to get the information on a particular member - what are the methods of ‘self’ for example? After a false start - I tried to collect the class/def information myself - I hit on what looked like the ideal answer: Antlr! I’ve used Antlr for generating the colour information and I’d written (what looked to me like) a complete parser, though I hadn’t used it. So it’s a small step to use the parser to create an AST (Abstract Syntax Tree) suitable for collecting IntelliSense info. And after figuring out Antlr’s tree building language (Antlr really is an extraordinarily flexible system) got an AST.

Then I hit a small problem: to get the information, the parse has to be complete. It’s not terribly useful if it gives up half way through a correct program (syntactically incorrect ones are a different issue). The small problem was that my parser wasn’t as good as I thought. I’ve spent the last two weeks (solid!) as punishment working through every example of Ruby code I could lay my hands on - namely, the two thousand files or so that come with the Ruby distribution plus various gems.

I’ve alternated between sheer disbelief about the really poor quality of some of the code ( being polite - ‘yuk!’) and amazement (‘ I didn’t know you could do that in Ruby!) but the exercise has been worthwhile - my parser now does cope with every Ruby construct (bar one - nested here documents) in 2600 files - and I’ve fixed several colouring bugs in the process. It’s fast too: processing 2600 files took 60 seconds - about 25ms per file.

However, it did strike me that what I was doing was generating a successive approximation to a Ruby parser. The actual C Ruby parser is a combination of Matz’s meanderings and bug fixes over the years - and it’s neither pretty nor elegant. I can get closer and closer to the real thing but the amount of effort required to do that increases. For example, the one construct I won’t deal with at present is nested ‘here’ documents. For various technical reasons, they are hard hard work to implement. And how many people use them, anyway? I’ve only come across two examples so far in all those 2600 files.

I can see why others have used the Ruby parser itself to do syntax analysis. But the downside is that first it’s very, very slow (Antlr is fast!) and second it’s difficult to integrate and expand. The upside is of course that you don’t have to do much work and it’s 100% accurate by definition. But I’ve thought about this quite a bit now, and I’m pretty happy with the direction I’ve chosen: once the parsing is done, adding class browers, navigators and other cross-referencing features become possible. Find All References, anyone?

Here’s a small example of a very simple IntelliSense operation. Kudos to Antlr and Terrence Parr!