Skip to main content
-
-Rozi-
Inspiring
September 26, 2014
Answered

Design Guidelines for Grammar Checker or Similar Background Story Monitoring

  • September 26, 2014
  • 2 replies
  • 2465 views

I have already built a spell checker based on CHLinguistic successfully.

 

Now, I was given a task to incorporate a proprietary grammar checker into InDesign CS5 for one of our customers. After initial review the task seemed difficult but doable. Unfortunately, I got lost in the complexity of InDesign after I started. Looking into SDK and documentation seems like I have to learn complete InDesign internals to get it done.

 

I’ve made two plugins by now: model and UI.

 

The UI is based on SpellingPanel source code. It attaches an observer to monitor “Dynamic Grammar Check” setting and installs IEventWatcher to watch for keyboard events. The later installs an idle task after user stops typing. The idle task invalidates the paragraph of the text user is editing. I’ve implemented kGlobalTextAdornmentServiceImpl and kGalleyTextAdornmentAfterServiceImpl services for drawing the adornments.

 

The model plugin is where I should implement the very grammar checking and my own data strand to save grammar meta-info, right? The grammar checking itself is very slow: up to 10 seconds per sentence, so it has to be asynchronous.

 

My questions are:

 

  1. How would you design a background task to check grammar?
    A thread to attach on each document when it’s is opened, and an observer of the IID_ITEXTMODEL to recheck what gets changed later on? Saving grammar results in a separate data strand?

    I am looking for a simple recipe. Hopefully, I can work out the details myself.

  2. Is it possible to get a source code of Linguistic plugin? After signing a non-disclosure agreement of course. There might be some valuable clues there. Basically, spell checking could be used for grammar checking if only the Linguistic wouldn’t chop text to single words.
  3. Does the Debug version of InDesign enable a developer to breakpoint and step thru the Adobe’s code? Are PDBs included? Is there really no way to get Debug version before buying Adobe Partnership?
  4. There are some grammar checkers already available for InDesign. Is there any vendor to build the grammar checking into InDesign in such a way, a third-party plugin could add another language support to it using a simple approach?

  

I am desperately searching for clues. Any help is greatly appreciated.

This topic has been closed for replies.
Correct answer -Rozi-

Hi Dirk,

I dropped the idea of my own grammar thread. It was way too unstable and no synchronization mechanism to access story and other interfaces in a thread-safe manner known to me.

I am playing with IIdleTask to perform background grammar proofing now. It looks promising. Actually, I already managed to do the real proofing and populate the grammar strand with true results.

The idea with IIdleTask is to process only the first dirty sentence in IIdleTask:: RunTask (), store results to my grammar strand (not using commands as you suggested - works fine), then return control to InDesign. InDesign calls IIdleTask:: RunTask () again and again when idle, so the text gets proofed to the end eventually. Once no more sentences to check, the idle task uninstalls itself.

I made the idle task a part of my grammar strand boss. The grammar strand implementation is non-persistent. It has internal state - markers for dirty/correct/incorrect sections of text - but it does not save to or load from the database.

Grammar strand’s constructor marks entire text dirty and installs the idle task. Each time the grammar strand is notified of the text change via IStrand::Insert(), IStrand::Cut(), or IStrand::Paste(), it uninstalls the idle task, invalidates affected markers, and reinstalls the idle task to (re)start proofing.

However, there are some issues with this approach I still need to address…

My idle task focuses on the main story thread for the time being. I need to add detection of other threads (footnotes, table cells etc.) to grammar proof them too. I don’t estimate many problems with this.

Undo/redo is not working properly. Probably because my grammar strand boss implements IID_IPMPERSIST as every strand has to, but it doesn’t use persistence to save its state at all. If it’ll make InDesign happy, then I’ll save grammar markers, together with grammar settings and grammar engine version. In case grammar settings or engine version is different on load, I’ll invalidate entire strand to make idle task re-proof entire text. A positive side effect would be a quick load of already proofed document in case grammar settings and engine did not change. I shall report on my progress about this.

I have a couple of issues with the UI part too…

I was experimenting with RtMouseText and dynamic menu already. The problem is I do not know how to get coordinates of the right-mouse-click or even better a story and a TextIndex where the click occurred. Probably, I shall add my own event watcher to intercept right mouse click, save the location of the click and use that location later when preparing dynamic menu in RtMouseText. However, rather than using RtMouseText I was thinking of adding my own popup menu: like dynamic spelling does when user right-clicks a misspelled word for seamless user experience. Mindsteam’s grammar checker does this too.

Furthermore, I need to address some issues with adornment refreshing. Consider the following scenario: User types a sentence then stops typing. After 1000ms idle task starts proofing it. However UI installs another delayed idle task to invalidate text and make InDesign refresh it. If grammar results are late, the text is refreshed too soon. I copied the adornment drawing and refreshing mechanism from SpellingPanel and I shall modify it not to rely on timers but rather make it observe changes in the grammar strand.

I am sorry for being so long, but it’s a complex issue. I really appreciate any of your past and future comments. My knowledge of InDesign is still insufficient.

Best regards,

Simon


My grammar engine is practically finished. This was a real challenge.

Background grammar processing:

- I must not grammar proof inside idle task, as it blocks other idle tasks badly.

- I've created my own boss to host as many worker threads as there are cores in the computer. All worker threads are spawned with an idle priority. No worker thread can touch ANY of the InDesign interfaces or suffer severe failure.

- Therefore, I've created an idle task for each story. It checks for a free worker thread, finds any pending paragraphs of text within own story, makes a COPY of the text, story UIDRef, and paragraph bounds and spawns a worker thread. When the worker thread is already processing the story's paragraph, the idle task checks if the thread has finished. When finished, idle task copies the result from worker thread to a grammar strand.

- In case the grammar strand receives any story update, it notifies worker thread's boss to either abort the checking of paragraph if the update hit it, or adjust paragraph bounds if the update hit before the paragraph being checked, so the grammar results are inserted to a grammar strand after the proofing is complete at correct location.

It was a bad idea to make the idle task part of my grammar strand boss. InDesign and InCopy had issues with heap corruption in Adobe's code. After I moved idle task (and observer to install/uninstall it) to a text model boss, the problems were gone.

I had to make my grammar strand fully persistent to support undo/redo operation seamlessly. This makes the grammar data to get saved to and loaded from the document too. Since grammar checking is so slow, this is a welcomed speed up when re-opening files. Of course if grammar settings or engine version didn't change meanwhile.

Under a time pressure, I dismissed the idea of having a custom right click menu like SpellingPanel has. I used RtMouseText and dynamic menu technique..

The redrawing of grammar adornmets was solved by using ISubject::ModelChange() on a document. I even took time to implement custom cookie, describing which story and part of text needs a redraw. UI attaches an observer and invalidates the part of text needing a refresh.

Best regards,

Simon

2 replies

mindsteam
mindsteam
Participating Frequently
October 14, 2014

There are some grammar checkers already available for InDesign. Is there any vendor to build the grammar checking into InDesign in such a way, a third-party plugin could add another language support to it using a simple approach?

Hello Rozi,

We could probably provide # 4 as a solution. Basically we could supply an API for the grammar service which you implement, and our engine handles the rest.

Best regards,

Heath Horton

Mindsteam Software

    -
    -Rozi-Author
    Inspiring
    October 15, 2014

    Dear Mr Horton,

    I am honoured to read your response. I see your name all over the InDesign SDK source code, assuming you must be one of the original InDesign spelling developers if not the very one. The #1 expert in the world for the exact subject I am working on.

    We are very interested in any kind of cooperation with Mindsteam. I will send you my e-mail address as a private message in a minute. Please contact me to continue our business discussion.

     

    Best regards,

    Simon Rozman

    Amebis, Slovenia

    D
    Dirk Becker
    Legend
    September 26, 2014

    I can only comment from an "outside" point - I haven't yet done anything at the guts of linguistics.

    IEventWatcher is about user interface, so it is correctly placed in a UI plugin. Unfortunately it won't catch all changes to the text. There can be edits by script, drag&drop, copy&paste, delete via menu action, updating links, changing text variables (if you consider those), the user may rebuild an index story, place XML and so forth. A better way to watch for text changes would be a text edit post process, as just recently mentioned in another thread.

    When you move your code into a separate thread, use your own memory management and communicate with anything owned by InDesign via IdleTask and so forth. Keep any persistent changes in the main thread.

    I'd also have a look at custom preflight rules.

    I haven't seen any documentation how to create an own strand, or whether it is possible at all. When I asked here, I only got silence. Since then I haven't spent time on it. Anyway a private strand sounds like a reasonable approach, *if* you get it going. Document storage that bypasses regular commands the same way as wax does. You will probably burn some (paid) support cases on the way - if you get any qualified support at that level of detail at all. Same answer as for 2) - ask Adobe directly. I think ages ago I have asked about the likelihood of a qualified answer to exactly that strands question, but not through a formal support case, so I don't want to discourage you. Just give it a try.

    Regarding the debug build - the Mac has some accidental symbols, but otherwise there are "only" tons of asserts, trace output and so forth. You can breakpoint and step through Adobe's code, but without sources that's limited fun. Symbols are provided again with another separate package, keyword "ITK". Minimum requirement (at least a while ago) was silver partner level.

    Dirk

      -
      -Rozi-Author
      Inspiring
      October 9, 2014

      Thank you for your reply, Dirk.

      Meanwhile, I’ve managed to successfully create my own strand to keep a list of grammar errors. Since it gets notifications about story changes, it can update the list accordingly. For the time being, I fill the strand with fictional grammar errors for testing.

      I’ve also made two services of type kGlobalTextAdornmentServiceImpl and kGalleyTextAdornmentAfterServiceImpl to mark grammar errors in text. My strand provides the data.

      Next, I am attempting to launch an idle task for each story to perform true grammar proofing and send a list of grammar errors to my strand using commands.

       

      Another problem arose along the way: How to capture right-mouse-click on the text to display custom pop-up menu with grammar options? I thought I’d find an example for this in the SpellingPanel plug-in. However, it is located in GalleyUI and Text Editor plug-ins. Unfortunately, their source code is not provided with the SDK.

      P.S.: Adobe Support was no help. They don’t do CS5 anymore.

      D
      Dirk Becker
      Legend
      October 9, 2014

      As already mentioned previously, for asynchronous work on a strand I would first try without commands - in a similar manner to the composer's bypassing the command system when it stores wax.

      The right mouse click is already in use by regular menus. Probably you mean to just add your menu items to the existing "RtMouseText:" or "RtMouseLayout:" as you would also add them to "Main:&Type:" ...

      Regarding Adobe tech support - many concepts within InDesign remain the same across the versions, so it may be sufficient if you ask your questions without that magic "CS5" ;-) On the other hand I don't bother them too frequently.

        Terms & ConditionsAccessibility statement
        Using the community Terms of use Privacy Cookie preferences Do not sell or share my personal information ad choicesAdChoices