I have an add-in that displays a modeless form. I have some text boxes on this form. The tab should cause the focus to switch between the controls on the form. Instead the Tab key has no effect.
Enter, ESC and CANCEL do not work in modeless dialog is as designed, since your dialog is child of Inventor MainFrame window. The message pump in this case is owned by Inventor. Therefore, all the keystroke messages are taken by the Inventor application and not dispatched to the modeless dialog box. You can workaround this by implementing windows hook WH_GETMESSAGE for modeless dialog. Please refer to following Microsoft KB article.
Or, display the form as a modal dialog using ShowDialog() instead of Show().