FCKEditor has recently released a new version called CKEditor. Unfortunately much of the documentation has not been updated which makes building things on top of the API somewhat difficult at the moment.
I have published a basic template which make it clear how to create a plugin that adds an item to the CKEditor toolbar, when that button is pressed the users is presented with a simple dialog box containing two tabs which can interact with the parent editor.