Plug-ins vs. Scripts
There are two main methods for controlling and extending modo: plug-ins and scripts. Each have their place, and are generally complementary. Which one is right for your project depends on what you intend to do.
Macros and scripts allow you to automate operations in modo by executing and querying commands. Macros are the simplest form of script, containing a series of commands that are executed in order. Scripts allow you to query commands and services for information about the scene state, as well as providing flow control, file I/O and everything else you might expect from the specific language. The currently supported languages are Perl, Python and Lua.
Scripts and macros are essentially fire-and-forget entities. A script is executed, runs to completion, and quits. Scripts do not persist, and cannot currently be used to create complex constructs like items or tools. It's best to think of scripts as macros with flow control and the ability to query scene state. Scripts can support user interfaces through the use dialogs via the dialog commands, and through User Values, either through command dialogs or Forms. User Values can also be used to store state in the configs. Since they are effectively simple commands, they obey the same limitations as commands regarding how and when they can be used.
Plug-ins are written in C++ using the Luxology SDK. They are much more versatile than scripts and more suited to performance-critical code. Plug-in can define new item types, modeling and animation tools, commands, animation modifiers, file loaders and savers, and a number of other constructs. They can also listen for and react to events, store state in the config system, and perform other more persistent task that are outside of the bounds of the scripting system.
The SDK uses C++ to wrap COM interfaces defined in C. For this reason, a basic understand of COM and reference counting is useful. While it is theoretically possible to write plug-ins in any language that supports COM, can use the Luxology SDK header files, and can be built into a compatible run-time library, in practice the Luxology-provided C++ classes hide a lot of the underlying complexity and simplify development.
Scripts vs. Plug-In Commands
Scripts can be thought of as simple commands. Plug-ins can be used to create full-fledged commands. For this reason there is a lot of overlap between the two.
A script can be easier to write, but will need to rely on other commands like User Values to store state and present UI. They also do not support being disabled in the UI, querying for a value, or notifying a Form View when it needs to update to show a new value or enable state (since they has no values to query and cannot be disabled). A script placed directly in a form appears as an always-enabled button. User Values can provide the ability to show and change values in the UI, and since changes to User Values can themselves launch scripts, scripts can be made reasonably interactive. You will often write scripts for relatively simple projects, want to write something quickly, want to easily create cross-platform kits (plug-ins need to be complied specifically for each platform), don't want to delve into the complexities of the SDK or simply prefer scripting over C++, and can work within the limitations of the scripting mechanism.
Commands provide the ability to be queried for a value, and can be put directly into the user interface to represent a variety of controls without requiring a User Value proxy. It also has full access to all of the COM interfaces as well as the ScriptQuery interfaces available to scripts. Being compiled, they run much faster than scripts, but you will need to compile a new plug-in for each platform you want to support. You would write a custom command if you want speed, want access to other parts of the application that aren't not available through the existing ScriptQuery interfaces, need to be able to query the command, need to do some other behavior that isn't possible through scripts, or simply prefer to write in C++.