There are two ways to prevent a window from appearing on the shell's taskbar and in the task list window that appears when you press ALT+TAB.
Give the window the WS_EX_TOOLWINDOW extended style, and remove the WS_EX_APPWINDOW style. As a side effect, the window will have a smaller caption than a normal window.
Give the window the WS_POPUP style and make it owned by a hidden window.