InstallShield 11.5 and earlier do not officially support Vista. Lots of installations can work correctly, but it will depend on what your setup needs and what project type you use exactly what the behavior might be. There were several architectural changes to InstallShield 12 to address incompatibilities that surfaced with Vista (see the stickies at the top of the InstallShield 12 or 2008 communities), and a few cases where deferred actions for IS functionality merely had to be marked to run in "System Context".
These are all generalities. To offer better guidance, we'd need to know exactly what sort of problems you're encountering - what exactly does and doesn't happen when it "doesn't work". The simple answer from our perspective is, of course, upgrade to InstallShield 2008, as the increased reliability is well worth the upgrade pains (if any). However if you can get by with a small change in your project, that may be a better choice for you.
Michael Urman - Staff Software Engineer - Flexera Software: InstallShield Team