Yeah, the problem is when Grobyc is not in your party, there are still 5 shadows.
I figured it out, that's probably because Grobyc is out of screen...that doesn't explain anything.
------------------
Hmm, still no clue how that shadow appears since I know nothing about room script.
But the shadow doesn't appear suddenly when you talk to Luccia, it is completely transparency at first, and will become darker and darker.