[UNet] Spawned GameObject's layer not syncing over network

I've been wrestling with this for a couple of days now. Basically what this is supposed to do is once the player presses "B", spawn a "ghost brick" which is essentially a representation of the actual brick that follows where the player is looking. If the player clicks, it will create an actual brick at the location of the ghost brick. This works perfectly fine on the server, however on the client, the ray seems to interact with the ghost brick, even though the Layer Mask is supposedly...

[UNet] Spawned GameObject's layer not syncing over network