I got prediction all working. I had been putting it off for a couple of months because I thought it was going to be a pain in the ass, in the end it took 2 days and wasn't that bad.
What is prediction
So a quick explanation of how prediction works in Source. The client makes a user command which is a struct that describes the inputs of the player. Basic stuff like buttons and aim angles. It sends this user command to the server and the server moves the player, shoots the gun etc.
While it's sending to the server it also runs it locally, so anything you do happens instantly. When the server runs its version and we get the results we compare them to the results our prediction got. If it all worked out the player should be in the same place, weapons should have the same amount of ammo etc.
Any variables you use during prediction need to be marked as predicted. If variables aren't marked as predicted, it could lead to further prediction errors because the client won't be able to get a good starting point where its in sync with the server.
There's nothing complicated here, it's like any other networked variable.
If there's a prediction error the client might go back and re-predict a few frames back to get everything back where it should be. When it's doing that you don't want to do things like play sounds again.
To guard against this we have Predict.FirstTime. It's always true on the server, but on the client it's only true if it's the first time predicting that frame.
When you shoot a gun you play a sound. The sound is predicted locally, but when the frame is run on the server the sound is always played. We do some magic in the background to stop that sound playing again if it was predicted by the local client.
But sometimes you're only running some code on the server. Like for example, if you shoot a bullet and then someone dies, you might want to play that sound to everyone - the local player included - because they couldn't predict that sound.
In these instances we can use Prediction.Off() to turn off prediction. It creates an IDisposable so we can easily sope it. I don't love this syntax because we still have to check whether it's the server, so it might change.