System.AccessViolationException means that you're trying to read/write from unmanaged memory that is not accessible to your application. If you're familiar with pointers and C++, I'm sure you've had access violations. (If you don't know about pointers, this might be unclear - let me know and I'll give a better explanation).
In managed C#, you never say explicitly which part of memory you want to read from - you just tell which object reference you want to use, and you're given the correct data from memory. This can sometimes be slower than using unmanaged code. My guess is that when Emgu.CV uses unmanaged code to read video files.
When reading .mp4 files, it might read too far. I'm not sure why it would do that - it could be a bug in the way Emgu.CV deals with compression.
So why does this happen only occasionally? I can guess: the important thing is, AccessViolationException happens only when you read from memory that you are not allowed to access. If you accidently access memory that you did not intend to access, it could find that the piece of memory is inaccessible so you get an AccessViolationException. But it can also find that some other part of the program is using that piece of memory, and then reading from it is fine!
Thus, my guess is that if you open the .mpg file first, you have to allocate a lot of memory. Then when opening .mp4 file, Emgu.CV happens to read from the wrong place in memory - but that memory is already accessible by the .mpg file, so it doesn't throw the same exception.
Unfortunately, this does not directly address your problem. I hope it helps shed some light on the problem, and maybe from this you can find out if there is a bug someplace else?
There is one thing you could check. If you're able to load the .mp4 successfully, are you also able to save both files? If so, are the saved .mp4 and .mpg files exactly the same as the ones you opened, or are they corrupted somehow? If the files become corrupted, it could support the notion that loading the .mp4 accesses bad memory. That would mean loading the .mp4 is bugged, also when it doesn't throw the AccessViolationException.