On firing the bullet, the rifle always gives a backward kick. Why?

Both gun as well as bullet are at rest, before firing. Therefore, momentum of system is zero.
A bullet moves in the forward direction, as soon as it is fired. According to the law of conservation of momentum, total momentum of system remains conserved in the absence of external force. Therefore, momentum of the gun and bullet after fire should remain zero. This is possible, if and only if gun moves in backward direction.