Kiren Rijiju said while PM Modi was not affected by the ridicule, his cabinet colleagues were perturbed by the Leader of Opposition's conduct