They are rather selective in their outrage as well, seeing as very few of them protested about the leader of the totalitarian Islamist group Hizb Ut Tahrir speaking at the Oxford Union earlier this year.
If any society which I belonged to invited a liar and fraud like the Hitler enthusiast David Irving to be a guest speaker then I would be inclined to do as some OU members have done and resign, I'm less well informed about Nick Griffin so I don't have a firm view on him. However letting Irving speak is not going to result in a big upturn for the neo nazi movement. Totalitarian movements don't gain support as a result of their well reasoned arguments or the eloquence of their propaganda. It is the element of coercion that make these movements dangerous as the late philosopher Eric Hoffer said sometime in the True Believer:
"Were propaganda by itself one tenth as potent as it is made out to be, the totalitarian regimes of Russia, Germany, Italy, and Spain would have been mild affairs. They would have been blatant and brazen but without the ghastly brutality of secret police, concentration camps, and masss extermination."This applies equally to the same movements before they took power, where the coercion took the form of Brownshirts and Redshirts acting as quasi-military wings for the various groups. To prevent the rise of fascists, communists or islamists the focus should not be on preventing them from speaking or even on making sure that their arguments are defeated (in practice the likes of Irving or Galloway are often highly eloquent and capable of winning debates even when they are wrong). Instead the role of the state is preventing them from exerting any physical threat.