It’s not forbidden, even though our elementary school teachers told us all not to do it. But those poor teachers were desperately trying to get us to internalize the concept of complete sentences. What they told us was expedient for their purposes, not advice for sophisticated writing.
Take the second sentence of this post. Replacing “but” with “however” would not improve it. And if you made the switch, you’d be no better off, because “however” itself would be functioning as a conjunction.
Beginning sentences with “and,” “but,” and “or” can be overdone. That is not good, and it is a sin I need to personally watch out for. Still, used moderately, beginning a sentence that way can be a pithy and effective communication technique.
On some other occasion, I’ll address splitting infinitives, which I know I did in the preceding paragraph.