in theory, you certainly can. but the kernel development team, and linus in particular, are not interested in an engineering effort/long term approach that makes this feasible. if you define a stable driver binary interface, you can change the kernel out around it and drivers keep working. linus has made it clear that he sees no reason to do this, and is perhaps even opposed to it for some possibly sound engineering arguments (though that is open to debate).