SPA (Single Page Application) Render client or server

Question

I have a simple yet not so easy question. (So, I suppose)

I have a current DurandalJS application in production. I am generating my views using C# Razor syntax. So, I am still using my controller to pass the HTML to the client. Rather complicated to get setup using routes/areas/etc but it has been working.

I am using server side for only one real reason. (other then validation, but we can ignore that for now as I am sure model validation has improved and there are ways to handle this now)

Lets say I have a button. And I only want to show that button on the screen if the user is an admin. Now, of course I can use KnockoutJs (or Auelia/Angular/whatever) to find out the users roles and "hide" the button. But as far as I know, the button is still inside the HTML? (it is in Durandal, not sure if the newer libraries handle this different). So, currently I use server code to do the check then the HTML gets rendered without the button.

Knowing how many people use SPA type libraries, what are most of you doing for simple use cases like this?

Oh, yes, on the server for the return API call, we would have security anyway, so assuming someone activated the button it would not work anyway, I simply do not want my users ever seeing it.