First, I suspect you didn't use a high quality brush, however, any brush is going to leave noticeable marks. You may have even used the wrong type of brush for the type of paint.
Second, I believe that there are deglossing agents that can be used to pull some of the shine off the surface of the paint without harming it. Short of that, as long as the paint is in good condition with good adhesion, there is no reason that you can't apply another coat of paint with a lesser sheen over what's already there, just lightly sand first to smooth the existing brush marks and roughen the surface for better adhesion of new paint layers. In most cases, there is no need to strip a surface to apply new paint, that is usually reserved for multiple layers of globbed on paint or failed paint.
Third, a roller is great for getting the paint onto the surface, but it's going to leave behind a stippled pattern that will look and feel rough. While it's more a personal preference, IMHO, brush strokes look better than stipple marks on smooth surfaces.
Fourth, there are additives for paint that slow the drying process, allowing it more time to settle out the brush marks before it sets, resulting in a smoother finish.
Fifth, if you want a truly smooth surface, you will have to spray the paint on, not manually apply it with a brush or roller.
I suffer from CDO ... Its like OCD, but in alphabetical order, LIKE IT SHOULD BE!!!