What application are you using?

The rounded corners effect works fine on an unfilled square for me in Illustrator.

humm...for me its not. i lose the corners if it unfilled. i wonder why

DO you lose the corners or can you just not see the corners?

With no fill and no stroke, you can only see the control points of the shape, which, unles you expand the effect, are still those of a square. But that doesn't mean the shape no longer has rounded corners.

And, because you haven't affected the actual shape of the box with the effect, if you use it to create a clipping mask, the clipping mask will also be square, as this is applied according to the actual shape - an effect changes the appearance of an object, not its actual shape.

If you expand the appearance of the shape then it will work fine as a clipping mask, but you will no longer be able to edit the corner radius. (You could equally do filter -> stylize -> round corners, which has the same result as applying the rounded corner effect and the expanding it.)

Oh, better yet - if you can use the pathfinder instead of a clipping mask, that will respect the rounded corners that you applied as an effect.

If you go to make a rounded corner rectangle with the tool, and you click the artboard instead of drawing the rectangle, you get a little popup that lets you set values for the radius of the corners and the size of the shape. Hope that helps.

In the Rounded Rectangle, use the up+down arrow keys as you drag to adjust the corner radii. The right+left arrow keys give you the min/max (either square corners or radius equal to the y dimension - like a capsule) radii, depending on the dimensions of the shape.

For the Stylize>Round Corners, this effect needs to be applied (or reapplied) after creating the clipping mask.