What exactly does the size of a layer mean?

I understand the pixel dimensions of a layer to be the smallest rectangle that encloses the most extreme non-transparent pixels of a layer. In non-math terms, it’s the size of the rectangle you see when you make the transform bounding box visible, by choosing Edit > Free Transform or enabling Show Transform Controls in the options bar when the Move tool is selected. If there are non-transparent pixels outside the canvas, the transform bounding box does enclose those.

The pixel dimensions of each layer have a direct effect on the document file size. Uniform areas of a pixel layer are very easy to compress. So if you have two documents of the same pixel dimensions and number of layers, where the layers in the first document don’t fill the canvas (they have smaller pixel dimensions than the document), and the same number of layers in the second document are fully painted from edge to edge, the second document will have a much larger file size.

There are different layer kinds. For a "standard" new empty artLayer - there is no size until there is content, it's just blank to the canvas size... And yes, a layer can have content that is beyond the visible canvas.

I've always understood it to be the box that contains the non-transparent pixels, like what you see when the Layers Panel Options is set to Layer Bounds for the thumbnail.