In this contribution we consider a rule based visual language for the specification of graphical user interfaces (GUIs). Our approach uses a graph rewrite approach in such a way that the state of a GUI is represented as a complex visual graph. An action is represented by a visual rewrite rule applicable to a graph of this form. Since the GUI features are represented as little pictures like windows, listboxes etc. it is necessary to define these problem oriented representations by a more abstract mathematical formalism. We use graphs to represent the semantics of the problem oriented representation. Complex actions and events are modelled by rewrite steps on these graphs. Usually complex GUIs are described by a hierarchy of interaction objects. Such a hierarchy is adequately expressed as a hierarchy of graph layers. We present how the various descriptions of picture graphs and graphs as well as visual rules and graph rewrite rules, respectively, are integrated by a method using special mappings.