In Lisp, a variable can remember one piece of data, that is, one Lisp object. The main operations on a variable are to recover that object and to alter the variable to remember a new object; these operations are often called access and update operations. The concept of variables named by symbols can be generalized to any storage location that can remember one piece of data, no matter how that location is named. Examples of such storage locations are the car and cdr of a cons, elements of an array, and components of a structure.
For each kind of generalized variable, typically there are two functions that implement the conceptual access and update operations. For a variable, merely mentioning the name of the variable accesses it, while the setq special operator can be used to update it. The function car accesses the car of a cons, and the function rplaca updates it. The function symbol-value accesses the dynamic value of a variable named by a given symbol, and the function set updates it.
Rather than thinking about two distinct functions that respectively access and update a storage location somehow deduced from their arguments, we can instead simply think of a call to the access function with given arguments as a name for the storage location. Thus, just as x may be considered a name for a storage location (a variable), so (car x) is a name for the car of some cons (which is in turn named by x). Now, rather than having to remember two functions for each kind of generalized variable (having to remember, for example, that rplaca corresponds to car), we adopt a uniform syntax for updating storage locations named in this way, using the setf macro. This is analogous to the way we use the setq special operator to convert the name of a variable (which is also a form that accesses it) into a form that updates it. The uniformity of this approach is illustrated in the following table.
| Access Function | Update Function | Update Using setf |
| x | (setq x datum) | (setf x datum) |
| (car x) | (rplaca x datum) | (setf (car x) datum) |
| (symbol-value x) | (set x datum) | (setf (symbol-value x) datum) |
setf is actually a macro that examines an access form and produces a call to the corresponding update function.
Given the existence of setf in Common Lisp, it is not necessary to have setq, rplaca, and set; they are redundant. They are retained in Common Lisp because of their historical importance in Lisp. However, most other update functions (such as putprop, the update function for get) have been eliminated from Common Lisp in the expectation that setf will be uniformly used in their place.
(setf place newvalue) takes a form place that when evaluated accesses a data object in some location and “inverts” it to produce a corresponding form to update the location. A call to the setf macro therefore expands into an update form that stores the result of evaluating the form newvalue into the place referred to by the access form.
If more than one place-newvalue pair is specified, the pairs are processed sequentially; that is,
is precisely equivalent to
For consistency, it is legal to write (setf), which simply returns nil.
The form place may be any one of the following:
This rule applies only when the function name refers to a global function definition and not to a locally defined function or macro.
This rule applies only when the function name refers to a global function definition and not to a locally defined function or macro.
is of the specified type so that it can be used to replace the specified “location” (which is in each of these cases not truly a generalized variable):
X3J13 voted in March 1989 to eliminate the type string-char and to redefine string to be the union of one or more specialized vector types, the types of whose elements are subtypes of the type character. In the preceding table, the type string-char should be replaced by some such phrase as “the element-type of the argument vector.”
X3J13 voted in March 1989 to clarify that this rule applies only when the function name refers to a global function definition and not to a locally defined function or macro.
In the case of subseq, the replacement value must be a sequence whose elements may be contained by the sequence argument to subseq. (Note that this is not so stringent as to require that the replacement value be a sequence of the same type as the sequence of which the subsequence is specified.) If the length of the replacement value does not equal the length of the subsequence to be replaced, then the shorter length determines the number of elements to be stored, as for the function replace.
| Function Name | Argument That Is a place | Update Function Used |
| ldb | second | dpb |
| mask-field | second | deposit-field |
This rule applies only when the function name refers to a global function definition and not to a locally defined function or macro.
is processed as if it were
The setf method for the function name must be such that
expands into a store form
That is, it must expand into a function call such that all arguments but the last may be any permutation or subset of the new value z0 and the arguments of the access form, but the last argument of the storing call must be the same as the last argument of the access call. See define-setf-method for more details on accessing and storing forms.
Given this, the setf-of-apply form shown above expands into
As an example, suppose that the variable indexes contains a list of subscripts for a multidimensional array foo whose rank is not known until run time. One may access the indicated element of the array by writing
and one may alter the value of the indicated element to that of newvalue by writing
This rule applies only when the function name apply refers to the global function definition and not to a locally defined function or macro named apply.
This step uses macroexpand-1, not macroexpand. This allows the chance to apply any of the rules preceding this one to any of the intermediate expansions.
This rule applies only when the function name in the form refers to a global function definition and not to a locally defined function or macro.
X3J13 voted in March 1989 to add one more rule to the preceding list, coming after all those listed above:
expands into a form with the same effect and value as
By convention, any function named (setf f ) should return its first argument as its only value, in order to preserve the specification that setf returns its newvalue.
X3J13 voted in March 1989 to add this case as well:
setf carefully arranges to preserve the usual left-to-right order in which the various subforms are evaluated. On the other hand, the exact expansion for any particular form is not guaranteed and may even be implementation-dependent; all that is guaranteed is that the expansion of a setf form will be an update form that works for that particular implementation, and that the left-to-right evaluation of subforms is preserved.
The ultimate result of evaluating a setf form is the value of newvalue. Therefore (setf (car x) y) does not expand into precisely (rplaca x y), but into something more like
the precise expansion being implementation-dependent.
The user can define new setf expansions by using defsetf.
X3J13 voted in June 1989 to extend the specification of setf to allow a place whose setf method has more than one store variable (see define-setf-method). In such a case as many values are accepted from the newvalue form as there are store variables; extra values are ignored and missing values default to nil, as is usual in situations involving multiple values.
A proposal was submitted to X3J13 in September 1989 to add a setf method for values so that one could in fact write, for example,
but unless this proposal is accepted users will have to define a setf method for values themselves (not a difficult task).
psetf is like setf except that if more than one place-newvalue pair is specified, then the assignments of new values to places are done in parallel. More precisely, all subforms that are to be evaluated are evaluated from left to right; after all evaluations have been performed, all of the assignments are performed in an unpredictable order. (The unpredictability matters only if more than one place form refers to the same place.) psetf always returns nil.
X3J13 voted in June 1989 to extend the specification of psetf to allow a place whose setf method has more than one store variable (see define-setf-method). In such a case as many values are accepted from the newvalue form as there are store variables; extra values are ignored and missing values default to nil, as is usual in situations involving multiple values.
Each place form may be any form acceptable as a generalized variable to setf. In the form (shiftf place1 place2 ... placen newvalue), the values in place1 through placen are accessed and saved, and newvalue is evaluated, for a total of n + 1 values in all. Values 2 through n + 1 are then stored into place1 through placen, and value 1 (the original value of place1) is returned. It is as if all the places form a shift register; the newvalue is shifted in from the right, all values shift over to the left one place, and the value shifted out of place1 is returned. For example:
The effect of (shiftf place1 place2 ... placen newvalue) is equivalent to
except that the latter would evaluate any subforms of each place twice, whereas shiftf takes care to evaluate them only once. For example:
Moreover, for certain place forms shiftf may be significantly more efficient than the prog1 version.
X3J13 voted in June 1989 to extend the specification of shiftf to allow a place whose setf method has more than one store variable (see define-setf-method). In such a case as many values are accepted from the newvalue form as there are store variables; extra values are ignored and missing values default to nil, as is usual in situations involving multiple values.
Rationale: shiftf and rotatef have been included in Common Lisp as generalizations of two-argument versions formerly called swapf and exchf. The two-argument versions have been found to be very useful, but the names were easily confused. The generalization to many argument forms and the change of names were both inspired by the work of Suzuki [47], which indicates that use of these primitives can make certain complex pointer-manipulation programs clearer and easier to prove correct.
Each place form may be any form acceptable as a generalized variable to setf. In the form (rotatef place1 place2 ... placen), the values in place1 through placen are accessed and saved. Values 2 through n and value 1 are then stored into place1 through placen. It is as if all the places form an end-around shift register that is rotated one place to the left, with the value of place1 being shifted around the end to placen. Note that (rotatef place1 place2) exchanges the contents of place1 and place2.
The effect of (rotatef place1 place2 ... placen) is roughly equivalent to
except that the latter would evaluate any subforms of each place twice, whereas rotatef takes care to evaluate them only once. Moreover, for certain place forms rotatef may be significantly more efficient.
X3J13 voted in June 1989 to extend the specification of rotatef to allow a place whose setf method has more than one store variable (see define-setf-method). In such a case as many values are accepted from the newvalue form as there are store variables; extra values are ignored and missing values default to nil, as is usual in situations involving multiple values.
Other macros that manipulate generalized variables include getf, remf, incf, decf, push, pop, assert, ctypecase, and ccase.
Macros that manipulate generalized variables must guarantee the “obvious” semantics: subforms of generalized-variable references are evaluated exactly as many times as they appear in the source program, and they are evaluated in exactly the same order as they appear in the source program.
In generalized-variable references such as shiftf, incf, push, and setf of ldb, the generalized variables are both read and written in the same reference. Preserving the source program order of evaluation and the number of evaluations is particularly important.
As an example of these semantic rules, in the generalized-variable reference (setf reference value) the value form must be evaluated after all the subforms of the reference because the value form appears to the right of them.
The expansion of these macros must consist of code that follows these rules or has the same effect as such code. This is accomplished by introducing temporary variables bound to the subforms of the reference. As an optimization in the implementation, temporary variables may be eliminated whenever it can be proved that removing them has no effect on the semantics of the program. For example, a constant need never be saved in a temporary variable. A variable, or for that matter any form that does not have side effects, need not be saved in a temporary variable if it can be proved that its value will not change within the scope of the generalized-variable reference.
Common Lisp provides built-in facilities to take care of these semantic complications and optimizations. Since the required semantics can be guaranteed by these facilities, the user does not have to worry about writing correct code for them, especially in complex cases. Even experts can become confused and make mistakes while writing this sort of code.
X3J13 voted in March 1988 to clarify the preceding discussion about the order of evaluation of subforms in calls to setf and related macros. The general intent is clear: evaluation proceeds from left to right whenever possible. However, the left-to-right rule does not remove the obligation on writers of macros and define-setf-method to work to ensure left-to-right order of evaluation.
Let it be emphasized that, in the following discussion, a form is something whose syntactic use is such that it will be evaluated. A subform means a form that is nested inside another form, not merely any Lisp object nested inside a form regardless of syntactic context.
The evaluation ordering of subforms within a generalized variable reference is determined by the order specified by the second value returned by get-setf-method. For all predefined generalized variable references (getf, ldb), this order of evaluation is exactly left-to-right. When a generalized variable reference is derived from a macro expansion, this rule is applied after the macro is expanded to find the appropriate generalized variable reference.
This is intended to make it clear that if the user writes a defmacro or define-setf-method macro that doesn’t preserve left-to-right evaluation order, the order specified in the user’s code holds. For example, given
then
will evaluate place2 first and then place1 because that is the order they are evaluated in the macro expansion.
For the macros that manipulate generalized variables (push, pushnew, getf, remf, incf, decf, shiftf, rotatef, psetf, setf, pop, and those defined with define-modify-macro) the subforms of the macro call are evaluated exactly once in left-to-right order, with the subforms of the generalized variable references evaluated in the order specified above.
Each of push, pushnew, getf, remf, incf, decf, shiftf, rotatef, psetf, and pop evaluates all subforms before modifying any of the generalized variable locations. Moreover, setf itself, in the case when a call on it has more than two arguments, performs its operation on each pair in sequence. That is, in
the subforms of place1 and value1 are evaluated, the location specified by place1 is modified to contain the value returned by value1, and then the rest of the setf form is processed in a like manner.
For the macros check-type, ctypecase, and ccase, subforms of the generalized variable reference are evaluated once per test of a generalized variable, but they may be evaluated again if the type check fails (in the case of check-type) or if none of the cases holds (in ctypecase or ccase).
For the macro assert, the order of evaluation of the generalized variable references is not specified.
Another reason for building in these functions is that the appropriate optimizations will differ from implementation to implementation. In some implementations most of the optimization is performed by the compiler, while in others a simpler compiler is used and most of the optimization is performed in the macros. The cost of binding a temporary variable relative to the cost of other Lisp operations may differ greatly between one implementation and another, and some implementations may find it best never to remove temporary variables except in the simplest cases.
A good example of the issues involved can be seen in the following generalized-variable reference:
This ought to expand into something like
In this expansion example we have ignored the further complexity of returning the correct value, which is the incremented byte, not the new value of variable. Note that the variable byte-field is evaluated twice, and the variable variable is referred to three times: once as the location in which to store a value, and twice during the computation of that value.
Now consider this expression:
It ought to expand into something like this:
Again we have ignored the complexity of returning the correct value. What is important here is that the expressions (incf i) and (determine-words-array) must not be duplicated because each may have a side effect or be affected by side effects.
X3J13 voted in January 1989 to specify more precisely the order of evaluation of subforms when setf is used with an access function that itself takes a place as an argument, for example, ldb, mask-field, and getf. (The vote also discussed the function char-bit, but another vote removed that function from the language.) The setf methods for such accessors produce expansions that effectively require explicit calls to get-setf-method.
The code produced as the macro expansion of a setf form that itself admits a generalized variable as an argument must essentially do the following major steps:
Doing the access for a generalized variable reference is not part of the series of evaluations that must be done in left-to-right order.
The place-specifier forms ldb, mask-field, and getf admit (other) place specifiers as arguments. During the setf expansion of these forms, it is necessary to call get-setf-method to determine how the inner, nested generalized variable must be treated.
In a form such as
the place referred to by the place-form must always be both accessed and updated; note that the update is to the generalized variable specified by place-form, not to any object of type integer.
Thus this call to setf should generate code to do the following:
If the evaluation of newvalue-form alters what is found in the given place—such as setting a different bit-field of the integer—then the change of the bit-field denoted by byte-spec will be to that altered integer, because the access step must be done after the newvalue-form evaluation. Nevertheless, the evaluations required for binding the temporaries are done before the evaluation of the newvalue-form, thereby preserving the required left-to-right evaluation order.
The treatment of mask-field is similar to that of ldb.
In a form such as:
the place referred to by the place-form must always be both accessed and updated; note that the update is to the generalized variable specified by place-form, not necessarily to the particular list which is the property list in question.
Thus this call to setf should generate code to do the following:
If the evaluation of newvalue-form alters what is found in the given place—such as setting a different named property in the list—then the change of the property denoted by ind-form will be to that altered list, because the access step is done after the newvalue-form evaluation. Nevertheless, the evaluations required for binding the temporaries are done before the evaluation of the newvalue-form, thereby preserving the required left-to-right evaluation order.
Note that the phrase “possibly new property list” treats the implementation of property lists as a “black box”; it can mean that the former property list is somehow destructively re-used, or it can mean partial or full copying of it. A side effect may or may not occur; therefore setf must proceed as if the resultant property list were a different copy needing to be stored back into the generalized variable.
The Common Lisp facilities provided to deal with these semantic issues include:
Also important are the changes that allow lexical environments to be used in appropriate ways in setf methods.
This macro defines a read-modify-write macro named name. An example of such a macro is incf. The first subform of the macro will be a generalized-variable reference. The function is literally the function to apply to the old contents of the generalized-variable to get the new contents; it is not evaluated. lambda-list describes the remaining arguments for the function; these arguments come from the remaining subforms of the macro after the generalized-variable reference. lambda-list may contain &optional and &rest markers. (The &key marker is not permitted here; &rest suffices for the purposes of define-modify-macro.) doc-string is documentation for the macro name being defined.
The expansion of a define-modify-macro is equivalent to the following, except that it generates code that follows the semantic rules outlined above.
where arg1, arg2, ..., are the parameters appearing in lambda-list; appropriate provision is made for a &rest parameter.
As an example, incf could have been defined by:
An example of a possibly useful macro not predefined in Common Lisp is
X3J13 voted in March 1988 to specify that define-modify-macro creates macros that take &environment arguments and perform the equivalent of correctly passing such lexical environments to get-setf-method in order to correctly maintain lexical references.
This defines how to setf a generalized-variable reference of the form (access-fn ...). The value of a generalized-variable reference can always be obtained simply by evaluating it, so access-fn should be the name of a function or a macro.
The user of defsetf provides a description of how to store into the generalized-variable reference and return the value that was stored (because setf is defined to return this value). The implementation of defsetf takes care of ensuring that subforms of the reference are evaluated exactly once and in the proper left-to-right order. In order to do this, defsetf requires that access-fn be a function or a macro that evaluates its arguments, behaving like a function. Furthermore, a setf of a call on access-fn will also evaluate all of access-fn’s arguments; it cannot treat any of them specially. This means that defsetf cannot be used to describe how to store into a generalized variable that is a byte, such as (ldb field reference). To handle situations that do not fit the restrictions imposed by defsetf, use define-setf-method, which gives the user additional control at the cost of increased complexity.
A defsetf declaration may take one of two forms. The simple form is
The update-fn must name a function (or macro) that takes one more argument than access-fn takes. When setf is given a place that is a call on access-fn, it expands into a call on update-fn that is given all the arguments to access-fn and also, as its last argument, the new value (which must be returned by update-fn as its value). For example, the effect of
is built into the Common Lisp system. This causes the expansion
for example. Note that
would be incorrect because rplaca does not return its last argument.
The complex form of defsetf looks like
and resembles defmacro. The body must compute the expansion of a setf of a call on access-fn.
The lambda-list describes the arguments of access-fn. &optional, &rest, and &key markers are permitted in lambda-list. Optional arguments may have defaults and “supplied-p” flags. The store-variable describes the value to be stored into the generalized-variable reference. _____________________________________
Rationale: The store-variable is enclosed in parentheses to provide for an extension to multiple store variables that would receive multiple values from the second subform of setf. The rules given below for coding setf methods discuss the proper handling of multiple store variables to allow for the possibility that this extension may be incorporated into Common Lisp in the future.
The body forms can be written as if the variables in the lambda-list were bound to subforms of the call on access-fn and the store-variable were bound to the second subform of setf. However, this is not actually the case. During the evaluation of the body forms, these variables are bound to names of temporary variables, generated as if by gensym or gentemp, that will be bound by the expansion of setf to the values of those subforms. This binding permits the body forms to be written without regard for order-of-evaluation issues. defsetf arranges for the temporary variables to be optimized out of the final result in cases where that is possible. In other words, an attempt is made by defsetf to generate the best code possible in a particular implementation.
Note that the code generated by the body forms must include provision for returning the correct value (the value of store-variable). This is handled by the body forms rather than by defsetf because in many cases this value can be returned at no extra cost, by calling a function that simultaneously stores into the generalized variable and returns the correct value.
An example of the use of the complex form of defsetf:
X3J13 voted in March 1988 to specify that the body of the expander function defined by the complex form of defsetf is implicitly enclosed in a block construct whose name is the same as the name of the access-fn. Therefore return-from may be used to exit from the function.
X3J13 voted in March 1989 to clarify that, while defining forms normally appear at top level, it is meaningful to place them in non-top-level contexts; the complex form of defsetf must define the expander function within the enclosing lexical environment, not within the global environment.
The underlying theory by which setf and related macros arrange to conform to the semantic rules given above is that from any generalized-variable reference one may derive its “setf method,” which describes how to store into that reference and which subforms of it are evaluated.
Given knowledge of the subforms of the reference, it is possible to avoid evaluating them multiple times or in the wrong order. A setf method for a given access form can be expressed as five values:
The temporary variables will be bound to the values of the value forms as if by let*; that is, the value forms will be evaluated in the order given and may refer to the values of earlier value forms by using the corresponding variables.
The store variables are to be bound to the values of the newvalue form, that is, the values to be stored into the generalized variable. In almost all cases only a single value is to be stored, and there is only one store variable.
The storing form and the accessing form may contain references to the temporary variables (and also, in the case of the storing form, to the store variables). The accessing form returns the value of the generalized variable. The storing form modifies the value of the generalized variable and guarantees to return the values of the store variables as its values; these are the correct values for setf to return. (Again, in most cases there is a single store variable and thus a single value to be returned.) The value returned by the accessing form is, of course, affected by execution of the storing form, but either of these forms may be evaluated any number of times and therefore should be free of side effects (other than the storing action of the storing form).
The temporary variables and the store variables are generated names, as if by gensym or gentemp, so that there is never any problem of name clashes among them, or between them and other variables in the program. This is necessary to make the special operators that do more than one setf in parallel work properly; these are psetf, shiftf, and rotatef. Computation of the setf method must always create new variable names; it may not return the same ones every time.
Some examples of setf methods for particular forms:
This defines how to setf a generalized-variable reference that is of the form (access-fn...). The value of a generalized-variable reference can always be obtained simply by evaluating it, so access-fn should be the name of a function or a macro.
The lambda-list describes the subforms of the generalized-variable reference, as with defmacro. The result of evaluating the forms in the body must be five values representing the setf method, as described above. Note that define-setf-method differs from the complex form of defsetf in that while the body is being executed the variables in lambda-list are bound to parts of the generalized-variable reference, not to temporary variables that will be bound to the values of such parts. In addition, define-setf-method does not have defsetf’s restriction that access-fn must be a function or a function-like macro; an arbitrary defmacro destructuring pattern is permitted in lambda-list.
By definition there are no good small examples of define-setf-method because the easy cases can all be handled by defsetf. A typical use is to define the setf method for ldb:
X3J13 voted in March 1988 to specify that the &environment lambda-list keyword may appear in the lambda-list in the same manner as for defmacro in order to obtain the lexical environment of the call to the setf macro. The preceding example should be modified to take advantage of this new feature. The setf method must accept an &environment parameter, which will receive the lexical environment of the call to setf; this environment must then be given to get-setf-method in order that it may correctly use any locally bound setf method that might be applicable to the place form that appears as the second argument to ldb in the call to setf.
X3J13 voted in March 1988 to specify that the body of the expander function defined by define-setf-method is implicitly enclosed in a block construct whose name is the same as the name of the access-fn. Therefore return-from may be used to exit from the function.
X3J13 voted in March 1989 to clarify that, while defining forms normally appear at top level, it is meaningful to place them in non-top-level contexts; define-setf-method must define the expander function within the enclosing lexical environment, not within the global environment.
X3J13 voted in March 1988 to add an optional environment argument to get-setf-method. The revised definition and example are as follows.
get-setf-method returns five values constituting the setf method for form. The form must be a generalized-variable reference. The env must be an environment of the sort obtained through the &environment lambda-list keyword; if env is nil or omitted, the null lexical environment is assumed. get-setf-method takes care of error checking and macro expansion and guarantees to return exactly one store variable.
As an example, an extremely simplified version of setf, allowing no more and no fewer than two subforms, containing no optimization to remove unnecessary variables, and not allowing storing of multiple values, could be defined by:
X3J13 voted in March 1988 to add an optional environment argument to get-setf-method. The revised definition is as follows.
get-setf-method-multiple-value returns five values constituting the setf method for form. The form must be a generalized-variable reference. The env must be an environment of the sort obtained through the &environment lambda-list keyword; if env is nil or omitted, the null lexical environment is assumed.
This is the same as get-setf-method except that it does not check the number of store variables; use this in cases that allow storing multiple values into a generalized variable. There are no such cases in standard Common Lisp, but this function is provided to allow for possible extensions.
X3J13 voted in March 1988 to clarify that a setf method for a functional name is applicable only when the global binding of that name is lexically visible. If such a name has a local binding introduced by flet, labels, or macrolet, then global definitions of setf methods for that name do not apply and are not visible. All of the standard Common Lisp macros that modify a setf place (for example, incf, decf, pop, and rotatef) obey this convention.