I want to make something, instead of thinking about how to make it.

[Erik Svedäng]

@ Erik & badvibration - Agreed. I didn't mean "language" ( hence the quotes ) in the literal sense, but in the way a teacher guides a dancer, or kids explain a game on the street as they go.

OK. Do you have any picture of how that "language" could work/look like, practically speaking?

[Kjell]

Do you have any picture of how that "language" could work/look like, practically speaking?

Somewhat similar to how 3D applications such as Maya / Softimage let you work on the end-result ( or at least significantly closer compared to code ) in real-time while hiding all the vertex shoving, matrix popping / pushing, quaternion lerping etc. In fact, most artist aren't aware of what exactly goes on behind the scenes ( and most of the time they don't really need to anyway ).

[Erik Svedäng]

Good points.

Maybe programming a game should be more like instructing actors? With invisible "stage workers" for things that should go on behind the scenes.

[axcho]

So I've been thinking about this a lot over the last several days. Actually I've been obsessively thinking about it non-stop and reading everything I can on related topics online.

I tend to do that for a different thing every week. This week, it's been this.

So there are a few pieces I've been focusing on, that seem most crucial to the success of a programming or game development tool for artists. There are probably others, but I thought I'd share what I've been thinking about so far.

One is readability through R-mode perception.

First of all, a disclaimer: When I say "R-mode" versus "L-mode", as in "Right brain" or "Relational" or "Rtistic" versus "Left brain" or "Logical" or "Linear", I don't mean to suggest that the brain is really divided into strict differences between its physical right and left halves. That is an outdated belief. But I find the terminology to be a useful shorthand.

Thinking on Michael's comments about visual flowcharts being easier for him to read than linguistic code, and looking back on my own experience, I think there really is something significant about how the code is presented and perceived, even when the underlying logic is the same.

When I am reading code (or a book!) I am usually using what I call "L-mode" perception - going through in a linear, linguistic way, and building up my mental model one step, one line of code at a time, following the logic that is expressed symbolically, in sequences like that.

However, sometimes my mind is in an "R-mode" of all-at-once, spatial perception like you'd use for looking at a painting or trying to find a certain LEGO piece in a big box of pieces. When I am in that state of mind, and I look at code (or to a lesser extent, written language) I see all the words at once and perceive the spatial relationships between them, and the underlying logic of the code is utterly incomprehensible to me. Obviously not the "right" way to read code.

But maybe it could be.

Ha, that would be a good tagline. "The right way to code."

The thing about R-mode perception is that it's a lot easier to be creative when you're in it. The other thing about R-mode perception is that artists are usually a lot more skilled at functioning in R-mode than they are in L-mode.

Therefore, if you had a tool that let you do programming while in R-mode rather than L-mode, it might be slightly easier to do creative things with it. At the least, there would be less inefficiency caused by switching between R-mode and L-mode whenever you think about what you want to change and then have to dive into the code to actually change it.

However, this may not even be possible.

All the visual programming editors I've seen, all the examples that have been posted here, require an uncomfortable mix of L-mode and R-mode perception in order to use. What I tend to see is a bunch of visually identical boxes connected by lines, and differentiated by text.

What you see in R-mode is the set of relationships between the boxes. But you can't tell what each of the boxes does. To do that, you must read the text and think symbolically, in L-mode. Really, very little information is conveyed through spatial relationships, through R-mode. Most of it is still sequential and symbolic.

For that reason, I find that pure written code, all L-mode, is much easier for me to deal with, since I don't need to switch around multiple times a second just to figure out what everything means. However, I suspect that there may be a way to create a pure R-mode method of programming too. But I'm not confident that it's actually possible. Just intrigued enough to try.

There are some programmers who hate the idea of visual programming, and say that it's a waste of time to use spatial relationships to convey the meaning of code. If you are one of those people and you use syntax coloring or indentation, you are a hypocrite.

So there's one aspect. Make sure your tool is R-mode accessible, if you want artists to be able to use it.

The second thing is building with functional pieces in real (or almost real) time.

Artists tend to appreciate tools where "what you see is what you get" - you're manipulating the end result, so you can immediately see the results of your actions. The process becomes more like sculpting.

Programmers tend to discount such tools as nice but unnecessary. They are used to typing in code for an hour, hitting a button, and waiting a minute for everything to compile and show up on the screen.

These are two fundamentally different mindsets, as different as a slideshow and an animation.

When you operate in the slow, "slideshow" approach, development and creativity tends to happen in an architected, "top-down" way. You have a plan, which is in your head, and then you put in a bunch of time and hard work to mold reality into the shape of that plan.

There is a fundamental shift that occurs as you decrease the time between action and result. It's as real as the shift that occurs when you hit 24 frames per second - from slideshow to animation. To your brain, it's alive, it's moving.

When you operate in the immediate, "animation" way, development tends to operate in a more exploratory, "bottom-up" process. You don't have to have an entire plan in your head. You see the results of your actions immediately, and if they are surprising or unexpected, you can adjust your plan. You can try random things and follow them if they prove to be interesting.

In the area of game design, innovation is much more likely to come out of an exploratory process than an architected one. As Jonathan Blow said earlier. It's hard to do things that haven't been done before if you have to plan it all out in your head first.

So we want a tool that allows us to sculpt the end result, with immediate feedback.

Part of this is that everything you can make should work. It may not work in the way you desire or expect, but it should still do something.

If you are painting with pixels in an art program, no matter how you put those pixels down on the screen, it will always be a functional, viewable image. It might not be pretty, but you can still see it. You are never going to run into a error message that says, "Invalid pixels at position 55, 46. Image cannot be displayed."

But if you are writing the code for a program, this sort of thing happens all the time. Most of the things you can type won't work at all. They won't turn into a program, even a broken one. There are right ways to write code, and wrong ways to write it.

I would say that this also makes a big difference. Perhaps the biggest difference is that writing code has a much higher barrier to entry, more learning how to do things at all before you can start learning how to do them well. But it also makes experimentation so much more difficult. You can't throw a bunch of random stuff together just to see what happens. Because what happens is nothing. It just won't do anything at all.

So if you can build only with pieces that work, and immediately see what changes, this would make truly artistic interactive art much easier to create.

The last thing is expressing general logic through specific examples.

This is probably the most impossible and most revolutionary but least important of the three. If you just had a tool that you'd use in R-mode, that let you shape the end results with immediate feedback, that could be awesome, and probably enough to make a huge difference.

But at the same time I am intrigued by this further vision I have of providing specific examples, which the system will extrapolate to create possible general rules for creating those example, which you will then provide feedback on and refine in order to guide the system's hypotheses toward the end you have in mind.

Because I don't see how to actually avoid symbols when describing logic, or how to directly manipulate end results in a general way. Because games are systems, and the end results happen when you take the rules that you have set up and run them through their paces.

So maybe this is the only way to achieve those first two goals in their entirety.

What am I talking about?

You know how you draw diagrams and mockups for different things that happen in different situations in a game? Like this. It's a pretty common way to organize your thoughts when you're designing. The thing about those is they're all organized around specific examples, not general rules. So you might draw a diagram with a guy hitting a wall, showing how he bounces off or breaks through it or whatever. It's not completely specific, as you might have an abstract line standing in for any kind of wall, and a stick figure representing any kind of guy, but at the same time it's very concrete. And you can add general connotations by writing in little notes, to explain the rules behind the example more clearly.

The reason that we don't just stop there is that our game development tools require everything to be spelled out exactly - they cannot extrapolate from these examples, because there is so much ambiguity. It could mean this or it could mean that.

However, we run into a similar problem when trying to communicate our ideas to other people who are helping us make them into a reality. Especially if we are designers and we are telling programmers what to do. How do we solve this problem with other people?

Part of this is by clarifying with more examples when an area is unclear. Kind of starting at the highest level and breaking it down into more specific situations when necessary. Another part is through conversation, asking "It sounds like you're describing this... Is that right?" and responding "Yes, exactly!" or "No, I was thinking something more like this..."

Both of these could be accomplished with a special computer program instead of a human programmer. Maybe not as well, especially in terms of accuracy of translation, but in some ways better - particularly, in the time between your description of a design and seeing something on the screen. And this increase in speed could make up for the lack of accuracy, since you can adjust and correct much more quickly. And as a result, make use of exploratory design instead of architecture.

Break dynamics down into stories instead of rules. A playthrough of the entire game could be an example story, and you could create example stories of successively smaller and smaller pieces of the game until you have specified it completely. Or completely enough.

The tool generates possible rules that could create the situations you specify, and presents several for you to try out. Most likely none of them work the way you want. Pick the one that's closest, and let it generate more possibilities based on that. It's an evolutionary search. Like Biomorphs.

And stories don't always have to be specific stories about specific instances. They can be more or less abstract and universal. Like Raven, with a capital "R", who is both the character Raven and all ravens and all tricksters at once. Or the princess in the tower, or the wise old man, or the dragon in the cave. Or the stick figure on the crosswalk sign who represents all pedestrians who could ever walk this way. There is a continuum between the specific and the symbol.

I am particularly inspired by the concept of the Dreamtime. The translation of this name is misleading, as it does not refer to a time in history. It is like a parallel slice of the world running alongside and underneath the specific, physical world, where the gods and heroes walk, creating and personifying the dynamics and processes that underlie everything we see in reality.

I want a tool where I can not only shape the world as a level designer, but also shift into the Dreamtime and shape the dynamics of that world as concretely I would shape the placement of coins and mushrooms.

When this happens, we will get our interactive art.

Update:
Just reposted on my blog...

[Erik Svedäng]

Wow, that's a lot of interesting thoughts Axcho!

I can see how you think this would work and it is an interesting and cool idea. Very hard to implement though... I'm trying to imagine how someone could make a very small "proof of concept" but it's hard. I'm thinking a lot about these things right now too, I will get back as soon as I have some ideas.

[God at play]

That's awesome axcho.

Here's a starting point.  You have a box in a physics environment, and you can change parameters for it visually.  If you want to change the bounciness, you can move the curve that predicts the movement of the box.  Want to make it less bouncy?  Lower the top of the first arc after it hits the ground.  Like this:


You could drag the second arc here up and down.  The second image is the result after you drag it down.

That's just one parameter.  Another could be drag, which could be adjusted by moving one of the first two circles in this image:

[Michaël Samyn]

Speaking of "natural language", I always wondered what a programming language would look like that was based on my own mother tongue, Dutch, instead of English. And would it give Dutch-speaking programmers an advantage?

[Erik Svedäng]

That's a great start, God at play.

And Michaël, please give us an example of a programming statement in Dutch

Here's one in Swedish, just for fun:

Flytta karaktären snabbt framåt (på ett snyggt sätt) när jag trycker på uppåtpilsknappen, tack.

[axcho]

Hey, glad you like the idea, thanks!

Starting with a physics simulation could be a good idea for a proof of concept.

Here's the image that first got me thinking along these lines:



This is brontosaurus showing the process behind his Tree Test drawing, so I could make a procedural version of it.



I never got around to it, but the interesting thing is that he didn't have any specific rules in mind, just some implicit guidelines expressed through these examples. The idea was for me to get inspiration from the examples in order to code up specific rules that would create similar patterns.

Procedural pixel art like this could be another interesting avenue for a proof of concept.

The step-by-step nature of it I think would be essential.
Stories, sequences. Like an IQ test for the computer.

[QXD-me]

That codebubbles thing looks pretty interesting: all the structure of code, but with some more 'natural' (I couldn't think of a better word) arrangements/flow. It looks like the sort of thing I would find very useful while coding.

(It took me a while to realise that it's easier to work out what's going on in that video with the sound on)


@axcho
Some interesting ideas. I can almost picture it, it's like one of those things you can just about see, but can't quite define. The videogame creation equivalent of playing an instrument, how dreamy 

Here's one in Swedish, just for fun:

Flytta karaktären snabbt framåt (på ett snyggt sätt) när jag trycker på uppåtpilsknappen, tack.

I'm glad programming isn't in sweedish, that looks very complicated (probably because of my complete and utter lack of familiarity with Sweedish or any similar languages).

So I'd say English speakers probably do have the 'native tongue advantage' when it comes to programming.

[Michaël Samyn]

All the visual programming editors I've seen, all the examples that have been posted here, require an uncomfortable mix of L-mode and R-mode perception in order to use. What I tend to see is a bunch of visually identical boxes connected by lines, and differentiated by text.

I do wish that the boxes were a little bit different. Though the colour coding of called, public and parameter boxes in Quest3D helps. But you know, also the little differences do. Some boxes have 1 child connector, others two or three. So even that small difference helps me to tell them apart. That plus their context. I have a personal way of making layouts with the boxes that helps me recognize an if-else-then structure very quickly, e.g. So, while it may not be easy to tell the difference between the different boxes, the different layouts of multiple boxes are much easier to differentiate. It's like making simple drawings, almost. Also, Quest3D allows you to add coloured boxes and pictures to your flow charts, which can also be very helpful.


I agree with your desire for a completely R-Mode environment. But I really don't want the "utopian" nature of this desire to hold back the very real possibility of graphical programming systems. Quest3D, while imperfect, is infinitely more fluent for me to program in than any code-based environment. Despite of being an artist, I have used it to create a multiplayer game, an reusable AI system and a rich game like The Path that has a sophisticated music system, dynamic graphics, autonomous behaviours, etc. There's no way I could have made any of these in Unity (because Unity's programming is script-based)!

I also agree with the real-time aspect. Again, as opposed to Unity, Quest3D is real time: the game is running all the time, as you program. The entire flowchart, in fact, represents a single frame in the game, a frame that is repeated over and over. Your flowchart simply contains switches to decide which part of the flowchart should be called when. All this time the game is running and in the flowchart, the boxes that are being called light up. Being able to change your code while the game is running is incredibly powerful for "R-Mode" design!
Unity allows for a little bit of that, thanks to being able to expose parameters in the editor's interface. But it's not the same as real real-time editing.

[axcho]

I do wish that the boxes were a little bit different. Though the colour coding of called, public and parameter boxes in Quest3D helps. But you know, also the little differences do. Some boxes have 1 child connector, others two or three. So even that small difference helps me to tell them apart. That plus their context.

I really appreciate your feedback on this, as you are one of the few here with actual development experience in a visual programming language!

Even the rudimentary R-mode readability that Quest3D offers is still very helpful, it sounds like you're saying.

On that note, I had an idea to bump the visual differences up a bit more - to use randomly generated "invader" sprites for each type of node, so they are visually distinctive. They're random bitmaps, mirrored across the vertical axis, so your brain recognizes them more easily, like faces. The variations are effectively infinite and they're easy for computers to generate. Whenever you create a new type it would pop up a new, unique random image to use. Still symbolic but more visually distinctive than boxes.

Like this:


Ideally the visuals would reflect the actual logic or behavior of what is represented, but as you said, I don't want the utopian vision to hold back practical but imperfect improvements. Like making each node visually distinct. Do you have any thoughts about how this random invader icon idea could be applied? Maybe the inputs and outputs could be incorporated into the image - the more meaningful the visuals, the better.

What do you think?

Another interesting question is what are you actually manipulating with these visual flowcharts?

In Quest3D you are manipulating the flow of logic. Commands and if-statements and all that.

However, there are other options. Like manipulating the flow of data - dataflow programming.
Check out this tutorial on Prograph, a visual programming language for manipulating the flow of data rather than logic. There are pictures.

I wonder if that might have any application for (not)games? Maybe in a modified form, to support data-oriented design, perhaps? I bet a visual programming language could be better suited to that paradigm than the traditional languages like C++ and such. Apparently that's where things are headed, anyway, as processors get faster and memory access times lag behind. It would be pretty ironic if people had to turn to visual programming in order to write optimally efficient code for their games!

In addition to manipulating the flow of logic, or data, there could be other opportunities for visual programming to shine. For example, representing the connections between objects. Usually the most verbose and confusing bits of code I see have to do with one object accessing data (or functions, whatever) from another object through some convoluted chain of references, with all sorts of annoying syntax and symbols needed to represent what should be a simple idea. Linear, linguistic communication was never so good at describing relationships. That's what R-mode is for. I'd really like to see this problem explored with a visual programming approach for organizing objects and groups and types of objects and drawing the connections between them.

In particular, one specific place that could benefit from a visual way to see and create connections between objects is in component-based game architecture. Like data-oriented design, it's another step up (or at least away from) traditional object-oriented programming, in this case to improve modularity. And it's great, it just can get tough to deal with all the interconnections - not just between objects, but between the components of one object, the components of multiple objects trying to communicate, and so on.

Also, a video tutorial on component-based architecture if you're new to the concept.

Thoughts?

[Kjell]

What do you think?

SunVox uses exactly that to label patterns. Personally I'm not a big fan though.

Being able to change your code while the game is running is incredibly powerful.

Agreed, can't live without it anymore.

[axcho]

SunVox uses exactly that to label patterns. Personally I'm not a big fan though.

I am astounded. I thought I was so clever to think of that. And it's already been done. This is a great opportunity then - it will be much easier to see what's good or bad about the idea!

So, you don't like the random icons much? This is very good to know. What do you dislike about them - what would have to be different for them to be actually helpful to you?

I'm really curious what problems you run into with using these random icons - what problems do they fail to address, what problems do they introduce? What could be improved? What do you like instead?

Looking forward to hearing your response.

[Kjell]

Random is always a bad idea when describing something. Perhaps colors would make them a little easier to distinguish. And i like words instead

Or even better, no words & icons at all.