Discussion Shadow Language Example & Language Design Discussion (non-technical, newbies welcome)

SSJCoder

Member
Context: for the past few years I've been working on language design, aiming to specialize in the design of programming languages, over that time I've created over 5 languages, but some are simple, others are complex, some are pretty much the same/similar/etc (and there's a few which didn't get finished.. I think..)

I wanted to talk to people about language design, features people are interested in, etc, etc

Here is a GM8 example of the latest language that I've been working on, which is right now still in the alpha/testing phase:


Information page: http://ssjstash.net/shadow-gm

So, feel free to let me know your thoughts & ideas, your own programming languages, etc

Keep in mind this is not a technical discussion when posting in this topic, thank you
 
Last edited:
D

Deleted member 45063

Guest
Ok, finally managed to see the video (and get permissions to post here).

1st - The nostalgia is real (GM 8).
2nd - Kudos on making your own language, that is always a nice accomplishment!

Now for the actual comments on the video.

You already know (from my status update comments) that I would personally prefer a slightly different syntax, however this video definitely made that more than just a personal preference. The issue I have is that with the current syntax there seems to be a mix between different concepts of the language (or from a language design standpoint). For example, there seems to be a mix between what are global variables to the Shadow language itself and its namespaces (e.g. $a = 50 and $a::a = 50, $a is both used for accessing the global variable and the namespace). Also, a global variable for the Shadow language and a global variable for GML don't really seem like two different constructs from a language perspective (although implementation wise I know they differ), but in the syntax they are again treated differently (e.g. $score = 50 vs &score = 50). Similarly, Shadow functions need to use the $function notation while built-in functions do not.

So the summary of that comment is that either the syntax is confusing and it seems that a lot of different concepts are mixed or maybe you intended it that way in which case my comment is that a lot of different concepts seem to be mixed up.

Another comment (but this would be fixed by the autocomplete support and isn't strictly language-related) is that if you need to switch between two code editors often to code then it is highly inefficient and off-putting. If I recall correctly in GM 8 there was still a file fnames somewhere that listed all of the functions and their arguments. You could use that for generating the autocompletion support (or as I started doing on one of my languages at that time - generate core modules that delegate to GameMaker function calls but that are declared as the custom language. That way your editor only needs to understand one of the languages, etc).

Aside from that it's very nice that you actually compile to GameMaker. Have you considered supporting an interpreter on the GM side to allow Shadow code execution at runtime? For GM 8 it isn't very relevant due to string_execute but if you port it to more recent versions of GameMaker it can be a very neat feature.

Some things you might find interesting:
  • The JAI programming language, created by Jonathan Blow (creator of Braid and The Witness). The compiler is not officially released yet but on his YouTube Channel you can find streams on the implementation of some parts of the compiler as well as the official demos of the language, where he discusses language design and his line of thought in detail.
  • The Odin programming language, which was created by gingerBill and draws a big influence from the JAI programming language. However this one does have a released compiler that you can look into, other than also some demos on YouTube and some streams on Twitch. As Jon Blow, Bill also seems to have some very strong opinions about language design. Unlike JAI, Odin is meant to be less game oriented and more systems programming oriented.
Also, have you considered just implementing a custom backend to an existing compiler? For example, imagine you like how Kotlin, Rust, Elm or any other language looks like and just want it to compile to GML. You could take their compiler and just replace the backend to produce GML instead of JVM bytecode, native code or JS. That way you have support from pretty much any editor and you leverage all of the optimizations and such that those compilers already bring. Of course that since it seems your goal is the language design itself this might seem counter-productive, but it could give you quite a nice look into the design decisions behind those languages, which can be valuable in itself.

Sorry for the wall of text and keep up the good work on that language! :)
 

SSJCoder

Member
Sorry for the wall of text and keep up the good work on that language! :)
Wall of texts are no problem ATM xD

Yes, the syntax is probably confusing you and I didn't explain it properly in the video lol (since I'm still going to make the actual documentation).

Here are the original plans for function-declaration syntax:
Code:
(function-address) (arguments) {block}/(single-instruction)
// or alternatively:
(function-address) {block}/(single-instruction)
// namespaces
$SystemGlobal
&Global
FileLocal // this one is the one that I'm experimenting with - whether to make it
           // file-local, or whether to make it output a gamemaker function that can be
           // accessed from outside - this is easy to change based on user preference tho
$GM::(variable-name) // Game Maker namespace
Examples of valid function declarations:
Code:
$add (a, b) {@ret a+b} // system namespace
&add (a, b) {@ret a+b} // global namespace
add (a, b) {@ret a+b} // file-local/other/experimental namespace
$emptyfunc {}
&emptyfunc {}
emptyfunc {}
$add (a, b) @ret a+b
&add (a, b) @ret a+b
add (a, b) @ret a+b
Then, you can also create namespaces inside namespaces:
Code:
$(namespace)::(address)
You can use that on any namespace (including local), but obviously anything in the "$GM" namespace (non case-sensitive/case-insensitive) will not be put any "sandboxed" namespace, but in the actual Game Maker namespace in the output.

Another comment (but this would be fixed by the autocomplete support and isn't strictly language-related) is that if you need to switch between two code editors often to code then it is highly inefficient and off-putting. If I recall correctly in GM 8 there was still a file fnames somewhere that listed all of the functions and their arguments. You could use that for generating the autocompletion support (or as I started doing on one of my languages at that time - generate core modules that delegate to GameMaker function calls but that are declared as the custom language. That way your editor only needs to understand one of the languages, etc).
May look into it

Aside from that it's very nice that you actually compile to GameMaker. Have you considered supporting an interpreter on the GM side to allow Shadow code execution at runtime? For GM 8 it isn't very relevant due to string_execute but if you port it to more recent versions of GameMaker it can be a very neat feature.
Have not considered it yet

Some things you might find interesting:
  • The JAI programming language, created by Jonathan Blow (creator of Braid and The Witness). The compiler is not officially released yet but on his YouTube Channel you can find streams on the implementation of some parts of the compiler as well as the official demos of the language, where he discusses language design and his line of thought in detail.
  • The Odin programming language, which was created by gingerBill and draws a big influence from the JAI programming language. However this one does have a released compiler that you can look into, other than also some demos on YouTube and some streams on Twitch. As Jon Blow, Bill also seems to have some very strong opinions about language design. Unlike JAI, Odin is meant to be less game oriented and more systems programming oriented.
May look into them

Also, have you considered just implementing a custom backend to an existing compiler? For example, imagine you like how Kotlin, Rust, Elm or any other language looks like and just want it to compile to GML. You could take their compiler and just replace the backend to produce GML instead of JVM bytecode, native code or JS. That way you have support from pretty much any editor and you leverage all of the optimizations and such that those compilers already bring. Of course that since it seems your goal is the language design itself this might seem counter-productive, but it could give you quite a nice look into the design decisions behind those languages, which can be valuable in itself.
Not sure exactly what that means xD
 
Last edited:
D

Deleted member 45063

Guest
Examples of valid function declarations:
From what I understand in that example is that the "system" namespace is the namespace global to the Shadow language itself?

I'm still going to make the actual documentation
I'm looking forward to it to try to make some more constructive critiques on the language itself :p

Not sure exactly what that means xD
A compiler has typically at least two, usually three layers: the frontend (lexer + parser) to handle the actual syntax and whatnot, the (optional) AST transformation / optimization phase and the backend (the part that generates the output of the compiler - native code, other programming languages, etc). That final remark was about whether you considered just taking a compiler for an existing programming language and replacing the backend to produce GML instead.
 

SSJCoder

Member
From what I understand in that example is that the "system" namespace is the namespace global to the Shadow language itself?
The global/system namespaces are basically two global namespaces which cannot be accessed from outside the shadow language (or are not meant to be naturally - but you can if you *really* have to)

The main difference is that the system namespace is used for special things as well like the "$GM::" namespace, besides that, the system namespace is pretty much a different global namespace - meant to be used for system-specific stuff (like libraries, system programming which is not part of the actual project, etc)

I'm looking forward to it to try to make some more constructive critiques on the language itself :p
It may take me a few days I have to get my brain "back in order" after basically burning out the last week xD
Then I'll write some simple docs :)

A compiler has at least two, usually three layers: the frontend (lexer + parser) to handle the actual syntax and whatnot, the (optional) AST transformation / optimization phase and the backend (the part that generates the output of the compiler - native code, other programming languages, etc). That final remark was about whether you consider just taking a compiler for an existing programming language and just replacing the backend to produce GML instead.
So for me, I have two layers: interpreter, and the "porting" program, obviously I can put stuff in between, but I have 100 % custom code basically to deal with all of that - looking into one of these existing compilers is (I am guessing) a pretty big investment, not sure if I could take time off other things to put into that (at the moment)

Maybe when I have some spare time, and I find a good (simple) starting guide. (if you know a good one maybe post here? xD)

Also, just so you know, if doing so requires installing many "dependencies" or "requirements" I'm not sure I could, I have had a hard time installing stuff and making it work - my current setup is just Node JS installed on my computer (takes a few minutes) and I can run the compiler.. I'd prefer to keep it that way.
 
D

Deleted member 45063

Guest
So for me, I have two layers: interpreter, and the "porting" program, obviously I can put stuff in between, but I have 100 % custom code basically to deal with all of that - looking into one of these existing compilers is (I am guessing) a pretty big investment, not sure if I could take time off other things to put into that (at the moment)

Maybe when I have some spare time, and I find a good (simple) starting guide. (if you know a good one maybe post here? xD)

Also, just so you know, if doing so requires installing many "dependencies" or "requirements" I'm not sure I could, I have had a hard time installing stuff and making it work - my current setup is just Node JS installed on my computer (takes a few minutes) and I can run the compiler.. I'd prefer to keep it that way.
That remark was mostly to note that you could reuse parts of an existing compiler, but yes it is likely that it will rely on a different technology stack than Node JS. Also it would involve getting into the actual code of the compiler. As for a guide I can't really give any as each compiler is (most likely) very different.

The global/system namespaces are basically two global namespaces which cannot be accessed from outside the shadow language (or are not meant to be naturally - but you can if you *really* have to)

The main difference is that the system namespace is used for special things as well like the "$GM::" namespace, besides that, the system namespace is pretty much a different global namespace - meant to be used for system-specific stuff (like libraries, system programming which is not part of the actual project, etc)
So when you do $FPS = 50 is that variable declared in the system namespace? If so, it doesn't seem to align 100% with your statements of what the system namespace seems to aim for. Also, I am curious: what is the reasoning behind having two different global namespaces, and how do they relate / interact with each other (e.g. is one a parent scope of the other)?
 

SSJCoder

Member
$FPS = 50
Well I am used to keeping that as part of the system, but you could argue (if you want) that it is project-specific, and you can also use:
&FPS = ..
instead

is that variable declared in the system namespace?
Yes

Also, I am curious: what is the reasoning behind having two different global namespaces, and how do they relate / interact with each other (e.g. is one a parent scope of the other)?
I like to keep libraries separate from the project, this is one easy way of achieving that.

Example, imagine you create a library called "GameMakerGridCollision", you could use:
Code:
$GMGC::(library-related-global-stuff)
to do all your functions/global variables related to that library.

They are not related to each other, and do not interact with each other.
 

SSJCoder

Member
Just want to also point out, the "return" statement in the video was a mistake, meant to be "@ret" - but it still compiled correctly and I didn't notice it 'till afterwards xD
 
D

Deleted member 45063

Guest
I like to keep libraries separate from the project, this is one easy way of achieving that.

Example, imagine you create a library called "GameMakerGridCollision", you could use:
$GMGC:: (library-related-global-stuff) to do all your functions/global variables related to that library.

They are not related to each other, and do not interact with each other.
But wouldn't that then be a matter of just namespacing, which could also theoretically happen on the & global scope? My doubt is whether there is an actual need for that distinction and also what happens in the case of &foo = 1 and $foo = 1? From the Shadow code it's clear which is which due to the prefixes but do they have different name mangling semantics? How would that code look like from the GM side of things?

Just want to also point out, the "return" statement in the video was a mistake, meant to be "@ret" - but it still compiled correctly and I didn't notice it 'till afterwards xD
That's a different topic :p (I think there are some well founded constructs common to most programming languages that don't need to be changed like the if, unless there is a concrete line of thought that I'm missing - for example in the @ret case it is more concise but also a lot harder to type on non-US keyboard layouts - for instance in my current German layout it requires pressing Right Alt + Q, in my Portuguese layout is Right Alt + 2 if I'm not mistaken).
 

SSJCoder

Member
But wouldn't that then be a matter of just namespacing, which could also theoretically happen on the & global scope? My doubt is whether there is an actual need for that distinction and also what happens in the case of &foo = 1 and $foo = 1? From the Shadow code it's clear which is which due to the prefixes but do they have different name mangling semantics? How would that code look like from the GM side of things?
Well, on the implementation I can explain it in a pretty basic way (implementation could change, but currently it works this way).
Every global variable has a unique ID attached to the end of it, so if you declare $foo it will get a unique ID, such as 3, the next variable gets 4, etc, and they are appended like this:
<namespace>__<variablename>_<ID>_
again, this could change.
They can never conflict with each other in GM.

As for the namespacing, yes, you can just not use $ namespace altogether and just use &

That's a different topic :p (I think there are some well founded constructs common to most programming languages that don't need to be changed like the if, unless there is a concrete line of thought that I'm missing - for example in the @ret case it is more concise but also a lot harder to type on non-US keyboard layouts - for instance in my current German layout it requires pressing Right Alt + Q, in my Portuguese layout is Right Alt + 2 if I'm not mistaken).
A lot of people in the past have had issues with my idea of completey removing keywords (such as if, else), and that is mostly my own personal preference of doing things - to clear up the "vanilla"/"default" namespace, meaning you could use:
Code:
~if = 400;
@test ( if == 30 ) {
    // do stuff
}
// or you can use 'test' from the standard (non-pre-fixed) namespace, without conflicts.
Of course, I do believe this can also be customizable based on user preference (which special character to use, whether to use keywords or not, etc)
(reason I say "believe" is because I haven't tried implementing that - yet xD)
 
Last edited:
D

Deleted member 45063

Guest
Well, on the implementation I can explain it in a pretty basic way (implementation could change, but currently it works this way).
Every global variable has a unique ID attached to the end of it, so if you declare "$foo" it will get a unique ID, such as 3, the next variable gets 4, etc, and they are appended like this:
<namespace>__<variablename>_<ID>_
again, this could change.
They can never conflict with each other in GM.
So is the system scope the only one that supports namespaces? It would seem so based on
Code:
$(address)::(namespace)
In which case wouldn't there be value in allowing the user to create namespaces outside of the "system" concept? Also how would that interface with libraries (another question regarding libraries follows), is the author expected to always declare the namespace itself or can the caller map the library to a specific namespace in order to also avoid collisions in case two authors create libraries with the same namespace?

The other question about libraries is: What plans are there for distribution / incorporation of external libraries? Is it just based on "Download & Copy into source tree"? Which then leads to: how are source files discovered? Just any Shadow file under the src directory or will it follow some sort of module inclusion directive?

<namespace>__<variablename>_<ID>_
Getting back to library distribution. Would they always be distributed in Shadow source code in which case the above name mangling should always kinda work or would / could they be distributed in GML source code in which case there might be conflicts with the above rule since the code was generated in two completely different compiler executions, each of which started numbering variable prefixes the same way (e.g. you use $FPS = 60 in your code and somewhere in the library that is also used. Would that effectively be the same reference?).

I do believe this can also be customizable based on user preference
I would highly suggest against this since this will make the compiler / language needlessly more complex and when you start having "support tickets" it might look like an entirely different language that you'll need to learn because the user decided to change every since option in the language.
 

SSJCoder

Member
So is the system scope the only one that supports namespaces? It would seem so based on
Every default namespace supports child namespaces, examples:
Code:
$a::a = 30;
&a::a = 25;
~a::a = 15;
// these do not conflict with each other
Also how would that interface with libraries (another question regarding libraries follows), is the author expected to always declare the namespace itself or can the caller map the library to a specific namespace in order to also avoid collisions in case two authors create libraries with the same namespace?
Well.. not that far into it yet, but I'm sure I can figure something out xD
(the first thought is obviously.. taking the conflicting library's code and performing a "replace" operation to <namespace-name>2 or something)

The other question about libraries is: What plans are there for distribution / incorporation of external libraries? Is it just based on "Download & Copy into source tree"?
For now, yes, in the future I may come up with other ideas.
(so for now just copy-paste libraries into your src directories.. but then, I think maybe it's better to do a small config file as described below to allow inclusion of code from other directories on your file-system, such as lib folder)

Which then leads to: how are source files discovered? Just any Shadow file under the src directory or will it follow some sort of module inclusion directive?
Umm.. currently just the src directory but I could change that up.. I have previously had a previous language read a small little config file to tell it which directories to scan.. may implement that in the future

Getting back to library distribution. Would they always be distributed in Shadow source code in which case the above name mangling should always kinda work or would / could they be distributed in GML source code in which case there might be conflicts with the above rule since the code was generated in two completely different compiler executions, each of which started numbering variable prefixes the same way (e.g. you use $FPS = 60 in your code and somewhere in the library that is also used. Would that effectively be the same reference?).
well yes, so if outside of the shadow language conflicts are not monitored.. if inside the shadow language, a library should not use $FPS as its own variable, it should use:
$(library-namespace)::FPS, as a general rule of thumb.

I would highly suggest against this since this will make the compiler / language needlessly more complex and when you start having "support tickets" it might look like an entirely different language that you'll need to learn because the user decided to change every since option in the language.
Well, I plan to create a special program that will allow you to view someone else's code according to your own language preferences, or the "vanilla" preferences, or whatever other preferences, so currently this is only an issue in today's technologies which just have text files - in my world, I want to have programs distributed in different forms to allow such viewing naturally
 

SSJCoder

Member
Code:
$(address)::(namespace)
Also that was a mistake, fixed, meant to be:
Code:
$(namespace)::(address)
 
D

Deleted member 45063

Guest
Too late to reply to the other things, I'll do it tomorrow, just wanted to leave the following note.

Well, I plan to create a special program that will allow you to view someone else's code according to your own language preferences, or the "vanilla" preferences, or whatever other preferences, so currently this is only an issue in today's technologies which just have text files - in my world, I want to have programs distributed in different forms to allow such viewing naturally
"Today's technologies" already do that, for example JetBrain's MPS tool allows you to define arbitrary Domain Specific Languages that are actually defined, stored, handled and written directly in AST form, so you can have as many visualizations of the same "code" as you want. I just wanted to note that for a custom language (which to be blunt the probability of it being widespread in the GMC is not that high due to the fact it is not a language endorsed by YYG) that might be more complexity than what is warranted, complexity that might be better used refining other features of such language (to increase the likelihood of it having a widespread usage).
 

SSJCoder

Member
Too late to reply to the other things, I'll do it tomorrow, just wanted to leave the following note.


"Today's technologies" already do that, for example JetBrain's MPS tool allows you to define arbitrary Domain Specific Languages that are actually defined, stored, handled and written directly in AST form, so you can have as many visualizations of the same "code" as you want. I just wanted to note that for a custom language (which to be blunt the probability of it being widespread in the GMC is not that high due to the fact it is not a language endorsed by YYG) that might be more complexity than what is warranted, complexity that might be better used refining other features of such language (to increase the likelihood of it having a widespread usage).
sure, I'll be here tomorrow xD

Well yes, widespread is maybe going to be an issue if that is my goal - but fortunately, I'm just looking at a few people who could (possibly) be interested in it, but even then, at the end of the day, this language's original purpose is for my personal use.

And then there's community projects that I want to host, and I want to host them using my own technologies, so that's that.

EDIT:
alright, so there a few technologies out there that have that, not bad, but yea, things like git-hub I think is where this would become an issue. (unless there's somehow a way to make a plugin or something? idk)
 
Last edited:

Mnementh

Member
GMC Elder
I'm curious, what's your motivation for creating this language? Hypothetically, why would I choose to use Shadow over GML?
 

SSJCoder

Member
I'm curious, what's your motivation for creating this language? Hypothetically, why would I choose to use Shadow over GML?
My motivation: I decided to specialize in Language Design, and so, this is one of the projects that I'm doing as a part of my "training", it's basically something I'm passionate about, but really just one of the many things I plan to make over my lifetime.

Why choose it over GML: depends which GM version you are using/your project/goal, and also your preferences in what kind of language you want.

If you happen to still be using GM8, then it could be beneficial if you type in a lot of code because of many powerful features that there are in the language, such as:
* creating many functions in a single file easily
* 'argument0'/'argument1'/etc do not exist, actual argument names are used (so code is more clear)
* copy paste huge sets of functions instead of having to dupe them manually in GM8
* custom namespaces allow for better organizing/management
* no need to worry about built in variables not being able to be 're-declared'
* infinite sized arrays (uses ds_lists instead of arrays)
* create arrays on the go like this: ~arr = [elm1, elm2, elm3]
* call results of functions easily, such as: a()(); // calls a, then calls the result, and also you can call any variable it will automatically call the result instead of throwing an error that it's an uknown function or script
* easy mass function renaming

Then there are planned features, still, such as:
* easy array operations (such as array->push( item1, item2 ); array->free(); // frees memory associated with array)
* infinite dimension arrays (this one I haven't researched properly yet, so it's a 'could be implemented' kind of feature)

But then, this is more of a specialized language, not a generic one, so you should not learn this language if:
* you are not prepared to learn something drastically different from what you're used to
* you are making small scale projects
* you don't like a customizable language
* you are a novice programmer
* you do not like the style of the language/all the customization available
* language features do not benefit your programming methods
* you want high performance (this language is not meant to be faster than natural GML)

I would say this is one of those things that you should try for 5 mins, then decide if it's for you or not - but the truth is, like I said, this is not a generic language, it's not meant for your average joe, but it's a very specialized language specifically for dedicated programmers.
 
Last edited:
Top