Graphics Convert Pixel Art to 3d Model in Real Time

angelwire

Member
GM Version: 1.4.1763
Target Platform: ALL
Download: GMS:1.4 .gmz file
Links: N/A

screeny.png

Summary:
This tutorial is an example of how to take pixel art sprites and convert them to 3d models in real time. This makes the sprites look solid and "voxelized" in the 3d space as opposed to flat planes. Since it's done dynamically in real time there's no need to use any modelling software or do any sort of 3d design.

Disclaimer:
I am by no means a perfect programmer, so let me know if you see any avenues for optimization or know how to make my code better.

Tutorial:
If you're reading this, hopefully you already have your 3d environment set up, but if not here's a quick way to set up a 3d camera object in an empty project (using 1.4)
camera_object CREATE EVENT:
Code:
d3d_start();
d3d_set_hidden(false);
d3d_set_lighting(false);
d3d_set_culling(false);
draw_set_color(c_white);
z = 0;
target_x = player_object.x;
target_y = player_object.y;
target_z = 0;
camera_object DRAW EVENT:
Code:
d3d_set_projection(x,y,z,target_x,target_y,target_z,0,0,1);

I'm going to call the object that creates and draws the model "player_object"

Just a note here, Make sure your sprite is checked to be used for 3d

In the player_object CREATE EVENT you're need 2 lines of code:
Code:
my_model = -1; //Initiate the model to -1
my_thickness = 2; //Set the thickness of the model
For the player_object STEP EVENT you need just a few more lines:
Code:
//Only update the model if the image has changed
if sprite_should_update() //Custom script
{
    d3d_model_destroy(my_model); //Destroy the past model
    texture_set_interpolation(false); //Interpolation should be turned off   
    //Create new model from the sprite, image frame, and the thickness that was set in the CREATE EVENT
    my_model = create_model_from_sprite(sprite_index, image_index, my_thickness); //Custom script
}
Here's the first custom script. It checks to see if the model should be updated. It only works with an image_speed that's smaller than 1. You may need to write this code yourself depending on how you're animating your sprite. Updating the model every step is pretty costly so I strongly suggest only updating the model when the image changes.

SCRIPT: sprite_should_update
Code:
///sprite_should_update()
return (floor(image_index) == round(image_index));
Now here's the big script, the one that does all the hard work. I’ll explain what I’m doing with the d3d_model_vertex_normal_texture at the end, but the rest is documented.

SCRIPT: create_model_from_sprite
Code:
///create_model_from_sprite(sprite,index,depth)
var sprite = argument0; //The sprite that should be modeled
var index = argument1; //The frame of the sprite that should be modeled
var dd = argument2; //This is for the depth/thickness
var temp = d3d_model_create(); //This is the model that will be returned
var ii = 0; //A loop variable
var bool_value = false; //Whether or not a pixel is visible
var last_bool_value = false; //Whether or not the pixel to the left is visible
var side_offset = .4; //This sets the UV coordinates to the edges of the pixel
var sprite_buffer; //The buffer that holds the sprite contents
var surface_size = sprite_get_width(sprite); //How big the surface is
var xx = 0; //The X position
var yy = surface_size-1; //The Y position (starts from bottom)
var op = 1/(surface_size); //one pixel on the UV map
var op_middle = op*.5; //Half of one pixel, used to place UV coordinates in the middle of a pixel

//Create the surface that will be converted to a buffer and set it as the target
var my_surface = surface_create(surface_size,surface_size);
surface_set_target(my_surface);

//Clear the surface and draw the sprite
draw_clear_alpha(c_black,0.0);
draw_sprite(sprite,index,sprite_get_xoffset(sprite),sprite_get_yoffset(sprite));

surface_reset_target(); //Reset the target

//Create the buffer and convert the surface to a buffer
sprite_buffer = buffer_create((surface_size*surface_size)*4,buffer_fast,1);
buffer_get_surface(sprite_buffer,my_surface,1,0,0);
surface_free(my_surface); //Free the surface from memory

d3d_model_primitive_begin(temp,pr_trianglelist); //Begin the model

while(ii < surface_size*surface_size) //For every pixel in the surface
{
    buffer_seek(sprite_buffer,buffer_seek_relative,3); //The RGB values are ignored so skip them

    //If the value of the pixel's alpha is larger than 128 count the pixel as visible
    bool_value = buffer_read(sprite_buffer,buffer_u8) > 128;
   
    if bool_value = true //If the pixel is visible
    {

    /*
    Currently I haven't figured out a way to selectively draw the needed faces
    So right now, all 6 sides are being drawn,
    This means even if a pixel is surrounded by other pixels,
    It will draw the top, bottom, left, and right sides,
    Those sides may not be visible so they are redundant
    (there is one exception and you'll see it below)
    */

    //Front
    d3d_model_vertex_normal_texture(temp,xx,yy,0,-1,-1,1,(op*xx)+(op_middle),(op*yy)+(op_middle));
    d3d_model_vertex_normal_texture(temp,xx+1,yy,0,1,-1,1,(op*xx)+(op_middle),(op*yy)+(op_middle));
    d3d_model_vertex_normal_texture(temp,xx,yy+1,0,-1,1,1,(op*xx)+(op_middle),(op*yy)+(op_middle));
   
    d3d_model_vertex_normal_texture(temp,xx+1,yy,0,1,-1,1,(op*xx)+(op_middle),(op*yy)+(op_middle));
    d3d_model_vertex_normal_texture(temp,xx+1,yy+1,0,1,1,1,(op*xx)+(op_middle),(op*yy)+(op_middle));
    d3d_model_vertex_normal_texture(temp,xx,yy+1,0,-1,1,1,(op*xx)+(op_middle),(op*yy)+(op_middle));
   
    //Back   
    d3d_model_vertex_normal_texture(temp,xx,yy,dd,-1,-1,-1,(op*xx)+(op_middle),(op*yy)+(op_middle));
    d3d_model_vertex_normal_texture(temp,xx+1,yy,dd,1,-1,-1,(op*xx)+(op_middle),(op*yy)+(op_middle));
    d3d_model_vertex_normal_texture(temp,xx,yy+1,dd,-1,1,-1,(op*xx)+(op_middle),(op*yy)+(op_middle));
   
    d3d_model_vertex_normal_texture(temp,xx+1,yy,dd,1,-1,-1,(op*xx)+(op_middle),(op*yy)+(op_middle));
    d3d_model_vertex_normal_texture(temp,xx+1,yy+1,dd,1,1,-1,(op*xx)+(op_middle),(op*yy)+(op_middle));
    d3d_model_vertex_normal_texture(temp,xx,yy+1,dd,-1,1,-1,(op*xx)+(op_middle),(op*yy)+(op_middle));
   
    /*
    Here's that one exception to what I mentioned earlier. If we know that the pixel to the left
    is solid, we don't need to draw the left faces. It only saves a handful of faces,
    but every little bit counts!
    */
   
    //Left
    if !last_bool_value
    {
    d3d_model_vertex_normal_texture(temp,xx,yy,0,-1,-1,1,(op*xx)-(op*side_offset)+(op_middle),(op*yy)+(op_middle));
    d3d_model_vertex_normal_texture(temp,xx,yy,dd,-1,-1,-1,(op*xx)-(op*side_offset)+(op_middle),(op*yy)+(op_middle));
    d3d_model_vertex_normal_texture(temp,xx,yy+1,dd,-1,1,-1,(op*xx)-(op*side_offset)+(op_middle),(op*yy)+(op_middle));

    d3d_model_vertex_normal_texture(temp,xx,yy,0,-1,-1,1,(op*xx)-(op*side_offset)+(op_middle),(op*yy)+(op_middle));
    d3d_model_vertex_normal_texture(temp,xx,yy+1,dd,-1,1,-1,(op*xx)-(op*side_offset)+(op_middle),(op*yy)+(op_middle));
    d3d_model_vertex_normal_texture(temp,xx,yy+1,0,-1,1,1,(op*xx)-(op*side_offset)+(op_middle),(op*yy)+(op_middle));
    }

    //Right
    d3d_model_vertex_normal_texture(temp,xx+1,yy,0,1,-1,1,(op*xx)+(op*side_offset)+(op_middle),(op*yy)+(op_middle));
    d3d_model_vertex_normal_texture(temp,xx+1,yy,dd,1,-1,-1,(op*xx)+(op*side_offset)+(op_middle),(op*yy)+(op_middle));
    d3d_model_vertex_normal_texture(temp,xx+1,yy+1,dd,1,1,-1,(op*xx)+(op*side_offset)+(op_middle),(op*yy)+(op_middle));

    d3d_model_vertex_normal_texture(temp,xx+1,yy,0,1,-1,1,(op*xx)+(op*side_offset)+(op_middle),(op*yy)+(op_middle));
    d3d_model_vertex_normal_texture(temp,xx+1,yy+1,dd,1,1,-1,(op*xx)+(op*side_offset)+(op_middle),(op*yy)+(op_middle));
    d3d_model_vertex_normal_texture(temp,xx+1,yy+1,0,1,1,1,(op*xx)+(op*side_offset)+(op_middle),(op*yy)+(op_middle));
   

    //Top
    d3d_model_vertex_normal_texture(temp,xx,yy,0,-1,-1,1,(op*xx)+(op_middle),(op*yy)-(op*side_offset)+(op_middle));
    d3d_model_vertex_normal_texture(temp,xx,yy,dd,-1,-1,-1,(op*xx)+(op_middle),(op*yy)-(op*side_offset)+(op_middle));
    d3d_model_vertex_normal_texture(temp,xx+1,yy,dd,1,-1,-1,(op*xx)+(op_middle),(op*yy)-(op*side_offset)+(op_middle));

    d3d_model_vertex_normal_texture(temp,xx,yy,0,-1,-1,1,(op*xx)+(op_middle),(op*yy)-(op*side_offset)+(op_middle));
    d3d_model_vertex_normal_texture(temp,xx+1,yy,0,1,-1,1,(op*xx)+(op_middle),(op*yy)-(op*side_offset)+(op_middle));
    d3d_model_vertex_normal_texture(temp,xx+1,yy,dd,1,-1,-1,(op*xx)+(op_middle),(op*yy)-(op*side_offset)+(op_middle));

    //Bottom
    d3d_model_vertex_normal_texture(temp,xx,yy+1,0,-1,1,1,(op*xx)+(op_middle),(op*yy)+(op*side_offset)+(op_middle));
    d3d_model_vertex_normal_texture(temp,xx,yy+1,dd,-1,1,-1,(op*xx)+(op_middle),(op*yy)+(op*side_offset)+(op_middle));
    d3d_model_vertex_normal_texture(temp,xx+1,yy+1,dd,1,1,-1,(op*xx)+(op_middle),(op*yy)+(op*side_offset)+(op_middle));

    d3d_model_vertex_normal_texture(temp,xx,yy+1,0,-1,1,1,(op*xx)+(op_middle),(op*yy)+(op*side_offset)+(op_middle));
    d3d_model_vertex_normal_texture(temp,xx+1,yy+1,0,1,1,1,(op*xx)+(op_middle),(op*yy)+(op*side_offset)+(op_middle));
    d3d_model_vertex_normal_texture(temp,xx+1,yy+1,dd,1,1,-1,(op*xx)+(op_middle),(op*yy)+(op*side_offset)+(op_middle));
    }
    /*
    We're saving the bool_value in another variable so we can check
    whether this pixel was solid next time the loop goes around
    */
    last_bool_value = bool_value;
   
    xx+=1; //Increment the xx value   
    if xx >= surface_size //If xx is too far over
    {
    yy-=1; //Move the yy up one
    xx = 0; //Move the xx back to the start
    }
    ii+=1; //Increment ii so we know how many times we've gone through this
}
d3d_model_primitive_end(temp); //The model has been created so we finalize it
buffer_delete(sprite_buffer); //The buffer is no longer needed so we delete it
return temp; //Return the model
Here's a rundown of how the d3d_model_vertex_normal_texture is working:
Temp is the model (pretty simple)
The xx and yy values are the current pixel position being read from the buffer (again, the yy value starts at the bottom and works its way to the top)
The normals are currently set so that each individual cube is smoothly shaded
The UV coordinates are placed in a way that allows for a neat trick. (see Extra 1 at the bottom) The top faces are each placed at the top of their respective pixel, the bottom faces at the bottom of that pixel, and the left and right on the left side and right side of the pixel. The front and back faces are in the center This doesn’t change the color of the pixel unless the texture you’re applying to the model is a different size than the sprite you’re converting to a model (which is where the cool thing comes in)
uvcoords.png


All that’s left now is to draw the model with the proper scale, position, and rotation. Here’s a quick start guide to doing that:
Code:
d3d_transform_set_identity()
d3d_transform_add_scaling(1,1,1);
d3d_transform_add_translation(x,y,0);
d3d_model_draw(my_model,-sprite_width*.5,-sprite_width*.5,-my_thickness*.5,sprite_get_texture(sprite_index,image_index));
d3d_transform_set_identity();
Now that's all it takes to get it up and running. It's really just the start, however.
Here are a couple options and extras things you can do:
Thanks to @MaddeMichael for the awesome vertex buffer guide
Code:
///create_vertex_buffer_from_sprite(sprite,index,depth)

var sprite = argument0; //The sprite that should be modeled
var index = argument1; //The frame of the sprite that should be modeled
var dd = argument2; //This is for the depth/thickness
var temp = d3d_model_create(); //This is the model that will be returned
var ii = 0; //A loop variable
var bool_value = false; //Whether or not a pixel is visible
var last_bool_value = false; //Whether or not the pixel to the left is visible
var side_offset = .4; //This sets the UV coordinates to the edges of the pixel
var sprite_buffer; //The buffer that holds the sprite contents
var surface_size = sprite_get_width(sprite); //How big the surface is
var xx = 0; //The X position
var yy = surface_size-1; //The Y position (starts from bottom)
var op = 1/(surface_size); //one pixel on the UV map
var op_middle = op*.5; //The pixel middle on the UV map

//Create the surface that will be converted to a buffer and set it as the target
var my_surface = surface_create(surface_size,surface_size);
surface_set_target(my_surface);

//Clear the surface and draw the sprite
draw_clear_alpha(c_black,0.0);
draw_sprite(sprite,index,sprite_get_xoffset(sprite),sprite_get_yoffset(sprite));

surface_reset_target(); //Reset the target

//Create the buffer and convert the surface to a buffer
sprite_buffer = buffer_create((surface_size*surface_size)*4,buffer_fast,1);
buffer_get_surface(sprite_buffer,my_surface,1,0,0);

surface_free(my_surface); //Free the surface from memory

//Begin defining a format
vertex_format_begin();

vertex_format_add_position_3d();//Add 3D position info
vertex_format_add_colour();//Add colour info
vertex_format_add_textcoord();//Texture coordinate info

//End building the format, and assign the format to the variable "format"

var format = vertex_format_end();

vertex_begin(temp,format);

while(ii < surface_size*surface_size) //For every pixel in the surface
{
    buffer_seek(sprite_buffer,buffer_seek_relative,3); //The RGB values are ignored so skip them

    //If the value of the pixel's alpha is larger than 128 count the pixel as visible
    bool_value = buffer_read(sprite_buffer,buffer_u8) > 128;
   
    if bool_value = true //If the pixel is visible
    {
   
    //Front
    buffer_vertex_normal_texture(temp,xx,yy,0,-1,-1,1,(op*xx)+(op_middle),(op*yy)+(op_middle));
    buffer_vertex_normal_texture(temp,xx+1,yy,0,1,-1,1,(op*xx)+(op_middle),(op*yy)+(op_middle));
    buffer_vertex_normal_texture(temp,xx,yy+1,0,-1,1,1,(op*xx)+(op_middle),(op*yy)+(op_middle));
   
    buffer_vertex_normal_texture(temp,xx+1,yy,0,1,-1,1,(op*xx)+(op_middle),(op*yy)+(op_middle));
    buffer_vertex_normal_texture(temp,xx+1,yy+1,0,1,1,1,(op*xx)+(op_middle),(op*yy)+(op_middle));
    buffer_vertex_normal_texture(temp,xx,yy+1,0,-1,1,1,(op*xx)+(op_middle),(op*yy)+(op_middle));
       
    //Back   
    buffer_vertex_normal_texture(temp,xx,yy,dd,-1,-1,-1,(op*xx)+(op_middle),(op*yy)+(op_middle));
    buffer_vertex_normal_texture(temp,xx+1,yy,dd,1,-1,-1,(op*xx)+(op_middle),(op*yy)+(op_middle));
    buffer_vertex_normal_texture(temp,xx,yy+1,dd,-1,1,-1,(op*xx)+(op_middle),(op*yy)+(op_middle));
   
    buffer_vertex_normal_texture(temp,xx+1,yy,dd,1,-1,-1,(op*xx)+(op_middle),(op*yy)+(op_middle));
    buffer_vertex_normal_texture(temp,xx+1,yy+1,dd,1,1,-1,(op*xx)+(op_middle),(op*yy)+(op_middle));
    buffer_vertex_normal_texture(temp,xx,yy+1,dd,-1,1,-1,(op*xx)+(op_middle),(op*yy)+(op_middle));

    //Left
    if !last_bool_value
    {
    buffer_vertex_normal_texture(temp,xx,yy,0,-1,1,0,op*xx,op*yy);
    buffer_vertex_normal_texture(temp,xx,yy,dd,-1,1,0,op*xx,op*yy);
    buffer_vertex_normal_texture(temp,xx,yy+1,dd,-1,1,0,op*xx,op*yy);

    buffer_vertex_normal_texture(temp,xx,yy,0,-1,1,0,op*xx,op*yy);
    buffer_vertex_normal_texture(temp,xx,yy+1,dd,-1,1,0,op*xx,op*yy);
    buffer_vertex_normal_texture(temp,xx,yy+1,0,-1,1,0,op*xx,op*yy);
    }

    //Right
    buffer_vertex_normal_texture(temp,xx+1,yy,0,-1,1,0,op*xx,op*yy);
    buffer_vertex_normal_texture(temp,xx+1,yy,dd,-1,1,0,op*xx,op*yy);
    buffer_vertex_normal_texture(temp,xx+1,yy+1,dd,-1,1,0,op*xx,op*yy);

    buffer_vertex_normal_texture(temp,xx+1,yy,0,-1,1,0,op*xx,op*yy);
    buffer_vertex_normal_texture(temp,xx+1,yy+1,dd,-1,1,0,op*xx,op*yy);
    buffer_vertex_normal_texture(temp,xx+1,yy+1,0,-1,1,0,op*xx,op*yy);
   
    //Top
    buffer_vertex_normal_texture(temp,xx,yy,0,-1,1,0,op*xx,op*yy);
    buffer_vertex_normal_texture(temp,xx,yy,dd,-1,1,0,op*xx,op*yy);
    buffer_vertex_normal_texture(temp,xx+1,yy,dd,-1,1,0,op*xx,op*yy);

    buffer_vertex_normal_texture(temp,xx,yy,0,-1,1,0,op*xx,op*yy);
    buffer_vertex_normal_texture(temp,xx+1,yy,0,-1,1,0,op*xx,op*yy);
    buffer_vertex_normal_texture(temp,xx+1,yy,dd,-1,1,0,op*xx,op*yy);

    //Bottom
    buffer_vertex_normal_texture(temp,xx,yy+1,0,-1,1,0,op*xx,op*yy);
    buffer_vertex_normal_texture(temp,xx,yy+1,dd,-1,1,0,op*xx,op*yy);
    buffer_vertex_normal_texture(temp,xx+1,yy+1,dd,-1,1,0,op*xx,op*yy);

    buffer_vertex_normal_texture(temp,xx,yy+1,0,-1,1,0,op*xx,op*yy);
    buffer_vertex_normal_texture(temp,xx+1,yy+1,0,-1,1,0,op*xx,op*yy);
    buffer_vertex_normal_texture(temp,xx+1,yy+1,dd,-1,1,0,op*xx,op*yy);
    }
    /*
    We're saving the bool_value in another variable so we can check
    whether this pixel was solid next time the loop goes around
    */
    last_bool_value = bool_value;
   
    xx+=1; //Increment the xx value   
    if xx >= surface_size //If xx is too far over
    {
    yy-=1; //Move the yy up one
    xx = 0; //Move the xx back to the start
    }
    ii+=1; //Increment ii so we know how many times we've gone through this
}
vertex_end(temp); //The model has been created so we finalize it

buffer_delete(sprite_buffer); //The buffer is no longer needed so we delete it

return temp; //Return the vertex buffer
Here's the custom buffer_vertex_normal_texture script I used, for some reason the normals don't work, I'm hoping someone smarter than me can figure it out.
Code:
///buffer_vertex_normal_texture(buffer,x,y,z,normal_x,normal_y,normal_z,tex_x,tex_y)
var temp = argument0;
var xx = argument1;
var yy = argument2;
var zz = argument3;
var nx = argument4;
var ny = argument5;
var nz = argument6;
var tx = argument7;
var ty = argument8;

var col = c_white;
var alpha = 1.0;

vertex_position_3d(temp, xx, yy, zz);
vertex_colour(temp, col, alpha);
vertex_texcoord(temp, tx, ty);
//vertex_normal(temp,nx,ny,nz);
Keep you original sprite, but make a copy. Now scale the copy up to 400% with the "Poor" quality selected (so there's no blurring). Now on each "pixel" of the sprite is made up of a square of 4x4 smaller pixels. Using the UV map image above, you can change the pixel colors for each of the sides of the pixel. For example, you can simulate lighting by making the top faces brighter and the bottom faces darker. Or by making all the side faces black you can give it an outline.
The alpha channel is the only part of the pixel that is being used. So you could pass in a heightmap to the create_model_from_sprite script to control how thick each individual pixel is.

Here's another download link to the project file: GMS:1.4 .gmz file

I'll be happy to answer any questions or make improvements that might be necessary.
 
J

jorgernes

Guest
Quiero convertir mi sprits a 3d3..lo hice con pixkel... Voy aaestudiar esta documentacion... Gracias
 
Top