Skip to content

A PLAY EXPERIENCE MAKER'S WORK LOG FOR FUTURE SELF©2001 – 2023 Kyle Li 李肅綱 All Rights Reserved.

antibody

Posted on August 26, 2021September 19, 2022 by admin
original stimuli

Tokusatsu nerds of my generation probably all mesmerized by Space Sheriff Gavan (宇宙刑事ギャバン)’s Laser Blade when the show aired in 1982, especially when he powers it up before attack. The latest Laser Blade toy by Tamashii Lab was able to bring the powering up experience to live. How do I bring this experience into VR in an embodied way? How does a real-life artifact come to live in the virtual world? This is my attempt. Here are some other inspirations that I would like to incorporate in this experiment:

Saint Seiya Omega – Mars
Lady of the Lake gives Excalibur to King Arthur
ALT.CTRL
Berserker – The Dragon Slayer
Characters by Michael Shillingburg
This feel (pastel low poly)
Low poly swamp and wetland:
https://assetstore.unity.com/packages/3d/environments/low-poly-modular-terrain-pack-91558
Table Eleven Paddle:
https://www.thingiverse.com/thing:4623428
Shaman King

Download

The artifact is the halt of the sword. When it is activated in VR, the galaxy blade will emerged on the virtual halt. The player can power up the blade with cosmic flares (or soul fire) for bigger attacks. I need to build a (bulky) halt that hosts both the touch controller, M5Stack, and the distance sensor unit. I gathered some 3D models from Thingverse including the Table Eleven paddle and a 3D scan of the left touch controller. I had to demesh the paddle before importing it to Tinkercad. I studied the paddle and decided to make my own. After some trial and error, I made a head piece that slide into the ring of the touch controller smoothly. I then built the whole halt from there.

I had a very vague image in my head of what the sword would look like. The alt controller is the artifact, a physical medium, that brings the virtual sword to the player. It has to be oversize, galaxy like, and burning with some sort of cosmic flares (or soul fire). After some tinkering in Unity, this is what I came up with:

The player can ignite the soul fire by putting one’s hand right in front of the distance sensor unit after certain time and increase the fire coverage on the blade by moving one’s hand away from the sensor. The player will be given a quick hands-on tutorial when the soul fire is awaken in the play experience. Testing video on Instagram was pick up by M5Stack, what a lovely surprise!

https://www.instagram.com/p/CTD4CqAlnX4

When I was living near the school around 2008, there was a cute German couple living in the same building. They look like they are in their 70s. I often ran into them when they are doing their grocery shopping in the late afternoon. They talk very loud as if they were arguing but when they split to do different things they always give each other a kiss on the cheek. From our elevator conversations, I found out the husband was teaching photography at Parsons for many years, and they escaped to the US in 1950s as a result of wars in Europe. I have heard The New School helped many artists and designers escape and offered them shelters and jobs. I was very honor to meet two of them!

I sometime walked a few blocks with them just to hear more stories. One day, when the wife found out that I am making games, she let me know that she won her battle against cancer earlier because of video games. When she was sick, she had to go to hospital to have chemotherapies. It always felt awful both physically and mentally after the treatment. Luckily, she found out a store across her hospital had a few arcade cabinets. It has become a routine of hers to go right into the store and play arcade games after every hospital visit. She said those games make her happy and stop the awful feelings from spreading. When her cancer was cured, she had thought that the video games are the unsound heroes of her victory.

Antibody

Imagine a future where medical treatments can be executed remotely in the form of immersive video games. 

This idea was inspired by my friend, Grace, a German grandma who lives in my apartment building. She is a proud cancer survivor and she has convinced me that playing Space Invader for 15 minutes after every hospital visit was the key to  curing her cancer. I have read several similar stories like this one in which patients dreamed about fighting against monstrous enemies in a video game and woke up fully recovered from their illness. I am intrigued by the prospect of immersive technology transforming these miracles into a universally practical cure. 

Antibody is my speculative scenario situated in a near future with advancement in neuroscience and nanotechnology. The medical facilities are capable of sending skilled gamers into infection zones as antibodies and helping white cells build up immunity. These gamers are equipped with various experimental nano-weapons that enable them to behave differently in the field. In this quick mission for beginners, a broadsword nano-weapon is available for action.

To enhance the level of immersion, a specialized controller is available for gamers to replace the standard controllers that come with their VR headset. These specialized controllers, a.k.a data relics, usually resemble the look and feel of the nano-weapons in the virtual world. They are capable of harvesting kinetic energy and associated data in the real world to aid the medical facilities in improving the technology and training better antibody agents. Experienced gamers customize their personal data relic to access advanced game mechanics. In this submission, the simulation is designed for a standard VR controller, no data relic is required.   

As a playful experience designer, I believe video games may contribute more to the world without compromising the fun. This is my attempt at envisioning a post-pandemic future with virtual gaming for social good and I hope you enjoy it. 
PCVR and Standalone VR ONLY

Kyle Li is a playful experience designer working and living in New York City. Experimental by nature, his body of work wraps around playful experiences manifested by interconnected physical and digital components.  He has done a wide range of works from concert stage visual to airline cockpit data visualization to the award-winning game-and-learning installations at middle-schools in both NYC and Chicago.

Stage 1: Spores
Spores detects the player based on proximity.

Stage 2: Virus infected lukes
Virus infected lukes behaves like zombie but they are curable. They will walk towards to the player and attack. The player has the ability to see the infected points on a luke’s body, breaks all the points on a luke with Galaxcalibur will cure it. Cured lukes will join the the group of lukes that is following the player.

Stage 2.5: Free imprisoned lukes
The player points the Galaxcalibur at the lock to activate the unlock sequence. Follow the rotating sequence by rotating the sword.

Stage 3: Boss Fight
No idea, it has to be huge, and lukes are going to help.

Alt Controller: Galaxcalibur
M5Stack will collect “Energy” based on Galaxcalibur’s movement, which effects the pulse of the vibration. When energy reaches a threshold. The player can use a devastated attack and this activated by putting player’s hand in front of the ToF sensor.


Zok Zok Heroes

Posted on July 20, 2021July 20, 2021 by admin

月刊コロコロコミック 2000年8月号

Zok Zok Heros was on KOROKORO Monthly Comic (月刊コロコロコミック) from June 2000 to December 2000. The 2000 August issue came with a poster of Zok Zok Heroes and their Universe Symbols (ぞくぞくヒーローズ超図鑑).

Make (X) Works for You!

Posted on July 4, 2021July 4, 2021 by admin

I had done a few projects in the past using IR remote as a way of wireless communication. I used FLIRC USB (https://flirc.tv/more/flirc-usb). It is essentially a USB IR receiver that plugs into your computer and turns IR signals into specific keystrokes. The great thing about FLIRC is that once you configured it on a computer with its application, you can use it anywhere with a USB port. The only downside is that the FLIRC USB only offers six customizable inputs (up, down, left, right, enter, and back) because it is meant to be used for media playback. I just figured out a way to do it with M5Stack and the IR Unit now.

IRremoteESP8266 Library also works for ESP32 (M5Stack)
https://github.com/crankyoldgit/IRremoteESP8266

I am using the M5Stack Core + IR Unit (Port B), so have to change the pin to 36.

Upload it to M5Stack, get the readout (Code) from the serial monitor. Since I don’t need to recreate the IR signal, the Code here will work just fine.

Now, M5Stack can read the remote and react!

In order to work with result.value, I need a uint64_t to String helper method to make the comparison easier for me.

String u64ToS(uint64_t input) {
  String result = "";
  uint8_t base = 10;
  do {
    char c = input % base;
    input /= base;
    if (c < 10)
      c +='0';
    else
      c += 'A' - 10;
    result = c + result;
  } while (input);
  return result;
}

or

String print64(uint64_t n) {
   char buf[21];
   char *str = &buf[sizeof(buf) - 1];
   String sdata = "";
   *str = '\0';
   do {
     uint64_t m = n;
     n /= 10;
     *--str = m - 10 * n + '0';
   } while (n);
   sdata += str;
   return sdata;
 }

Kano Wand

Posted on June 24, 2021 by admin

Useful links:

Kano Wand Hack:
https://hackaday.io/project/161832-kano-wand-hack
Bluepy Based Solution:
https://www.instructables.com/SmartWand/
Arduino Wifi Rev2 and ESP32:
https://github.com/sabas1080/KanoWandHack
Unofficial library: 
https://github.com/anzellai/wandkit
Bidirectional communication between Kano Coding Wand and Client connection. 
https://github.com/anzellai/bluewand
SOAI (Tiny Open Source API) to speak with the KANO Wand 
https://github.com/wizofe/bluewand-python
Demos using the Kano Wand python module
https://github.com/GammaGames/kano-wand-demos

Put that Gucci on, VRChat style!

Posted on June 16, 2021June 19, 2021 by admin
  • Purchase the virtual product in the Gucci app for $12.99
  • Before sending the download to yourself or anyone you would want to share the content with, there is a long instruction, long but still unclear.
  • I first create an avatar with https://vrchat.readyplayer.me/avatar and download the FBX file.
  • Two weeks later – I finally got my New User status.
    It was a little frustrating at the beginning. Based on the internet, I need 10 hours of gameplay to get to the New User status which allows me to upload Avatars and Worlds to the lab community. I did that but nothing happened. A kind player in VRChat told me that I need to make friends too. Oh, why I didn’t think of that. After I made one friend, the next time I logged in, I was promoted to New User. And I got this in the e-mail.
  • I brought the Gucci model into Maya and separate the shoes from the model they came with and put them on my VRChat avatar.
  • Install Unity 2018.4.20f1 that is recommended by VRChat.
  • Upload my Avatar by following this tutorial – in 5 minutes.
    https://vrchat.fandom.com/wiki/Quick_Start_-_Mixamo_Avatar_Creation
    I believe this was written for SDK2 but works for SDK3, especially when selecting the Animator Controller. I use the T-pose one from SDK3 and it works fine. I was able to upload, however, my sitting and ducking animations were messed up…
  • To make it work for both Quest and PC, I have to upload the avatar twice. Once in Windows mode and the other in Android mode. The official tutorial online recommends duplicating the project.

Now upload D1208:

  • I am following this tutorial:
    https://docs.vrchat.com/docs/vrchat-home-kit
  • I realized SDK3 World and SDK3 Avatar can’t be in the same project.

ZOOM Meeting Exit Act III – End of Semester Celebration

Posted on May 12, 2021May 17, 2021 by admin
ZOOM EXIT

Just in a few hours, I am going to participate in the graduation ceremony for our Class 2021 cohort. I worked with a good portion of them extensively in their first year of school in virtual reality and playful experience design. I wanted to make something to remember them by. It is purely impulsive and based on no research or data. It is very last minute, but I decided to celebrate this moment with a new exit act.

After we moved the learning online after the pandemic last year, I have been experiment with different online identities. From animated characters with face tracking to cosmetic filters, it was an unexpected adventure into a world of digital tools that I didn’t care much for. During those experiments, I created a thing called exit acts – exit zoom meetings with style. I created one for taking off in a Gundam RX-78 and another one for flying away with Spike’s Swordfish II from Cowboy Bebop. Jamming it out with an EVA – 3 from Neo Genesis Evangelion seems like a reasonable next project.

MFADT Class 2021 Visual by @major_benny

My teaching assistant, Sharon, told me that the overall visual scheme of their thesis website was inspired by the color scheme of the animation -Evangelion. Time to collect some assets and put together a theme. I want to use a cityscape silhouette to create a high contrast between foreground and background similar to the technique used in some of the Fortnite banners.

Evangelion Unit 1:
https://sketchfab.com/3d-models/evangelion-unit-01-49c7a77272c84154a711d7cfd5cb47f4

Evangelion – Entry Plug(with sound):
https://sketchfab.com/3d-models/evangelion-entry-plugwith-sound-3e4245a5f40b45eca02dab55e6b47a93

Here is the end result:

M5Stack Go

Posted on April 29, 2021May 22, 2021 by admin

SETUP

*Thanks to Kobayashi sensei of IMAAS for the initial introduction to M5Stack.*

The M5Stack Go kit is available on Amazon.com:
https://www.amazon.com/gp/product/B07F8QR7NB/

1 stop for all the tools, plugins, and drivers:
https://shop.m5stack.com/pages/download
Downloaded CP2104 Driver and M5Burner.

Run M5Burner and update the firmware of M5Stack Go to v1.7.6-en
(The latest version available at the moment)

Install CP2104 Driver.

Follow this tutorial to set up Arduino IDE. I am using Arduino 1.8.13, it works with this setup.

Useful links from the video:
Virtual com port drivers https://www.silabs.com/developers/usb-to-uart-bridge-vcp-drivers
**This link opens up a blank page in my Chrome but works in Firefox.
Arduino IDE arduino.cc/en/Main/Software
Boards manager link https://dl.espressif.com/dl/package_esp32_index.json​
M5stack library https://github.com/m5stack/M5Stack​
**Now M5stack Library appears directly in the Arduino library manager on searching M5Stack

TEST 1 – HID KEYBOARD

The USB HID (Human Interface Device) specification allows us to build plug-and-play alternative controllers for a variety of devices without the need to install additional drivers. In this test, M5Stack Go is disguised as a normal HID Keyboard. Instead of serial communication, M5Stack GO will process the sensor data locally and send a correspondent keystroke defined by the designer. Keystrokes are like digital switches, so HID Keyboard can’t send over analog data.

Install library ESP32-BLE-Keyboard manually from GitHub:
https://github.com/T-vK/ESP32-BLE-Keyboard

Based on this tutorial and combine it with the DUAL_BUTTON example.
https://programresource.net/en/2020/04/09/3251.html

Tested and worked with STYLY in Unity, on the Web, in AR (Mobile), in Oculus Quest (standalone), and on PCVR (Oculus Link or Air Link).

PCVR and WEB TEST SCENE
Connect the controller to your computer, go to the following page and add it to your list. PCVR users find the scene in My List in STYLY app. Web users can just open it in the browser.
https://gallery.styly.cc/scene/871fdff9-475c-4224-af41-9e8afba32922

Quest (Portable)
Connect the controller to your headset, go to the link above and add it to your list. You can find it in My List in your STYLY app.

AR TEST SCENE
Connect the controller to your mobile phone and scan the QR code with the STYLY app.

Advanced Topic: Would it be possible to read bleKeyboard.print() all at once as a string. If so, I can send over analog data as a string without having to scale it down to letters.

Arduino Project: M5HIDKEYBOARD
Test Scene Unitypackage *require Playmaker and STYLY installed

TEST 2 – HID GAMEPAD I

The biggest difference with HID Gamepad is the ability to send over joystick information. A joystick is made of 2 axes (usually 10K potentiometers), so with left and right joysticks and the two shoulder buttons, HID Gamepad is capable of communicating 6 analog data over in the format of axes by default. It’s worth mentioning that HID mouse is also capable of communicating over 1 analog value over its scroll wheel.

Download and install library ESP32-BLE-GAMEPAD from GitHub:
https://github.com/lemmingDev/ESP32-BLE-Gamepad

Combine IndividualAxes and PotAsAxis example in ESP32-BLE-Gamepad and JOYSTICK example in M5Stack/Units.

After some initial testing, the old input system in Unity editor can’t recognize the ESP32 BLE Gamepad. When connected through Bluetooth, it doesn’t show up as a gamepad as the TEST 3 did. The new input system recognizes it but the action needed to access the inputs is a custom one which is not included in Playmaker’s standard actions.

Digitom (New) Input System actions:
https://hutonggames.com/playmakerforum/index.php?topic=21797.msg95729#msg95729

Install link:
https://gitlab.com/toomasio/playmakerinputsystemactions.git

Tutorial worth watching:
https://www.youtube.com/watch?v=tfI6KEf5CHA

TEST 3 – HID GAMEPAD II

This test is based on a custom library put together by Shigeru Kobayashi. This library allows the ESP32 gamepad to be discovered by both old and new input systems in Unity 3D as the default Axes X and Y and buttons 0 to button 4.

TEST 4 – SPEECH TO TEXT

https://github.com/MhageGH/esp32_CloudSpeech

  1. Set up a (free) billing account on Google and enable the service: https://cloud.google.com/speech-to-text/
  2. Recommend to go through the tutorial and it will create the service with a click of a button.
    https://cloud.google.com/speech-to-text/docs/quickstart-protocol
  3. Under the Credentials, create an API KEY
  4. Set network parameter and your account information in network_param.h.
  5. Say to the microphone and see the serial monitor.

Test this in the command prompt (Windows 10):
curl -s -X POST -H "Content-Type: application/json" -d @request.json "https://speech.googleapis.com/v1/speech:recognize?key= API_KEY"

  • Make sure the request.json file is in the same directory in which the curl command is executed.

In order to get this sketch to run on M5Stack, I have to set up OPENSSL on my Windows 10. Here is a recent tutorial:
https://tecadmin.net/install-openssl-on-windows/
One confusing thing in this tutorial is the directory name. I downloaded the Winx64 version as instructed, so my directory should be C:\OpenSSL-Win64 not C:\OpenSSL-Win32 as shown in the tutorial. The environmental variables should be like this:

set OPENSSL_CONF=C:\OpenSSL-Win64\bin\openssl.cfg
set Path=......Other Values here......;C:\OpenSSL-Win64\bin
  • Don’t forget to restart the computer afterward.
  • After generating the certificate with OpenSSL in the command prompt, copy and replace the value of root_ca. Make sure to keep the same exact format.
  • One last thing that caused an error callback for me was the name of a property in the HttpBody1 was wrong. It should be “sampleRateHertz\” not “sampleRate\”. While I was there, I also changed “languageCode\”:\”ja-JP\” to “languageCode\”:\”en-US\”.

I was able to get it to work after all these changes. I said “Hello Hello” and this came back in the Serial Monitor, pretty satisfying!

{
 "results": [
  {
   "alternatives": [
    {
     "transcript": "hello hello",
     "confidence": 0.95336735
    }
   ]
  }
 ]
}

I realized the client.read() returns more than just the JSON data, there is a huge header too. After some research online, didn’t find anything useful on how to get rid of the header, so I went for the old-school way -> remove(); After cleaning up extra strings both before and after the data, it works beautifully with the ArduinoJSON library.

Helps to calculate the size of the data and also shows how to access specific data in the JSON tree. I was having problems understanding the examples in the library but this tool did it for me :
https://arduinojson.org/v6/assistant/

To get this code to work, I made a total of 48 requests to the Google Cloud Platform and used 2.75 minutes. I think as an indie experimentalist, 60 minutes/month for free is a really good deal.

View this post on Instagram

A post shared by Kyle Li (@beastmarias)

Proudly powered by WordPress | Theme: MiniZen by Martin Stehle.